1

I'm parsing a German text with many hyphens in it. To check if a word is a proper German word (and only got seperated by a hyphen because it was the end of the line) or needs those hyphens because that is actually how it should be written, I am currently extending a collection of lemmatized words that I found here: https://github.com/michmech/lemmatization-lists

Can you point me to a way how that can be done with nltk?

What I do: when my parser encounters a word with a hyphen, I check spelling without hyphen (i.e. if it is contained in my list with lemmatized words). If it is not contained in my list (currently some 420,000 words) I will check myself if it should be added to my list or written with hyphen.

This is the function that does the work:

 **function(sents, german_words, hyphened_words):**

 clutter = '[*!?,.;:_\s()\u201C\u201D\u201E\u201F\u2033\u2036\u0022]'
 sentences = list()
 new_hyphened_words = list()
 new_german_words = list()
 skip = False

 for i, sentence in enumerate(sents):
     if skip:
         skip = False
         continue
     new_sentence = ''
     words = sentence.split(' ')
     words = list(filter(None, words))
     new_words = list()      # words to make a correct sentence
     last_word = words[-1]
     last_word = last_word.strip()
     if last_word[-1] == '-':
         try:
             next_sentence = sents[i+1]
         except IndexError as e:
             raise e
         next_words = next_sentence.split(' ')
         next_words = list(filter(None, next_words))
         first_word = next_words[0]
         new_word = last_word[:-1] + first_word
         new_word = re.sub(clutter, '', word)
         if _is_url_or_mail_address(new_word):
             new_words = words[:-1] + [new_word] + next_words[1:]
             skip = True
             continue
         elif new_word in german_stopwords:
             new_words = words[:-1] + [new_word] + next_words[1:]
             skip = True
             continue
         elif new_word in german_words:
             new_words = words[:-1] + [new_word] + next_words[1:]
             skip = True
             continue
         else:
             new_word = last_word + first_word       # now with hyphen!
             new_word = re.sub(clutter, '', word)
             if new_word in hyphened_words:
                 new_words = words[:-1] + [new_word] + next_words[1:]
                 skip = True
                 continue
             else:           # found neither with nor without hyphen
                 with_hyphen = re.sub(clutter, '', last_word + first_word)
                 without_hyphen = re.sub(clutter, '', last_word[:-1] + first_word)
                 print(f'1: {with_hyphen}, 2: {without_hyphen}')
                 choose = input('1 or 2, or . if correction')
                 if choose == '1':
                     new_hyphened_words.append(with_hyphen)
                     new_words = words[:-1] + [last_word+first_word] + next_words[1:]
                     skip = True
                     continue
                 elif choose == '2':
                     new_german_words.append(without_hyphen)
                     new_words = words[:-1] + [last_word[:-1]+first_word] +\
                             next_words[1:]
                     skip = True
                     continue
                 else:
                     corrected_word = input('Corrected word: ')
                     print()
                     new_german_words.append(corrected_word)
                     print(f'Added to dict: "{corrected_word}"')
                     ok = input('Also add to speech? ./n')
                     if ok == 'n':
                         speech_word = input('Speech word: ')
                         new_words = words[:-1] + [speech_word] + next_words[1:]
                         skip = True
                         continue
                     else:
                         new_words = words[:-1] + [corrected_word] + next_words[1:]
                         skip = True
                         continue
     else:
         new_words = words

     new_sentence = ' '.join(w for w in new_words)
     sentences.append(new_sentence)

 return sentences

The lists "german_words" and "hyphened_words" get updated every now and again so they contain the new words from the sessions before.

What I do works, however it is slow work. I have been searching for ways to do this with nltk but I seem to have looked at the wrong places. Can you point me to a way that trains an nltk collection of words or that uses a more efficient way of processing this?

4

0 回答 0