nltk - 导入我自己的文本以使用 NLTK 词性标注器

Question

我是这方面的初学者，但我想创建一个文件夹，其中有很多文本（假设小说保存为 .txt）。然后我想请用户选择其中一本小说，然后自动让词性标注器分析整个文本。这可能吗？我一直在尝试：

text = nltk.word_tokenize("And now for something completely different")
nltk.pos_tag(text)

如何让它分析用户选择的文本而不是这句话？以及如何导入这些文本？

Answer 1

有几种方法可以读取文本文件目录。

让我们先从终端/控制台/命令提示符尝试原生 python 方式：

~$ mkdir ~/testcorpora
~$ cd ~/testcorpora/
~/testcorpora$ ls
~/testcorpora$ echo 'this is a foo foo bar bar.\n bar foo, dah dah.' > somefoobar.txt
~/testcorpora$ echo 'what are you talking about?' > talkingabout.txt
~/testcorpora$ ls
somefoobar.txt  talkingabout.txt
~/testcorpora$ cd ..
~$ python
>>> import os
>>> from nltk.tokenize import word_tokenize
>>> from nltk.tag import pos_tag
>>> corpus_directory = 'testcorpora/'
>>> for infile in os.listdir(corpus_directory):
...     with open(corpus_directory+infile, 'r') as fin:
...             pos_tag(word_tokenize(fin.read()))
... 
[('what', 'WP'), ('are', 'VBP'), ('you', 'PRP'), ('talking', 'VBG'), ('about', 'IN'), ('?', '.')]
[('this', 'DT'), ('is', 'VBZ'), ('a', 'DT'), ('foo', 'NN'), ('foo', 'NN'), ('bar', 'NN'), ('bar.\\n', 'NN'), ('bar', 'NN'), ('foo', 'NN'), (',', ','), ('dah', 'NN'), ('dah', 'NN'), ('.', '.')]

另一个解决方案是PlaintextCorpusReader在 NLTK 中使用，然后运行word_tokenize并pos_tag在语料库上查看使用 NLTK 创建新语料库：

>>> from nltk.corpus.reader.plaintext import PlaintextCorpusReader
>>> from nltk.tag import pos_tag
>>> corpusdir = 'testcorpora/'
>>> newcorpus = PlaintextCorpusReader(corpusdir,'.*')
>>> dir(newcorpus)
['CorpusView', '__class__', '__delattr__', '__dict__', '__doc__', '__format__', '__getattribute__', '__hash__', '__init__', '__module__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', '_encoding', '_fileids', '_get_root', '_para_block_reader', '_read_para_block', '_read_sent_block', '_read_word_block', '_root', '_sent_tokenizer', '_tag_mapping_function', '_word_tokenizer', 'abspath', 'abspaths', 'encoding', 'fileids', 'open', 'paras', 'raw', 'readme', 'root', 'sents', 'words']
# POS tagging all the words in all text files at the same time.
>>> newcorpus.words()
['this', 'is', 'a', 'foo', 'foo', 'bar', 'bar', '.\\', ...]
>>> pos_tag(newcorpus.words())
[('this', 'DT'), ('is', 'VBZ'), ('a', 'DT'), ('foo', 'NN'), ('foo', 'NN'), ('bar', 'NN'), ('bar', 'NN'), ('.\\', ':'), ('n', 'NN'), ('bar', 'NN'), ('foo', 'NN'), (',', ','), ('dah', 'NN'), ('dah', 'NN'), ('.', '.'), ('what', 'WP'), ('are', 'VBP'), ('you', 'PRP'), ('talking', 'VBG'), ('about', 'IN'), ('?', '.')]