0

我有两段代码。一种有效，一种无效。

以下代码按预期运行，没有错误：（注意：postrain、、、和是前面定义的字符串列表。）negtrainpostestnegtest

from sklearn.feature_extraction.text import CountVectorizer

vector = CountVectorizer()
train_vector = vector.fit_transform(postrain+negtrain)
test_vector = vector.transform(postest+negtest)
print test_vector.shape

但是，此代码会引发错误：

import re

stop = [re.split('\n|\t', open('stop_words.txt').read())]

vector2 = CountVectorizer(stop_words=stop)
train_vector = vector2.fit_transform(postrain+negtrain) # <-- Error occurs here
test_vector = vector2.transform(postest+negtest)
print test_vector.shape

错误：

TypeErrorTraceback (most recent call last)
<ipython-input-43-cf5f4754d58c> in <module>()
      7 
      8 vector2 = CountVectorizer(stop_words=stop)
----> 9 train_vector = vector2.fit_transform(postrain+negtrain)
     10 test_vector = vector2.transform(postest+negtest)
     11 

C:\Users\Nsth\Anaconda2\envs\cs489\lib\site-packages\sklearn\feature_extraction\text.pyc in fit_transform(self, raw_documents, y)
    815 
    816         vocabulary, X = self._count_vocab(raw_documents,
--> 817                                           self.fixed_vocabulary_)
    818 
    819         if self.binary:

C:\Users\Nsth\Anaconda2\envs\cs489\lib\site-packages\sklearn\feature_extraction\text.pyc in _count_vocab(self, raw_documents, fixed_vocab)
    745             vocabulary.default_factory = vocabulary.__len__
    746 
--> 747         analyze = self.build_analyzer()
    748         j_indices = _make_int_array()
    749         indptr = _make_int_array()

C:\Users\Nsth\Anaconda2\envs\cs489\lib\site-packages\sklearn\feature_extraction\text.pyc in build_analyzer(self)
    232 
    233         elif self.analyzer == 'word':
--> 234             stop_words = self.get_stop_words()
    235             tokenize = self.build_tokenizer()
    236 

C:\Users\Nsth\Anaconda2\envs\cs489\lib\site-packages\sklearn\feature_extraction\text.pyc in get_stop_words(self)
    215     def get_stop_words(self):
    216         """Build or fetch the effective stop words list"""
--> 217         return _check_stop_list(self.stop_words)
    218 
    219     def build_analyzer(self):

C:\Users\Nsth\Anaconda2\envs\cs489\lib\site-packages\sklearn\feature_extraction\text.pyc in _check_stop_list(stop)
     92         return None
     93     else:               # assume it's a collection
---> 94         return frozenset(stop)
     95 
     96 

TypeError: unhashable type: 'list'

添加停用词是如何导致错误的？

4

1 回答 1

1

我很笨。它应该是：

stop = re.split('\n|\t', open('stop_words.txt').read())

没有括号。不知道为什么它在那之后抛出了错误。

于 2016-11-02T04:09:32.340 回答