传递df[language]
适用于停用词,但不适用于雪球词干分析器。有没有办法解决这个问题?
到目前为止,我还没有真正找到任何线索...
import nltk
from nltk.corpus import stopwords
import pandas as pd
import re
df = pd.DataFrame([['A sentence in English', 'english'], ['En mening på svenska', 'swedish']], columns = ['text', 'language'])
def tokenize(text):
tokens = re.split('\W+', text)
return tokens
def remove_stopwords(tokenized_list, language):
stopword = nltk.corpus.stopwords.words(language)
text = [word for word in tokenized_list if word not in stopword]
return text
def stemming(tokenized_text, l):
ss = nltk.stem.SnowballStemmer(l)
text = [ss.stem(word) for word in tokenized_text]
return text
df['text_tokenized'] = df['text'].apply(lambda x: tokenize(x.lower()))
df['text_nostop'] = df['text_tokenized'].apply(lambda x: remove_stopwords(x, df['language']))
df['text_stemmed'] = df['text_nostop'].apply(lambda x: stemming(x, df['language']))
我希望它能够像删除停用词一样使用英语和瑞典语作为语言进行雪球词干提取。我收到error
如下消息:
ValueError:Series 的真值不明确。使用 a.empty、a.bool()、a.item()、a.any() 或 a.all()。