analytics - Javascript 中的停用词删除

Question

嗨，我正在寻找一个可以从 in 文本中删除停用词的库Javascript，我的最终目标是计算 tf-idf，然后将给定的文档转换为向量空间，而这一切都是Javascript. 任何人都可以将我指向一个可以帮助我做到这一点的图书馆。只是一个删除停用词的图书馆也会很棒。

Answer 1

使用NLTK 库提供的停用词：

stopwords = ['i','me','my','myself','we','our','ours','ourselves','you','your','yours','yourself','yourselves','he','him','his','himself','she','her','hers','herself','it','its','itself','they','them','their','theirs','themselves','what','which','who','whom','this','that','these','those','am','is','are','was','were','be','been','being','have','has','had','having','do','does','did','doing','a','an','the','and','but','if','or','because','as','until','while','of','at','by','for','with','about','against','between','into','through','during','before','after','above','below','to','from','up','down','in','out','on','off','over','under','again','further','then','once','here','there','when','where','why','how','all','any','both','each','few','more','most','other','some','such','no','nor','not','only','own','same','so','than','too','very','s','t','can','will','just','don','should','now']

然后只需将您的字符串传递给以下函数：

function remove_stopwords(str) {
    res = []
    words = str.split(' ')
    for(i=0;i<words.length;i++) {
       word_clean = words[i].split(".").join("")
       if(!stopwords.includes(word_clean)) {
           res.push(word_clean)
       }
    }
    return(res.join(' '))
}  

示例：

remove_stopwords("I will go to the place where there are things for me.")

结果：

I go place things

只需向您的 NLTK 数组中添加尚未涵盖的任何单词。

Answer 2

我认为没有此类库，您需要从https://www.ranks.nl/stopwords下载这些单词。

然后按如下方式替换单词：

text = text.replace(stopword, "")

Answer 3

这是一个带有英文停用词的数组。希望能帮助到你。来自http://www.ranks.nl/stopwords（在前面的答案中提到）。

此外，这对您来说可能是一个有用的资源。

https://github.com/shiffman/A2Z-F16/tree/gh-pages/week5-analysis

http://shiffman.net/a2z/text-analysis/

var stopwords = ["a", "about", "above", "after", "again", "against", "all", "am", "an", "and", "any","are","aren't","as","at","be","because","been","before","being","below","between","both","but","by","can't","cannot","could","couldn't","did","didn't","do","does","doesn't","doing","don't","down","during","each","few","for","from","further","had","hadn't","has","hasn't","have","haven't","having","he","he'd","he'll","he's","her","here","here's","hers","herself","him","himself","his","how","how's","i","i'd","i'll","i'm","i've","if","in","into","is","isn't","it","it's","its","itself","let's","me","more","most","mustn't","my","myself","no","nor","not","of","off","on","once","only","or","other","ought","our","ours","ourselves","out","over","own","same","shan't","she","she'd","she'll","she's","should","shouldn't","so","some","such","than","that","that's","the","their","theirs","them","themselves","then","there","there's","these","they","they'd","they'll","they're","they've","this","those","through","to","too","under","until","up","very","was","wasn't","we","we'd","we'll","we're","we've","were","weren't","what","what's","when","when's","where","where's","which","while","who","who's","whom","why","why's","with","won't","would","wouldn't","you","you'd","you'll","you're","you've","your","yours","yourself","yourselves"];

Answer 4

这里有一个用于删除停用词的 Javascript 库：http: //geeklad.com/remove-stop-words-in-javascript