3

虽然被列为示例用例之一......我还没有找到从 Hive n-gram 中过滤掉垃圾词(and、or 等)的示例。

SELECT explode(context_ngrams(sentences(lower(description)), array("criminal", null), 10)) AS x FROM mapped_discussions;

{"ngram":["justice"],"estfrequency":274.0}
{"ngram":["behavior"],"estfrequency":121.0}
{"ngram":["law"],"estfrequency":92.0}
{"ngram":["activity"],"estfrequency":69.0}
{"ngram":["acts"],"estfrequency":41.0}
{"ngram":["procedure"],"estfrequency":35.0}
{"ngram":["and"],"estfrequency":29.0}
{"ngram":["or"],"estfrequency":27.0}
{"ngram":["case"],"estfrequency":26.0}
{"ngram":["cases"],"estfrequency":25.0}

有任何想法吗?谢谢!

4

1 回答 1

3

这里有一篇关于这个主题的优秀文章。http://bigdatabloggin.blogspot.com/2012/08/trending-topics-in-hive-ngrams.html

于 2013-05-24T01:39:31.473 回答