虽然被列为示例用例之一......我还没有找到从 Hive n-gram 中过滤掉垃圾词(and、or 等)的示例。
SELECT explode(context_ngrams(sentences(lower(description)), array("criminal", null), 10)) AS x FROM mapped_discussions;
{"ngram":["justice"],"estfrequency":274.0}
{"ngram":["behavior"],"estfrequency":121.0}
{"ngram":["law"],"estfrequency":92.0}
{"ngram":["activity"],"estfrequency":69.0}
{"ngram":["acts"],"estfrequency":41.0}
{"ngram":["procedure"],"estfrequency":35.0}
{"ngram":["and"],"estfrequency":29.0}
{"ngram":["or"],"estfrequency":27.0}
{"ngram":["case"],"estfrequency":26.0}
{"ngram":["cases"],"estfrequency":25.0}
有任何想法吗?谢谢!