我正在寻找一个 Java 库来从文本块中提取关键字。
该过程应如下所示：
停止词清理 -> 词干提取 -> 根据英语语言学统计信息搜索关键字 - 意思是如果一个词在文本中出现的次数比在英语中出现的次数多于它作为候选关键字的概率。
是否有执行此任务的库？
问问题
22877 次
3 回答
43
这是使用Apache Lucene的可能解决方案。我没有使用最后一个版本，而是使用3.6.2版本，因为这是我最了解的版本。除了/lucene-core-x.x.x.jar，不要忘记将/contrib/analyzers/common/lucene-analyzers-x.x.x.jar下载的存档中的 添加到您的项目中：它包含特定于语言的分析器（尤其是您的情况下的英语分析器）。
请注意，这只会根据它们各自的词干找到输入文本词的频率。之后应将这些频率与英语统计数据进行比较（顺便说一下，这个答案可能会有所帮助）。
数据模型
一个词干一个关键词。不同的词可能有相同的词干，因此是terms集合。每次找到新术语时，关键字频率都会增加（即使已经找到 - 一组会自动删除重复项）。
public class Keyword implements Comparable<Keyword> {
private final String stem;
private final Set<String> terms = new HashSet<String>();
private int frequency = 0;
public Keyword(String stem) {
this.stem = stem;
}
public void add(String term) {
terms.add(term);
frequency++;
}
@Override
public int compareTo(Keyword o) {
// descending order
return Integer.valueOf(o.frequency).compareTo(frequency);
}
@Override
public boolean equals(Object obj) {
if (this == obj) {
return true;
} else if (!(obj instanceof Keyword)) {
return false;
} else {
return stem.equals(((Keyword) obj).stem);
}
}
@Override
public int hashCode() {
return Arrays.hashCode(new Object[] { stem });
}
public String getStem() {
return stem;
}
public Set<String> getTerms() {
return terms;
}
public int getFrequency() {
return frequency;
}
}
实用程序
词干：
public static String stem(String term) throws IOException {
TokenStream tokenStream = null;
try {
// tokenize
tokenStream = new ClassicTokenizer(Version.LUCENE_36, new StringReader(term));
// stem
tokenStream = new PorterStemFilter(tokenStream);
// add each token in a set, so that duplicates are removed
Set<String> stems = new HashSet<String>();
CharTermAttribute token = tokenStream.getAttribute(CharTermAttribute.class);
tokenStream.reset();
while (tokenStream.incrementToken()) {
stems.add(token.toString());
}
// if no stem or 2+ stems have been found, return null
if (stems.size() != 1) {
return null;
}
String stem = stems.iterator().next();
// if the stem has non-alphanumerical chars, return null
if (!stem.matches("[a-zA-Z0-9-]+")) {
return null;
}
return stem;
} finally {
if (tokenStream != null) {
tokenStream.close();
}
}
}
要搜索集合（将由潜在关键字列表使用）：
public static <T> T find(Collection<T> collection, T example) {
for (T element : collection) {
if (element.equals(example)) {
return element;
}
}
collection.add(example);
return example;
}
核
下面是主要的输入法：
public static List<Keyword> guessFromString(String input) throws IOException {
TokenStream tokenStream = null;
try {
// hack to keep dashed words (e.g. "non-specific" rather than "non" and "specific")
input = input.replaceAll("-+", "-0");
// replace any punctuation char but apostrophes and dashes by a space
input = input.replaceAll("[\\p{Punct}&&[^'-]]+", " ");
// replace most common english contractions
input = input.replaceAll("(?:'(?:[tdsm]|[vr]e|ll))+\\b", "");
// tokenize input
tokenStream = new ClassicTokenizer(Version.LUCENE_36, new StringReader(input));
// to lowercase
tokenStream = new LowerCaseFilter(Version.LUCENE_36, tokenStream);
// remove dots from acronyms (and "'s" but already done manually above)
tokenStream = new ClassicFilter(tokenStream);
// convert any char to ASCII
tokenStream = new ASCIIFoldingFilter(tokenStream);
// remove english stop words
tokenStream = new StopFilter(Version.LUCENE_36, tokenStream, EnglishAnalyzer.getDefaultStopSet());
List<Keyword> keywords = new LinkedList<Keyword>();
CharTermAttribute token = tokenStream.getAttribute(CharTermAttribute.class);
tokenStream.reset();
while (tokenStream.incrementToken()) {
String term = token.toString();
// stem each term
String stem = stem(term);
if (stem != null) {
// create the keyword or get the existing one if any
Keyword keyword = find(keywords, new Keyword(stem.replaceAll("-0", "-")));
// add its corresponding initial token
keyword.add(term.replaceAll("-0", "-"));
}
}
// reverse sort by frequency
Collections.sort(keywords);
return keywords;
} finally {
if (tokenStream != null) {
tokenStream.close();
}
}
}
例子
使用Java 维基百科文章介绍部分guessFromString中的方法，这里是找到的前 10 个最常见的关键字（即词干）：
java x12 [java]
compil x5 [compiled, compiler, compilers]
sun x5 [sun]
develop x4 [developed, developers]
languag x3 [languages, language]
implement x3 [implementation, implementations]
applic x3 [application, applications]
run x3 [run]
origin x3 [originally, original]
gnu x3 [gnu]
通过获取集合（在上面的示例中显示在括号中） ，遍历输出列表以了解每个词干的原始找到的单词。terms[...]
下一步是什么
将词干频率/频率总和比率与英语语言统计数据进行比较，如果你能做到的话，让我随时了解：我也可能很感兴趣:)
于 2013-07-03T16:18:21.320 回答
6
上面提出的代码的更新和即用型版本。
此代码与Apache Lucene5.x…6.x 兼容。
CardKeyword类：
import java.util.HashSet;
import java.util.Set;
/**
* Keyword card with stem form, terms dictionary and frequency rank
*/
class CardKeyword implements Comparable<CardKeyword> {
/**
* Stem form of the keyword
*/
private final String stem;
/**
* Terms dictionary
*/
private final Set<String> terms = new HashSet<>();
/**
* Frequency rank
*/
private int frequency;
/**
* Build keyword card with stem form
*
* @param stem
*/
public CardKeyword(String stem) {
this.stem = stem;
}
/**
* Add term to the dictionary and update its frequency rank
*
* @param term
*/
public void add(String term) {
this.terms.add(term);
this.frequency++;
}
/**
* Compare two keywords by frequency rank
*
* @param keyword
* @return int, which contains comparison results
*/
@Override
public int compareTo(CardKeyword keyword) {
return Integer.valueOf(keyword.frequency).compareTo(this.frequency);
}
/**
* Get stem's hashcode
*
* @return int, which contains stem's hashcode
*/
@Override
public int hashCode() {
return this.getStem().hashCode();
}
/**
* Check if two stems are equal
*
* @param o
* @return boolean, true if two stems are equal
*/
@Override
public boolean equals(Object o) {
if (this == o) return true;
if (!(o instanceof CardKeyword)) return false;
CardKeyword that = (CardKeyword) o;
return this.getStem().equals(that.getStem());
}
/**
* Get stem form of keyword
*
* @return String, which contains getStemForm form
*/
public String getStem() {
return this.stem;
}
/**
* Get terms dictionary of the stem
*
* @return Set<String>, which contains set of terms of the getStemForm
*/
public Set<String> getTerms() {
return this.terms;
}
/**
* Get stem frequency rank
*
* @return int, which contains getStemForm frequency
*/
public int getFrequency() {
return this.frequency;
}
}
关键字提取器类：
import org.apache.lucene.analysis.TokenStream;
import org.apache.lucene.analysis.core.LowerCaseFilter;
import org.apache.lucene.analysis.core.StopFilter;
import org.apache.lucene.analysis.en.EnglishAnalyzer;
import org.apache.lucene.analysis.en.PorterStemFilter;
import org.apache.lucene.analysis.miscellaneous.ASCIIFoldingFilter;
import org.apache.lucene.analysis.standard.ClassicFilter;
import org.apache.lucene.analysis.standard.StandardTokenizer;
import org.apache.lucene.analysis.tokenattributes.CharTermAttribute;
import java.io.IOException;
import java.io.StringReader;
import java.util.*;
/**
* Keywords extractor functionality handler
*/
class KeywordsExtractor {
/**
* Get list of keywords with stem form, frequency rank, and terms dictionary
*
* @param fullText
* @return List<CardKeyword>, which contains keywords cards
* @throws IOException
*/
static List<CardKeyword> getKeywordsList(String fullText) throws IOException {
TokenStream tokenStream = null;
try {
// treat the dashed words, don't let separate them during the processing
fullText = fullText.replaceAll("-+", "-0");
// replace any punctuation char but apostrophes and dashes with a space
fullText = fullText.replaceAll("[\\p{Punct}&&[^'-]]+", " ");
// replace most common English contractions
fullText = fullText.replaceAll("(?:'(?:[tdsm]|[vr]e|ll))+\\b", "");
StandardTokenizer stdToken = new StandardTokenizer();
stdToken.setReader(new StringReader(fullText));
tokenStream = new StopFilter(new ASCIIFoldingFilter(new ClassicFilter(new LowerCaseFilter(stdToken))), EnglishAnalyzer.getDefaultStopSet());
tokenStream.reset();
List<CardKeyword> cardKeywords = new LinkedList<>();
CharTermAttribute token = tokenStream.getAttribute(CharTermAttribute.class);
while (tokenStream.incrementToken()) {
String term = token.toString();
String stem = getStemForm(term);
if (stem != null) {
CardKeyword cardKeyword = find(cardKeywords, new CardKeyword(stem.replaceAll("-0", "-")));
// treat the dashed words back, let look them pretty
cardKeyword.add(term.replaceAll("-0", "-"));
}
}
// reverse sort by frequency
Collections.sort(cardKeywords);
return cardKeywords;
} finally {
if (tokenStream != null) {
try {
tokenStream.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
}
/**
* Get stem form of the term
*
* @param term
* @return String, which contains the stemmed form of the term
* @throws IOException
*/
private static String getStemForm(String term) throws IOException {
TokenStream tokenStream = null;
try {
StandardTokenizer stdToken = new StandardTokenizer();
stdToken.setReader(new StringReader(term));
tokenStream = new PorterStemFilter(stdToken);
tokenStream.reset();
// eliminate duplicate tokens by adding them to a set
Set<String> stems = new HashSet<>();
CharTermAttribute token = tokenStream.getAttribute(CharTermAttribute.class);
while (tokenStream.incrementToken()) {
stems.add(token.toString());
}
// if stem form was not found or more than 2 stems have been found, return null
if (stems.size() != 1) {
return null;
}
String stem = stems.iterator().next();
// if the stem form has non-alphanumerical chars, return null
if (!stem.matches("[a-zA-Z0-9-]+")) {
return null;
}
return stem;
} finally {
if (tokenStream != null) {
try {
tokenStream.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
}
/**
* Find sample in collection
*
* @param collection
* @param sample
* @param <T>
* @return <T> T, which contains the found object within collection if exists, otherwise the initially searched object
*/
private static <T> T find(Collection<T> collection, T sample) {
for (T element : collection) {
if (element.equals(sample)) {
return element;
}
}
collection.add(sample);
return sample;
}
}
函数调用：
String text = "…";
List<CardKeyword> keywordsList = KeywordsExtractor.getKeywordsList(text);
于 2016-03-26T16:26:13.040 回答
1
一种基于 RAKE 算法和由rapidrake-java库包装的opennlp 模型的相对简单的方法。
import java.io.IOException;
import java.io.InputStream;
import org.apache.commons.io.IOUtils;
import io.github.crew102.rapidrake.RakeAlgorithm;
import io.github.crew102.rapidrake.model.RakeParams;
import io.github.crew102.rapidrake.model.Result;
public class KeywordExtractor {
private static String delims = "[-,.?():;\"!/]";
private static String posUrl = "model-bin/en-pos-maxent.bin";
private static String sentUrl = "model-bin/en-sent.bin";
public static void main(String[] args) throws IOException {
InputStream stopWordsStream = KeywordExtractor.class.getResourceAsStream("/stopword-list.txt");
String[] stopWords = IOUtils.readLines(stopWordsStream, "UTF-8").toArray(new String[0]);
String[] stopPOS = {"VBD"};
RakeParams params = new RakeParams(stopWords, stopPOS, 0, true, delims);
RakeAlgorithm rakeAlg = new RakeAlgorithm(params, posUrl, sentUrl);
Result aRes = rakeAlg.rake("I'm looking for a Java library to extract keywords from a block of text.");
System.out.println(aRes);
// OUTPUT:
// [looking (1), java library (4), extract keywords (4), block (1), text (1)]
}
}
正如您从示例输出中看到的那样，您会得到一张关键字地图及其相对权重。
如https://github.com/crew102/rapidrake-java所述，您需要从opennlp 下载页面en-pos-maxent.bin下载文件，并将它们放入项目根目录中的文件夹中（如果使用Maven项目结构）。停用词文件应该放在下面（假设 maven 结构），例如可以从https://github.com/terrier-org/terrier-desktop/blob/master/share/stopword-list.txt下载。
en-sent.binmodel-binsrcsrc/main/resources/stopword-list.txt
于 2020-06-30T20:54:38.860 回答