我目前正在尝试为 lucene 索引中的术语计算 tf-idf 矩阵。我尝试使用以下功能来做到这一点:
public Table<Integer, BytesRef, Double> tfidf(String field) throws IOException, ParseException{
//variables in complete context
int totalNoOfDocs = reader.numDocs(); //total no of docs
HashBasedTable<Integer, BytesRef, Double> tfidfPerDocAndTerm = HashBasedTable.create(); //tfidf value for each document(integer) and term(Byteref) pair.
//variables in loop context
BytesRef term; //term as BytesRef
int noOfDocs; //number of documents (a term occours in)
int tf; //term frequency (of a term in a doc)
double idf; //inverse document frequency (of a term in a doc)
double tfidf; //term frequency - inverse document frequency value (of a term in a doc)
Terms termVector; //all terms of current doc in current field
TermsEnum termsEnum; //iterator for terms
DocsEnum docsEnum; //iterator for documents (of current term)
List<Integer> docIds = getDocIds(totalNoOfDocs); //get internal documentIds of documents
try {
for(int doc : docIds){
termVector = reader.getTermVector(doc, field); //get termvector for document
termsEnum = termVector.iterator(null); //get iterator of termvector to iterate over terms
while((term = termsEnum.next()) != null){ //iterate of terms
noOfDocs = termsEnum.docFreq(); //add no of docs the term occurs in to list
docsEnum = termsEnum.docs(null, null); //get document iterator for this term (all documents the term occours in)
while((doc = docsEnum.nextDoc()) != DocIdSetIterator.NO_MORE_DOCS){ //iterate over documents - computation of all tf-idf values for this term
tf = docsEnum.freq(); //get termfrequency of current term in current doc
idf = Math.log((double)totalNoOfDocs / (double)noOfDocs); //calculate idf
tfidf = (double) tf * idf; //caculate tfidf
tfidfPerDocAndTerm.put(doc, term, tfidf); //add tf-idf value to matrix
}
}
}
} catch (IOException ex) {
Logger.getLogger(Index.class.getName()).log(Level.SEVERE, null, ex);
}
return tfidfPerDocAndTerm;
}
问题是: noOfDocs = termsEnum.docFreq(); 总是返回 1。即使显然存在多个文档中的术语(通过打印“术语”手动检查)。
我还发现,我检索的 docsEnum 是: docsEnum = termsEnum.docs(null, null); 总是只包含 1 个文档(doc 0)。
在创建索引时,我使用了带有停用词列表的标准分析器,因此所有术语都是小写的。
那我的问题是什么?:/
感谢您的帮助!