我正在实现用于文本分类的朴素贝叶斯算法。我有大约 1000 个用于培训的文档和 400 个用于测试的文档。我想我已经正确地实施了培训部分,但我在测试部分感到困惑。这是我所做的简要介绍:
在我的训练功能中:
vocabularySize= GetUniqueTermsInCollection();//get all unique terms in the entire collection
spamModelArray[vocabularySize];
nonspamModelArray[vocabularySize];
for each training_file{
class = GetClassLabel(); // 0 for spam or 1 = non-spam
document = GetDocumentID();
counterTotalTrainingDocs ++;
if(class == 0){
counterTotalSpamTrainingDocs++;
}
for each term in document{
freq = GetTermFrequency; // how many times this term appears in this document?
id = GetTermID; // unique id of the term
if(class = 0){ //SPAM
spamModelArray[id]+= freq;
totalNumberofSpamWords++; // total number of terms marked as spam in the training docs
}else{ // NON-SPAM
nonspamModelArray[id]+= freq;
totalNumberofNonSpamWords++; // total number of terms marked as non-spam in the training docs
}
}//for
for i in vocabularySize{
spamModelArray[i] = spamModelArray[i]/totalNumberofSpamWords;
nonspamModelArray[i] = nonspamModelArray[i]/totalNumberofNonSpamWords;
}//for
priorProb = counterTotalSpamTrainingDocs/counterTotalTrainingDocs;// calculate prior probability of the spam documents
}
我想我正确理解并实施了培训部分,但我不确定我能否正确实施测试部分。在这里,我试图检查每个测试文档,并为每个文档计算 logP(spam|d) 和 logP(non-spam|d)。然后我比较这两个数量以确定类别(垃圾邮件/非垃圾邮件)。
在我的测试功能中:
vocabularySize= GetUniqueTermsInCollection;//get all unique terms in the entire collection
for each testing_file:
document = getDocumentID;
logProbabilityofSpam = 0;
logProbabilityofNonSpam = 0;
for each term in document{
freq = GetTermFrequency; // how many times this term appears in this document?
id = GetTermID; // unique id of the term
// logP(w1w2.. wn) = C(wj)∗logP(wj)
logProbabilityofSpam+= freq*log(spamModelArray[id]);
logProbabilityofNonSpam+= freq*log(nonspamModelArray[id]);
}//for
// Now I am calculating the probability of being spam for this document
if (logProbabilityofNonSpam + log(1-priorProb) > logProbabilityofSpam +log(priorProb)) { // argmax[logP(i|ck) + logP(ck)]
newclass = 1; //not spam
}else{
newclass = 0; // spam
}
}//for
我的问题是;我想返回每个类的概率,而不是精确的 1 和 0(垃圾邮件/非垃圾邮件)。我想看看例如 newclass = 0.8684212 所以我可以稍后应用阈值。但我在这里很困惑。如何计算每个文档的概率?我可以使用 logProbabilities 来计算吗?