python - 余弦相似度 [Python]

Question

使用我的函数的以下代码计算查询与数据的余弦相似度：

def rank_retrieve(self, query):
        """
        Given a query (a list of words), return a rank-ordered list of
        documents and score for the query.
        self.docs : list of documents
        self.docs[i] : list of words in doc number i -> [word1,word2,...,wordN]
        self.boolean_retrieve(query) : giving a list of words this return the index of
        documents wich contains all of these words.
        self.tfidf(word,documentIndex) : returns the value tfidf of a word in a document
        self.get_posting(word): returns a list of document index where that word appears
        """
    scores = [0.0 for xx in range(len(self.docs))]

    # Apply Cosine Similarity
    for i in self.boolean_retrieve(query):
        normDoc = 0.0
        normQuery = 0.0
        dt = 0.0
        qtdt = 0.0
        for word in query:
            dt = self.get_tfidf(word,i)
            normDoc+= math.pow(dt,2)

            qt = 1.0 + ( math.log10( len(query) ) )
            normQuery+=math.pow(qt,2)

            qtdt += dt * qt
        scores[i] = qtdt / ( math.sqrt(normDoc) )

    return scores

我只有下一个理论：

那么，你能帮我写代码吗？我返回错误的值，我不知道为什么。谢谢。

doc 56 评分结果：

Cosine Similarity Test
doc 56, query :   ['separ', 'of', 'church', 'and', 'state']

Separ: 
QTDT:  0.105587429399 
DT 0.0621479067488 
QT 1.69897000434 
NormDoc:  0.00386236231326  normQuery 2.88649907563

Of:
QTDT:  0.105587429399 
DT 0.0 
QT 1.69897000434 
NormDoc:  0.00386236231326  normQuery 5.77299815127

Church :
QTDT:  0.653857934128 
DT 0.322707583613 
QT 1.69897000434 
NormDoc:  0.108002546834  normQuery 8.6594972269

And:
QTDT:  0.653857934128 
DT 0.0 
QT 1.69897000434 
NormDoc:  0.108002546834  normQuery 11.5459963025

State:
QTDT:  0.674927180008 
DT 0.0124011876763 
QT 1.69897000434 
NormDoc:  0.10815633629  normQuery 14.4324953782

Scores of 56 must be 0.010676611271744128 found : 2.05225316563

Answer 1

您是否包括了红色条款？参考解决方案是否包含它们？

它们对于获得相同的排名不是必需的，但如果我没记错的话，它们应该对数值产生影响。

1+log此外，由于使用了这些术语，您对余弦相似度的定义似乎是非标准的。

https://en.wikipedia.org/wiki/Cosine_similarity

Answer 2

def calctfidfvec(tfvec, withidf):
    tfidfvec = {}
    veclen = 0.0

    for token in tfvec:
        if withidf:
            tfidf = (1+log10(tfvec[token])) * getidf(token)
        else:
            tfidf = (1+log10(tfvec[token]))
        tfidfvec[token] = tfidf 
        veclen += pow(tfidf,2)

    if veclen > 0:
        for token in tfvec: 
            tfidfvec[token] /= sqrt(veclen)

    return tfidfvec

def cosinesim(vec1, vec2):
    commonterms = set(vec1).intersection(vec2)
    sim = 0.0
    for token in commonterms:
        sim += vec1[token]*vec2[token]

    return sim