python-dedupe - How to understand Dedupe library?

Question

Two questions:

• How to interpret the 'confidence score' when there is cluster with 3 rows and 3 confidence scores (0.98, 0.45, 0.45). Where this confidence scores come from? From logistic regression or somehow from hierarchical clustering?

• 10 000 of my 16 millions is labeled as duplicates, should I put this all as trening data? or only 10 positive and 10 negative will be enough? what number will be better for quality and time of execution?

Answer 1

置信度得分1 - square root of the average squared distance介于记录和集群中的其他记录之间，distance其中1 - predicted probability that a pair of records are coreferent

有关详细信息，请参阅https://docs.dedupe.io/en/latest/API-documentation.html#dedupe.Dedupe.cluster