1

How do I implement levenshtein distance on records in a database table using python? I know how to connect python with database, coding in python may not be problem, and I also have the records in a database table. I understand the theory and the dynamic programming of levenshtein distance. The problem here is, how do I write the codes in such a way that after connecting to the database table, I can compare two records having up to three fields and output their similarity score. Below is a snipet of my database table:

Record 1:
Author : Michael I James
Title : Advancement in networking
Journal: ACM

Record 2: Author: Michael J Inse
Title: Advancement in networking
Journal: ACM

Any ideas is welcome. I'm a newbie in this area, please try explain with a little detail. Thanks.

4

1 回答 1

1

我对您的问题的理解是，您确实需要识别可能重复的非常相似的记录。我会在数据库本身解决这个问题。无需编程。如果您的数据库中没有可用的 Levenshtein 函数，您可能需要创建一个用户定义的函数。

以下是 MySQL 的示例：

CREATE FUNCTION `levenshtein`(s1 VARCHAR(255), s2 VARCHAR(255)) RETURNS int(11) DETERMINISTIC  
BEGIN    
  DECLARE s1_len, s2_len, i, j, c, c_temp, cost INT;
  DECLARE s1_char CHAR;    DECLARE cv0, cv1 VARBINARY(256);
  SET s1_len = CHAR_LENGTH(s1), s2_len = CHAR_LENGTH(s2), cv1 = 0x00, j = 1, i = 1, c = 0;
  IF s1 = s2 THEN 
    RETURN 0;
  ELSEIF s1_len = 0 THEN 
    RETURN s2_len;
  ELSEIF s2_len = 0 THEN 
    RETURN s1_len;
  ELSE 
    WHILE j <= s2_len DO 
      SET cv1 = CONCAT(cv1, UNHEX(HEX(j))), j = j + 1; 
    END WHILE; 
    WHILE i <= s1_len DO 
      SET s1_char = SUBSTRING(s1, i, 1), c = i, cv0 = UNHEX(HEX(i)), j = 1; 
      WHILE j <= s2_len DO 
        SET c = c + 1; 
        IF s1_char = SUBSTRING(s2, j, 1) THEN 
          SET cost = 0; 
        ELSE 
          SET cost = 1; 
        END IF; 
        SET c_temp = CONV(HEX(SUBSTRING(cv1, j, 1)), 16, 10) + cost; 
        IF c > c_temp THEN 
          SET c = c_temp; 
        END IF; 
        SET c_temp = CONV(HEX(SUBSTRING(cv1, j+1, 1)), 16, 10) + 1; 
        IF c > c_temp THEN 
          SET c = c_temp; 
        END IF; 
        SET cv0 = CONCAT(cv0, UNHEX(HEX(c))), j = j + 1; 
      END WHILE; 
      SET cv1 = cv0, i = i + 1; 
    END WHILE;
  END IF;
  RETURN c;  
END

然后，您确实需要将所有记录相互比较。这需要一个自我完全连接，这当然可能有点重。如果太重，您将需要采用 Python 方式，这样可以避免重复（比较不同时间的相同记录）。

这就是我要做的。请注意，我宁愿使用 ID 以便于识别：

SELECT a.ID AS IDa,
  b.ID AS IDb,
  a.Author AS AuthorA, 
  b.Author AS AuthorB, 
  ap.levenshtein(a.Author, b.Author) AS Lev_Aut,
  a.Title AS TitleA, b.Title AS TitleB, ap.levenshtein(a.Title, b.Title) AS Lev_Title,
  a.Journal AS JounalA , b.Journal AS JournalB, ap.levenshtein(a.Journal, b.Journal) AS Lev_Journal,
  ap.levenshtein(a.Author, b.Author) + ap.levenshtein(a.Title, b.Title) + ap.levenshtein(a.Journal, b.Journal) AS Composite
FROM test.zzz AS a, test.zzz AS b 
WHERE a.ID != b.ID
ORDER BY 8;

将返回从最佳匹配到最差（复合列）排序的 Levenshtein 值列表。该条件避免将记录与其自身进行比较。

于 2013-06-09T09:41:28.023 回答