-2

我有 2 个数据集，使用来自 df1 的数据我想使用 4 个条件识别 df2 中的重复数据。

  • 条件：

如果 df1 'Name' 列的一行与 df2 中 'Name' 列的任何行匹配超过 80%

（和）

(df1['Class'] == df2['Class'] (OR) df1['Amt $'] == df2['Amt $'])

（和）

如果 df1 中 'Category' 列的行与 df2 中 'Category' 列的任何行项匹配超过 80%

  • 结果：

如果满足所有条件，则仅保留 df2 中的新数据并删除其他行。

df1

Name    Class   Amt $   Category
Apple      1    5       Fruit
Banana     2    8       Fruit
Cat        3    4       Animal

df2

Index   Name              Class Amt $   Category
    1   Apple is Red       1    5       Fruit
    2   Banana             2    8       fruits
    3   Cat is cute        3    4       animals
    4   Green Apple        1    5       fruis
    5   Banana is Yellow   2    8       fruet
    6   Cat                3    4       anemal
    7   Apple              1    5       anemal
    8   Ripe Banana        2    8       frut
    9   Royal Gala Apple   1    5       Fruit
    10  Cats               3    4       animol
    11  Green Banana       2    8       Fruit
    12  Green Apple        1    5       fruits
    13  White Cat          3    4       Animal
    14  Banana is sweet    2    8       appel
    15  Apple is Red       1    5       fruits
    16  Ginger Cat         3    4       fruits
    17  Cat house          3    4       animals
    18  Royal Gala Apple   1    5       fret
    19  Banana is Yellow   2    8       fruit market
    20  Cat is cute        3    4       anemal
  • 我试过的代码：
for i in df1['Name']:
    for u in df2['Name']:
        for k in df1['Class']:
            for l in df2['Class']:
                for m in df1['Amt $']:
                    for n in df2['Amt $']:
                        for o in df1['Category']:
                            for p in df2['Category']:
                                if SequenceMatcher(None, i, u).ratio() > .8 and k == l and m == n and SequenceMatcher(None, o, p).ratio() > 0.8:
                                    print(i, u)

所需的输出数据框应如下所示：

Name              Class Amt $   Category
Apple is Red        1   5       Fruit
Banana              2   8       fruits
Cat is cute         3   4       animals
Green Apple         1   5       fruis
Banana is Yellow    2   8       fruet
Cat                 3   4       anemal
Ripe Banana         2   8       frut
Royal Gala Apple    1   5       Fruit
Cats                3   4       animol
Green Banana        2   8       Fruit
Green Apple         1   5       fruits
White Cat           3   4       Animal
Apple is Red        1   5       fruits
Cat house           3   4       animals
Banana is Yellow    2   8       fruit market
Cat is cute         3   4       anemal

请帮助我找到最佳解决方案！:)

4

1 回答 1

1

首先，您必须遍历您的两个 dfs 并使用条件匹配并在 df2 中设置一个变量。

df2['match'] = False
for idx2, row2 in df2.iterrows():
    match = False
    for idx1, row1 in df1.iterrows():
        if (SequenceMatcher(None, row1['Name'], row2['Name']).ratio())>=0.8 and \
                (SequenceMatcher(None, row1['Category'], row2['Category']).ratio())>=0.8 and \
                (row1['Class'] == row2['Class'] or row1['Amt $'] == row2['Amt $']):
            match = True
            break
    df2.at[idx2, 'match'] = match

获得匹配项后，您可以从匹配项中删除重复项df2['match']==True。

df2[df2['match']==True].drop_duplicates(keep='first')

接下来，您可以将上述结果与非匹配项连接起来df2['match']==False

df2[df2['match']==False].append(df2[df2['match']==True].drop_duplicates(keep='first'))

在这里，我假设您要删除直接重复项。您要根据条件删除重复项还是直接重复项？

根据您在此处拥有的测试数据集，“Apple”和“Apple is red”的匹配率为 80%。但SequenceMatcher(None, 'Apple', 'Apple is Red').ratio()只给出 0.5882352941176471。同样SequenceMatcher(None, 'Fruit', 'fruits').ratio()只有 0.7272727272727273。你在这里期待别的什么吗？还是预期的结果不对？

无论如何，我希望这能让您对这种方法有所了解。

编辑 1如果你想得到匹配的df1['Name'].

我只重置df2['match']为字符串而不是布尔值并分配df1['Name']给df2['match']而不是分配给True. 然后在最后的 df 中，我df2将具有的行df2['match']==False和不重复的行连接起来df2['match']==True。希望这可以帮助。 

df2['match'] = ''
for idx2, row2 in df2.iterrows():
    match = ''
    for idx1, row1 in df1.iterrows():
        if (SequenceMatcher(None, row1['Name'], row2['Name']).ratio())>=0.5 and \
                (SequenceMatcher(None, row1['Category'], row2['Category']).ratio())>=0.5 and \
                (row1['Class'] == row2['Class'] or row1['Amt $'] == row2['Amt $']):
            match = row1['Name']
            break
    df2.at[idx2, 'match'] = match


print(df2[df2['match']==''].append(df2[df2['match']!=''].drop_duplicates(keep='first')))
于 2020-03-12T07:18:47.587 回答