6

我最近正在研究使用 nltk 从文本中提取关系。所以我建立了一个示例文本：“汤姆是微软的联合创始人。” 并使用以下程序测试并返回任何内容。我不知道为什么。

我正在使用 NLTK 版本：3.2.1，python 版本：3.5.2。

这是我的代码：

import re
import nltk
from nltk.sem.relextract import extract_rels, rtuple
from nltk.tokenize import sent_tokenize, word_tokenize


def test():
    with open('sample.txt', 'r') as f:
        sample = f.read()   # "Tom is the cofounder of Microsoft"

    sentences = sent_tokenize(sample)
    tokenized_sentences = [word_tokenize(sentence) for sentence in sentences]
    tagged_sentences = [nltk.tag.pos_tag(sentence) for sentence in tokenized_sentences]

    OF = re.compile(r'.*\bof\b.*')

    for i, sent in enumerate(tagged_sentences):
        sent = nltk.chunk.ne_chunk(sent) # ne_chunk method expects one tagged sentence
        rels = extract_rels('PER', 'GPE', sent, corpus='ace', pattern=OF, window=10) 
        for rel in rels:
            print('{0:<5}{1}'.format(i, rtuple(rel)))

if __name__ == '__main__':
    test()

1.经过一些调试，如果发现当我将输入更改为

“盖茨于 1955 年 10 月 28 日出生于华盛顿州西雅图。”

nltk.chunk.ne_chunk() 输出为：

(S (PERSON Gates/NNS) 出生于/VBD/VBN 在/IN (GPE Seattle/NNP) ,/, (GPE Washington/NNP) on/IN October/NNP 28/CD ,/, 1955/CD ./。 )

test() 返回：

[PER: 'Gates/NNS'] '是/VBD 出生/VBN 在/IN' [GPE: 'Seattle/NNP']

2.在我将输入更改为：

“盖茨于 1955 年 10 月 28 日出生在西雅图。”

test() 什么都不返回。

3. 我挖掘到nltk/sem/relextract.py发现这很奇怪

输出是由函数： semi_rel2reldict(pairs, window=5, trace=False)引起的，它只在len(pairs) > 2时返回结果，这就是为什么当一个句子少于三个NE时会返回None。

这是一个错误还是我以错误的方式使用了 NLTK？

4

1 回答 1

9

首先，用 来分块 NE ne_chunk，成语看起来像这样

>>> from nltk import ne_chunk, pos_tag, word_tokenize
>>> text = "Tom is the cofounder of Microsoft"
>>> chunked = ne_chunk(pos_tag(word_tokenize(text)))
>>> chunked
Tree('S', [Tree('PERSON', [('Tom', 'NNP')]), ('is', 'VBZ'), ('the', 'DT'), ('cofounder', 'NN'), ('of', 'IN'), Tree('ORGANIZATION', [('Microsoft', 'NNP')])])

（另见https://stackoverflow.com/a/31838373/610569）

接下来我们看一下extract_rels函数。

def extract_rels(subjclass, objclass, doc, corpus='ace', pattern=None, window=10):
    """
    Filter the output of ``semi_rel2reldict`` according to specified NE classes and a filler pattern.
    The parameters ``subjclass`` and ``objclass`` can be used to restrict the
    Named Entities to particular types (any of 'LOCATION', 'ORGANIZATION',
    'PERSON', 'DURATION', 'DATE', 'CARDINAL', 'PERCENT', 'MONEY', 'MEASURE').
    """

当您调用此功能时：

extract_rels('PER', 'GPE', sent, corpus='ace', pattern=OF, window=10)

它依次执行 4 个过程。

1.它检查你的subjclass和objclass是否有效

即https://github.com/nltk/nltk/blob/develop/nltk/sem/relextract.py#L202：

if subjclass and subjclass not in NE_CLASSES[corpus]:
    if _expand(subjclass) in NE_CLASSES[corpus]:
        subjclass = _expand(subjclass)
    else:
        raise ValueError("your value for the subject type has not been recognized: %s" % subjclass)
if objclass and objclass not in NE_CLASSES[corpus]:
    if _expand(objclass) in NE_CLASSES[corpus]:
        objclass = _expand(objclass)
    else:
        raise ValueError("your value for the object type has not been recognized: %s" % objclass)

2. 它从您的 NE 标记输入中提取“对”：

if corpus == 'ace' or corpus == 'conll2002':
    pairs = tree2semi_rel(doc)
elif corpus == 'ieer':
    pairs = tree2semi_rel(doc.text) + tree2semi_rel(doc.headline)
else:
    raise ValueError("corpus type not recognized")

现在让我们看看给定您的输入语句Tom is the cofounder of Microsoft，返回什么tree2semi_rel()：

>>> from nltk.sem.relextract import tree2semi_rel, semi_rel2reldict
>>> from nltk import word_tokenize, pos_tag, ne_chunk
>>> text = "Tom is the cofounder of Microsoft"
>>> chunked = ne_chunk(pos_tag(word_tokenize(text)))
>>> tree2semi_rel(chunked)
[[[], Tree('PERSON', [('Tom', 'NNP')])], [[('is', 'VBZ'), ('the', 'DT'), ('cofounder', 'NN'), ('of', 'IN')], Tree('ORGANIZATION', [('Microsoft', 'NNP')])]]

所以它返回一个包含 2 个列表的列表，第一个内部列表由一个空白列表和Tree包含“PERSON”标签的列表组成。

[[], Tree('PERSON', [('Tom', 'NNP')])]

第二个列表由短语is the cofounder of和Tree包含“组织”的 组成。

让我们继续前进。

3.extract_rel然后尝试将对更改为某种关系字典

reldicts = semi_rel2reldict(pairs)

如果我们semi_rel2reldict用您的例句查看函​​数返回的内容，我们会看到这是空列表返回的地方：

>>> tree2semi_rel(chunked)
[[[], Tree('PERSON', [('Tom', 'NNP')])], [[('is', 'VBZ'), ('the', 'DT'), ('cofounder', 'NN'), ('of', 'IN')], Tree('ORGANIZATION', [('Microsoft', 'NNP')])]]
>>> semi_rel2reldict(tree2semi_rel(chunked))
[]

那么让我们看看semi_rel2reldict https://github.com/nltk/nltk/blob/develop/nltk/sem/relextract.py#L144的代码：

def semi_rel2reldict(pairs, window=5, trace=False):
    """
    Converts the pairs generated by ``tree2semi_rel`` into a 'reldict': a dictionary which
    stores information about the subject and object NEs plus the filler between them.
    Additionally, a left and right context of length =< window are captured (within
    a given input sentence).
    :param pairs: a pair of list(str) and ``Tree``, as generated by
    :param window: a threshold for the number of items to include in the left and right context
    :type window: int
    :return: 'relation' dictionaries whose keys are 'lcon', 'subjclass', 'subjtext', 'subjsym', 'filler', objclass', objtext', 'objsym' and 'rcon'
    :rtype: list(defaultdict)
    """
    result = []
    while len(pairs) > 2:
        reldict = defaultdict(str)
        reldict['lcon'] = _join(pairs[0][0][-window:])
        reldict['subjclass'] = pairs[0][1].label()
        reldict['subjtext'] = _join(pairs[0][1].leaves())
        reldict['subjsym'] = list2sym(pairs[0][1].leaves())
        reldict['filler'] = _join(pairs[1][0])
        reldict['untagged_filler'] = _join(pairs[1][0], untag=True)
        reldict['objclass'] = pairs[1][1].label()
        reldict['objtext'] = _join(pairs[1][1].leaves())
        reldict['objsym'] = list2sym(pairs[1][1].leaves())
        reldict['rcon'] = _join(pairs[2][0][:window])
        if trace:
            print("(%s(%s, %s)" % (reldict['untagged_filler'], reldict['subjclass'], reldict['objclass']))
        result.append(reldict)
        pairs = pairs[1:]
    return result

要做的第一件事semi_rel2reldict()是检查输出的位置有超过 2 个元素tree2semi_rel()，而您的例句没有：

>>> tree2semi_rel(chunked)
[[[], Tree('PERSON', [('Tom', 'NNP')])], [[('is', 'VBZ'), ('the', 'DT'), ('cofounder', 'NN'), ('of', 'IN')], Tree('ORGANIZATION', [('Microsoft', 'NNP')])]]
>>> len(tree2semi_rel(chunked))
2
>>> len(tree2semi_rel(chunked)) > 2
False

啊哈，这就是为什么 theextract_rel什么都不返回。

现在出现的问题是，extract_rel()即使有 2 个元素，如何从 ? 中返回一些东西tree2semi_rel()？这甚至可能吗？

让我们尝试一个不同的句子：

>>> text = "Tom is the cofounder of Microsoft and now he is the founder of Marcohard"
>>> chunked = ne_chunk(pos_tag(word_tokenize(text)))
>>> chunked
Tree('S', [Tree('PERSON', [('Tom', 'NNP')]), ('is', 'VBZ'), ('the', 'DT'), ('cofounder', 'NN'), ('of', 'IN'), Tree('ORGANIZATION', [('Microsoft', 'NNP')]), ('and', 'CC'), ('now', 'RB'), ('he', 'PRP'), ('is', 'VBZ'), ('the', 'DT'), ('founder', 'NN'), ('of', 'IN'), Tree('PERSON', [('Marcohard', 'NNP')])])
>>> tree2semi_rel(chunked)
[[[], Tree('PERSON', [('Tom', 'NNP')])], [[('is', 'VBZ'), ('the', 'DT'), ('cofounder', 'NN'), ('of', 'IN')], Tree('ORGANIZATION', [('Microsoft', 'NNP')])], [[('and', 'CC'), ('now', 'RB'), ('he', 'PRP'), ('is', 'VBZ'), ('the', 'DT'), ('founder', 'NN'), ('of', 'IN')], Tree('PERSON', [('Marcohard', 'NNP')])]]
>>> len(tree2semi_rel(chunked)) > 2
True
>>> semi_rel2reldict(tree2semi_rel(chunked))
[defaultdict(<type 'str'>, {'lcon': '', 'untagged_filler': 'is the cofounder of', 'filler': 'is/VBZ the/DT cofounder/NN of/IN', 'objsym': 'microsoft', 'objclass': 'ORGANIZATION', 'objtext': 'Microsoft/NNP', 'subjsym': 'tom', 'subjclass': 'PERSON', 'rcon': 'and/CC now/RB he/PRP is/VBZ the/DT', 'subjtext': 'Tom/NNP'})]

但这只能确认extract_rel当返回 < 2 对时无法提取tree2semi_rel。如果我们删除 的条件会发生什么while len(pairs) > 2？

为什么我们做不到while len(pairs) > 1？

如果我们仔细查看代码，我们会看到填充 reldict 的最后一行，https ://github.com/nltk/nltk/blob/develop/nltk/sem/relextract.py#L169 ：

reldict['rcon'] = _join(pairs[2][0][:window])

它尝试访问 的第三个元素，pairs如果 的长度pairs为 2，您将得到一个IndexError.

那么，如果我们删除该rcon键并简单地将其更改为 会发生什么while len(pairs) >= 2？

为此，我们必须重写该semi_rel2redict()函数：

>>> from nltk.sem.relextract import _join, list2sym
>>> from collections import defaultdict
>>> def semi_rel2reldict(pairs, window=5, trace=False):
...     """
...     Converts the pairs generated by ``tree2semi_rel`` into a 'reldict': a dictionary which
...     stores information about the subject and object NEs plus the filler between them.
...     Additionally, a left and right context of length =< window are captured (within
...     a given input sentence).
...     :param pairs: a pair of list(str) and ``Tree``, as generated by
...     :param window: a threshold for the number of items to include in the left and right context
...     :type window: int
...     :return: 'relation' dictionaries whose keys are 'lcon', 'subjclass', 'subjtext', 'subjsym', 'filler', objclass', objtext', 'objsym' and 'rcon'
...     :rtype: list(defaultdict)
...     """
...     result = []
...     while len(pairs) >= 2:
...         reldict = defaultdict(str)
...         reldict['lcon'] = _join(pairs[0][0][-window:])
...         reldict['subjclass'] = pairs[0][1].label()
...         reldict['subjtext'] = _join(pairs[0][1].leaves())
...         reldict['subjsym'] = list2sym(pairs[0][1].leaves())
...         reldict['filler'] = _join(pairs[1][0])
...         reldict['untagged_filler'] = _join(pairs[1][0], untag=True)
...         reldict['objclass'] = pairs[1][1].label()
...         reldict['objtext'] = _join(pairs[1][1].leaves())
...         reldict['objsym'] = list2sym(pairs[1][1].leaves())
...         reldict['rcon'] = []
...         if trace:
...             print("(%s(%s, %s)" % (reldict['untagged_filler'], reldict['subjclass'], reldict['objclass']))
...         result.append(reldict)
...         pairs = pairs[1:]
...     return result
... 
>>> text = "Tom is the cofounder of Microsoft"
>>> chunked = ne_chunk(pos_tag(word_tokenize(text)))
>>> tree2semi_rel(chunked)
[[[], Tree('PERSON', [('Tom', 'NNP')])], [[('is', 'VBZ'), ('the', 'DT'), ('cofounder', 'NN'), ('of', 'IN')], Tree('ORGANIZATION', [('Microsoft', 'NNP')])]]
>>> semi_rel2reldict(tree2semi_rel(chunked))
[defaultdict(<type 'str'>, {'lcon': '', 'untagged_filler': 'is the cofounder of', 'filler': 'is/VBZ the/DT cofounder/NN of/IN', 'objsym': 'microsoft', 'objclass': 'ORGANIZATION', 'objtext': 'Microsoft/NNP', 'subjsym': 'tom', 'subjclass': 'PERSON', 'rcon': [], 'subjtext': 'Tom/NNP'})]

啊! 它有效，但仍有第四步extract_rels()。

pattern4. 给定您提供给参数的正则表达式，它执行 reldict 过滤器， https ://github.com/nltk/nltk/blob/develop/nltk/sem/relextract.py#L222 ：

relfilter = lambda x: (x['subjclass'] == subjclass and
                       len(x['filler'].split()) <= window and
                       pattern.match(x['filler']) and
                       x['objclass'] == objclass)

现在让我们试试破解版semi_rel2reldict：

>>> text = "Tom is the cofounder of Microsoft"
>>> chunked = ne_chunk(pos_tag(word_tokenize(text)))
>>> tree2semi_rel(chunked)
[[[], Tree('PERSON', [('Tom', 'NNP')])], [[('is', 'VBZ'), ('the', 'DT'), ('cofounder', 'NN'), ('of', 'IN')], Tree('ORGANIZATION', [('Microsoft', 'NNP')])]]
>>> semi_rel2reldict(tree2semi_rel(chunked))
[defaultdict(<type 'str'>, {'lcon': '', 'untagged_filler': 'is the cofounder of', 'filler': 'is/VBZ the/DT cofounder/NN of/IN', 'objsym': 'microsoft', 'objclass': 'ORGANIZATION', 'objtext': 'Microsoft/NNP', 'subjsym': 'tom', 'subjclass': 'PERSON', 'rcon': [], 'subjtext': 'Tom/NNP'})]
>>> 
>>> pattern = re.compile(r'.*\bof\b.*')
>>> reldicts = semi_rel2reldict(tree2semi_rel(chunked))
>>> relfilter = lambda x: (x['subjclass'] == subjclass and
...                            len(x['filler'].split()) <= window and
...                            pattern.match(x['filler']) and
...                            x['objclass'] == objclass)
>>> relfilter
<function <lambda> at 0x112e591b8>
>>> subjclass = 'PERSON'
>>> objclass = 'ORGANIZATION'
>>> window = 5
>>> list(filter(relfilter, reldicts))
[defaultdict(<type 'str'>, {'lcon': '', 'untagged_filler': 'is the cofounder of', 'filler': 'is/VBZ the/DT cofounder/NN of/IN', 'objsym': 'microsoft', 'objclass': 'ORGANIZATION', 'objtext': 'Microsoft/NNP', 'subjsym': 'tom', 'subjclass': 'PERSON', 'rcon': [], 'subjtext': 'Tom/NNP'})]

有用！现在让我们以元组形式查看它：

>>> from nltk.sem.relextract import rtuple
>>> rels = list(filter(relfilter, reldicts))
>>> for rel in rels:
...     print rtuple(rel)
... 
[PER: 'Tom/NNP'] 'is/VBZ the/DT cofounder/NN of/IN' [ORG: 'Microsoft/NNP']
于 2016-11-09T01:11:02.327 回答