0

我需要为 NER 任务预处理 XML 文件，并且我正在努力转换 XML 文件。我想有一种很好且简单的方法可以解决以下问题。

给定具有以下结构的 XML 中的注释文本作为输入：

<doc>
   Some <tag1>annotated text</tag1> in <tag2>XML</tag2>.
</doc>

我想要一个 IOB2 标记格式的 CoNLL 文件，如下所示：

Some          O
annotated     B-TAG1
text          I-TAG1
in            O
XML           B-TAG2
.             O
4

2 回答 2

2

让我们将您的 XML 文件转换为 TXT（称为“read.txt”），如下所示：

<doc>
   Some <tag1>annotated text</tag1> in <tag2>Tag2 entity</tag2> <tag1>tag1 entity</tag1>.
   Some <tag3>annotated text</tag3> in <tag2>XML</tag2>!
</doc>

然后使用正则表达式和几个 if-else 条件，下面的代码根据需要返回 CONNL 格式的“output.txt”文件。

import re

sentences, connl = [], []

with open('read.txt', 'r', encoding='utf-8') as file:
    for line in file:
        line = line.strip()
        if line not in ['<doc>', '</doc>']:
            sentences.append(line)

for sentence in sentences:
    tag1 = re.findall(r'<tag1>(.+?)</tag1>', sentence)
    tag2 = re.findall(r'<tag2>(.+?)</tag2>', sentence)
    tag3 = re.findall(r'<tag3>(.+?)</tag3>', sentence)
    splitted = re.split('<tag1>|</tag1>|<tag2>|</tag2>|<tag3>|</tag3>', sentence)  # splitted considering tags
    if tag1 or tag2 or tag3:  # if any tag in sentence
        for split in splitted:  # search each index
            if split in tag1:
                counter = 0
                for token in split.split():
                    if counter > 0:
                        connl.append(token + ' I-TAG1')
                    else:
                        connl.append(token + ' B-TAG1')
                    counter += 1

            elif split in tag2:
                counter = 0
                for token in split.split():
                    if counter > 0:
                        connl.append(token + ' I-TAG2')
                    else:
                        connl.append(token + ' B-TAG2')
                    counter += 1

            elif split in tag3:
                counter = 0
                for token in split.split():
                    if counter > 0:
                        connl.append(token + ' I-TAG3')
                    else:
                        connl.append(token + ' B-TAG3')
                    counter += 1

            else:  # current word is not an entity
                for token in split.split():
                    connl.append(token + ' O')

    else:  # if no entity in sentence
        for word in sentence.split():
            connl.append(word + ' O')

    connl.append('')

with open('output.txt', 'w', encoding='utf-8') as output:
    for element in connl:
        output.write(element + "\n")

输出.txt：

Some O
annotated B-TAG1
text I-TAG1
in O
XML B-TAG2
other B-TAG1
tag I-TAG1
. O

Some O
annotated B-TAG3
text I-TAG3
in O
XML B-TAG2
! O
于 2021-12-24T22:32:41.373 回答
0

或者，您可以使用 XML 解析器，例如lxmlorBeautifulSoup和 spaCy 作为分词器。

pip install lxml
pip install spacy
python3 -m spacy download en_core_web_sm

一个例子：

from lxml import etree
from io import StringIO
import re
import spacy

DISABLED = [
    "ner", "tok2vec", "tagger", "parser", "attribute_ruler", "lemmatizer"]

def type_to_iob(enttype, idx):
    mapping = {
        "tag1": "TAG1",
        "tag2": "TAG2",
        "tag3": "TAG3",
    }
    iob = 'B' if idx == 0 else 'I'
    return '{}_{}'.format(iob, mapping.get(enttype))

def transform_to_iob(item):
    tokens = list(nlp(item.text, disable=DISABLED))
    return [
        (ent, type_to_iob(item.tag, idx))
        for idx, ent in enumerate(tokens)
    ]

xmltext = """<doc>
   Some example of <tag1>annotated text</tag1> in <tag2>XML</tag2>.
   Some other sample of <tag3>another annotated text</tag3> in <tag2>XML</tag2>!
</doc>"""

nlp = spacy.load("en_core_web_sm")
nlp.add_pipe('sentencizer')

tree = etree.parse(StringIO(xmltext))
for item in tree.getroot().xpath('/doc/node()'):
    if isinstance(item, etree._ElementUnicodeResult):
        doc = nlp(str(item).replace("\n", "").strip(), disable=DISABLED)
        for sentence in doc.sents:
            for token in sentence:
                if re.match(r'\s*$', str(token)):
                    print()
                    continue
                print(f"{token} O")
    elif isinstance(item, etree._Element):
        for iob_tag in transform_to_iob(item):
            print(f'{iob_tag[0]} {iob_tag[1]}')

结果：

❯ python3 test.py
Some O
example O
of O
annotated B_TAG1
text I_TAG1
in O
XML B_TAG2
. O

Some O
other O
sample O
of O
another B_TAG3
annotated I_TAG3
text I_TAG3
in O
XML B_TAG2
! O
于 2022-02-17T14:08:27.683 回答