nlp - 如何在没有标记化的情况下在拥抱脸中使用变压器？

Question

我有以下代码：

from transformers import AutoTokenizer, AutoModelForTokenClassification, pipeline
tokenizer = AutoTokenizer.from_pretrained("sagorsarker/codeswitch-spaeng-lid-lince")
model = AutoModelForTokenClassification.from_pretrained("sagorsarker/codeswitch-spaeng-lid-lince")
pipeline = pipeline('ner', model=model, tokenizer=tokenizer)
sentence = "some example sentence here"
results = pipeline(sentence)

这很好用。但不是 a str，我不想传递 a listof 令牌。我怎么做？

我想这样做的原因是，我的句子已经被标记化并且简单" ".join()并不能正确地重现句子。例如，isn't已被标记为is和n't。但是一个简单的" ".join()会产生is n't

Answer 1

我假设原始数据由 NLTK 标记，所以尝试NLTK detokenizer：

from nltk.tokenize.treebank import TreebankWordDetokenizer
toks = ['hello', ',', 'i', 'ca', "n't", 'feel', 'my', 'feet', '!', 'Help', '!', '!']
twd = TreebankWordDetokenizer()
twd.detokenize(toks)
# "hello, i can't feel my feet! Help!!"