3

我正在使用优秀的tidytext包来标记几个段落中的句子。例如，我想采取以下段落：

“我完全相信达西先生没有缺陷。他自己拥有它，毫不掩饰。”

并将其标记为两个句子

  1. “我完全相信达西先生没有缺陷。”
  2. “他自己拥有它，毫不掩饰。”

但是，当我使用默认的句子标记器时，tidytext我得到了三个句子。

代码

df <- data_frame(Example_Text = c("I am perfectly convinced by it that Mr. Darcy has no defect. He owns it himself without disguise."))


unnest_tokens(df, input = "Example_Text", output = "Sentence", token = "sentences")

结果

# A tibble: 3 x 1
                              Sentence
                                <chr>
1 i am perfectly convinced by it that mr.
2                    darcy has no defect.
3    he owns it himself without disguise.

什么是tidytext用于标记句子的简单方法，但不会遇到常见缩写（例如“先生”）的问题？或“博士” 被解释为句尾？

4

2 回答 2

5

您可以使用正则表达式作为拆分条件，但不能保证这将包括所有常见的恐怖：

unnest_tokens(df, input = "Example_Text", output = "Sentence", token = "regex",
              pattern = "(?<!\\b\\p{L}r)\\.")

结果：

# A tibble: 2 x 1
                                                     Sentence
                                                        <chr>
1 i am perfectly convinced by it that mr. darcy has no defect
2                         he owns it himself without disguise

当然，您始终可以创建自己的常用标题列表，并基于该列表创建正则表达式：

titles =  c("Mr", "Dr", "Mrs", "Ms", "Sr", "Jr")
regex = paste0("(?<!\\b(", paste(titles, collapse = "|"), "))\\.")
# > regex
# [1] "(?<!\\b(Mr|Dr|Mrs|Ms|Sr|Jr))\\."

unnest_tokens(df, input = "Example_Text", output = "Sentence", token = "regex",
              pattern = regex)
于 2017-11-09T21:28:20.287 回答
4

在确定句子边界时，语料库和quanteda都对缩写进行了特殊处理。以下是使用语料库拆分句子的方法：

library(dplyr)
library(corpus)
df <- data_frame(Example_Text = c("I am perfectly convinced by it that Mr. Darcy has no defect. He owns it himself without disguise."))

text_split(df$Example_Text, "sentences")
##   parent index text                                                         
## 1 1          1 I am perfectly convinced by it that Mr. Darcy has no defect. 
## 2 1          2 He owns it himself without disguise.

如果您想坚持使用unnest_tokens，但想要更详尽的英文缩写列表，您可以遵循@user 的建议，但使用语料库缩写列表（其中大部分取自 Common Locale Data Repository）：

abbrevations_en
##  [1] "A."       "A.D."     "a.m."     "A.M."     "A.S."     "AA."       
##  [7] "AB."      "Abs."     "AD."      "Adj."     "Adv."     "Alt."    
## [13] "Approx."  "Apr."     "Aug."     "B."       "B.V."     "C."      
## [19] "C.F."     "C.O.D."   "Capt."    "Card."    "cf."      "Col."    
## [25] "Comm."    "Conn."    "Cont."    "D."       "D.A."     "D.C."    
## (etc., 155 total)
于 2017-11-10T13:58:00.183 回答