3

我正在尝试将单词列表与句子列表进行匹配，并使用匹配的单词和句子形成数据框。例如：

words <- c("far better","good","great","sombre","happy")
sentences <- c("This document is far better","This is a great app","The night skies were sombre and starless", "The app is too good and i am happy using it", "This is how it works")

预期结果（一个数据框）如下：

sentences                                               words
This document is far better                               better
This is a great app                                       great
The night skies were sombre and starless                  sombre 
The app is too good and i am happy using it               good, happy
This is how it works                                      -

我正在使用以下代码来实现这一点。

lengthOfData <- nrow(sentence_df)
pos.words <- polarity_table[polarity_table$y>0]$x
neg.words <- polarity_table[polarity_table$y<0]$x
positiveWordsList <- list()
negativeWordsList <- list()
for(i in 1:lengthOfData){
        sentence <- sentence_df[i,]$comment
        #sentence <- gsub('[[:punct:]]', "", sentence)
        #sentence <- gsub('[[:cntrl:]]', "", sentence)
        #sentence <- gsub('\\d+', "", sentence)
        sentence <- tolower(sentence)
        # get  unigrams  from the sentence
        unigrams <- unlist(strsplit(sentence, " ", fixed=TRUE))

        # get bigrams from the sentence
        bigrams <- unlist(lapply(1:length(unigrams)-1, function(i) {paste(unigrams[i],unigrams[i+1])} ))

        # .. and combine into data frame
        words <- c(unigrams, bigrams)
        #if(sentence_df[i,]$ave_sentiment)

        pos.matches <- match(words, pos.words)
        neg.matches <- match(words, neg.words)
        pos.matches <- na.omit(pos.matches)
        neg.matches <- na.omit(neg.matches)
        positiveList <- pos.words[pos.matches]
        negativeList <- neg.words[neg.matches]

        if(length(positiveList)==0){
          positiveList <- c("-")
        }
        if(length(negativeList)==0){
          negativeList <- c("-")
        }
        negativeWordsList[i]<- paste(as.character(unique(negativeList)), collapse=", ")
        positiveWordsList[i]<- paste(as.character(unique(positiveList)), collapse=", ")

        positiveWordsList[i] <- sapply(positiveWordsList[i], function(x) toString(x))
        negativeWordsList[i] <- sapply(negativeWordsList[i], function(x) toString(x))

    }    
positiveWordsList <- as.vector(unlist(positiveWordsList))
negativeWordsList <- as.vector(unlist(negativeWordsList))
scores.df <- data.frame(ave_sentiment=sentence_df$ave_sentiment, comment=sentence_df$comment,pos=positiveWordsList,neg=negativeWordsList, year=sentence_df$year,month=sentence_df$month,stringsAsFactors = FALSE)

我有 28k 个句子和 65k 个单词要匹配。上面的代码需要 45 秒才能完成任务。由于当前方法需要大量时间，因此有关如何提高代码性能的任何建议？

编辑：

我只想得到那些与句子中的单词完全匹配的单词。例如 ：

words <- c('sin','vice','crashes') 
sentences <- ('Since the app crashes frequently, I advice you guys to fix the issue ASAP')

现在对于上述情况，我的输出应该如下：

sentences                                                           words
Since the app crashes frequently, I advice you guys to fix        crahses
the issue ASAP
4

2 回答 2

1

我能够通过一些修改来使用@David Arenburg 的答案。这就是我所做的。我使用以下（大卫建议）来形成数据框。

df <- data.frame(sentences) ; 
df$words <- sapply(sentences, function(x) toString(words[stri_detect_fixed(x, words)]))

上述方法的问题在于它没有进行精确的单词匹配。所以我用下面的方法过滤掉了与句子中的单词不完全匹配的单词。

df <- data.frame(fil=unlist(s),text=rep(df$sentence, sapply(s, FUN=length)))

应用上述行后，输出数据帧更改如下。

sentences                                                      words
This document is far better                                    better
This is a great app                                            great
The night skies were sombre and starless                       sombre 
The app is too good and i am happy using it                    good
The app is too good and i am happy using it                    happy
This is how it works                                            -
Since the app crashes frequently, I advice you guys to fix     
the issue ASAP                                                 crahses
Since the app crashes frequently, I advice you guys to fix     
the issue ASAP                                                 vice
Since the app crashes frequently, I advice you guys to fix     
the issue ASAP                                                 sin

现在将以下过滤器应用于数据框以删除与句子中存在的单词不完全匹配的单词。

df <- df[apply(df, 1, function(x) tolower(x[1]) %in% tolower(unlist(strsplit(x[2], split='\\s+')))),]

现在我生成的数据框将如下所示。

    sentences                                                      words
    This document is far better                                    better
    This is a great app                                            great
    The night skies were sombre and starless                       sombre 
    The app is too good and i am happy using it                    good
    The app is too good and i am happy using it                    happy
    This is how it works                                            -
    Since the app crashes frequently, I advice you guys to fix     
    the issue ASAP                                                 crahses

stri_detect_fixed 大大减少了我的计算时间。剩下的过程并没有占用太多时间。感谢@David 为我指出了正确的方向。

于 2016-10-03T09:03:18.393 回答
0

您可以在最新版本的感测器中使用 来执行此操作，但您extract_sentiment_terms必须先创建一个感测键并为单词赋值：

pos <- c("far better","good","great","sombre","happy")
neg <- c('sin','vice','crashes') 

sentences <- c('Since the app crashes frequently, I advice you guys to fix the issue ASAP',
    "This document is far better", "This is a great app","The night skies were sombre and starless", 
    "The app is too good and i am happy using it", "This is how it works")

library(sentimentr)
(sentkey <- as_key(data.frame(c(pos, neg), c(rep(1, length(pos)), rep(-1, length(neg))), stringsAsFactors = FALSE)))

##             x  y
## 1:    crashes -1
## 2: far better  1
## 3:       good  1
## 4:      great  1
## 5:      happy  1
## 6:        sin -1
## 7:     sombre  1
## 8:       vice -1

extract_sentiment_terms(sentences, sentkey)

##    element_id sentence_id negative   positive
## 1:          1           1  crashes           
## 2:          2           1          far better
## 3:          3           1               great
## 4:          4           1              sombre
## 5:          5           1          good,happy
## 6:          6           1
于 2017-04-12T14:04:12.340 回答