1

我正在从头开始创建一个词袋模块。我不确定是否删除标点符号是否是这种方法的最佳做法。考虑句子

I've been "DMX world center" for long time ago.Are u?

问题：对于词袋，我应该考虑

  • 标记DMX（无引号）或"DMX（包括左引号）
  • u（不带问号）或u?（带问号）

简而言之，当我得到不同的单词时，我应该删除所有的标点符号吗？

提前致谢

更新了这是我实现的代码

示例文本 ：ham , im .. On the snowboarding trip. I was wondering if your planning to get everyone together befor we go..a meet and greet kind of affair? Cheers,

   HashSet<String> bagOfWords = new HashSet<String>();
   BufferedReader reader = new BufferedReader(new FileReader(path));
   while (reader.ready()) {
       String msg = reader.readLine().split("\t", 2)[1].toLowerCase(); // I get only the 2nd part. 1st part indicate wether message is spam or ham
       String[] words = msg.split("[\\s+\n.\t!?+,]"); // this is the regex that I've used to split words
       for (String word : words) {
           bagOfWords.add(word);
       }
   }
4

1 回答 1

2

尝试替换您的代码

 while (reader.ready()) {
       String msg = reader.readLine().split("\t", 2)[1].toLowerCase(); // I get only the 2nd part. 1st part indicate wether message is spam or ham
       String[] words = msg.split("[\\s+\n.\t!?+,]"); // this is the regex that I've used to split words
       for (String word : words) {
           bagOfWords.add(word.replaceAll("[!-+.^:,\"?]"," ").trim()); // it removes all sepecial characters what you mentioned
       }
   }
于 2013-09-04T05:31:46.917 回答