0

我是 python 新手。我想删除重复的单词

除了英文单词，我想删除所有其他单词和空行。

只有我想提取的纯英文单词。

我有一些文本文件，其中包含如下

aaa
bbb
aaa223

aaa
ccc
ddd

kei60:
sj@6999


jack02
jparkj

所以在处理重复之后我想得到以下结果

aaa
bbb
ccc
ddd
jparkj

以下是我尝试过的脚本源。

如果有人帮助我，非常感谢！谢谢！

# read a text file, replace multiple words specified in a dictionary
# write the modified text back to a file

import re

def replace_words(text, word_dic):
    """
    take a text and replace words that match a key in a dictionary with
    the associated value, return the changed text
    """
    rc = re.compile('|'.join(map(re.escape, word_dic)))
    def translate(match):
        return word_dic[match.group(0)]
    return rc.sub(translate, text)



def main():
    test_file = "prxtest.txt"
    # read the file
    fin = open(test_file, "r")
    str2 = fin.read()
    fin.close()

    # the dictionary has target_word:replacement_word pairs
    word_dic = {
    '.': '\n',
    '"': '\n',
    '<': '\n',
    '>': '\n',
    '!': '\n',
    "'": '\n',
    '(': '\n',
    ')': '\n',
    '[': '\n',
    ']': '\n',
    '@': '\n',
    '#': '\n',
    '$': '\n',
    '%': '\n',
    '^': '\n',
    "&": '\n',
    '*': '\n',
    '_': '\n',
    '+': '\n',
    '-': '\n',
    '=': '\n',
    '}': '\n',
    '{': '\n',
    '"': '\n',
    ";": '\n',
    ':': '\n',
    '?': '\n',
    ',': '\n',
    '`': '\n',
    '~': '\n',
    '1': '\n',
    '2': '\n',
    '3': '\n',
    '4': '\n',
    "5": '\n',
    '6': '\n',
    '7': '\n',
    '8': '\n',
    '9': '\n',
    '0': '\n',
    ' ': '\n'}

    # call the function and get the changed text
    str3 = replace_words(str2, word_dic)

    # write changed text back out
    fout = open("clean.txt", "w")
    fout.write(str3)
    fout.close()

if __name__ == "__main__":

    main()
4

5 回答 5

2

这将捕获仅包含字母的行：

fin = open(test_file, 'r')
fout = open('clean.txt', 'w')

s = set()
for line in fin:
    if line.rstrip().isalpha():
        if not line in s:
            s.add(line)
            fout.write(line)

fin.close()
fout.close()
于 2012-06-02T06:50:30.847 回答
1

像这样的东西应该工作：

import re
found = []
with open(test_file) as fd:
   for line in fd:
      word = line.strip()
      if word:
         if word not in found and re.search(r'^[[:alpha:]]+$', word):
            print word
            found.append(word)
于 2012-06-02T06:13:35.880 回答
1

可以分两行完成：

import re

data ="""aaa
bbb
aaa223

aaa
ccc
ddd

kei60:
sj@6999


jack02
jparkj"""

lines = data.splitlines()   # use f.readlines() instead if reading from file

# split the words and only take ones that are all alpha
words = filter(lambda x: re.match('^[^\W\d]+$', x), lines)
# remove duplicates and print out
print '\n'.join(set(words))
于 2012-06-02T07:45:50.727 回答
0

我知道这是一个 python 问题，但你所问的问题似乎更简单，因为它是带有 grep 的 *nix 脚本：

cat infile | grep '^[a-zA-Z]+$' > outfile

如果您只想要仅包含 alpha 字符的唯一行：

cat infile | grep '^[a-zA-Z]+$' | sort -u > outfile

我猜在python中你可以这样做：

import re
inf = open('infile', 'r')
for line in inf:
   if (re.match('\A[a-zA-A]+\Z', line):
      print line
于 2012-06-02T06:31:32.230 回答
0

想要的输出中的某些字符串可能用作感叹词，但其他字符串似乎不是英语单词。如果需要纯英文单词，建议使用稍微复杂的方法：

import nltk
from nltk.corpus import words


tokens = nltk.word_tokenize(open('prxtest.txt').read())
en_words = [x for x in tokens if x.lower() in words.words()]

# en_words now contains purely English words
于 2012-06-02T07:23:00.543 回答