unix - 如何通过忽略特定模式在 Unix 文件中查找唯一行

Question

我在 Unix 中有一个文件，如下所示

">hello"
"hello"
"newuser"
"<newuser"
"newone"

现在我想在文件中找到唯一的匹配项（排除<or>仅在搜索时）和输出为：

">hello"
"<newuser"
"newone"

Answer 1

#!/usr/bin/env python

import sys
seen = set()
for line in sys.stdin:
    word = line.strip().replace('>', '').replace('<', '')
    if word not in seen:
        seen.add(word)
        sys.stdout.write(line)

---

$ ./uniq.py < file1
">hello"
"newuser"
"newone"

Answer 2

$ awk '{ w = $1; sub(/[<>]/, "", w) } word[w] == 0 { word[w]++; print $1 }' file1
">hello"
"newuser"
"newone"

Answer 3

这是 Ruby 中的关联数组思想。

2.0.0p195 :005 > entries= [">hello", "hello", "newuser", "<newuser", "newone"]
 => [">hello", "hello", "newuser", "<newuser", "newone"] 
2.0.0p195 :006 > entries.reduce({}) { |hash, entry| hash[entry.sub(/[<>]/,'')]=entry; hash}.values
 => ["hello", "<newuser", "newone"]