2

我有一个四列 CSV 文件，@用作分隔符，例如：

0001 @ fish @ animal @ eats worms

第一列是唯一保证唯一的列。

我需要对第 2、3 和 4 列执行四次排序操作。

首先，第 2 列按字母数字排序。这种排序的重要特征是它必须保证第 2 列中的任何重复条目彼此相邻，例如：

@ a @ @
@ a @ @
@ a @ @
@ a @ @
@ a @ @
@ b @ @
@ b @ @
@ c @ @  
@ c @ @  
@ c @ @  
@ c @ @  
@ c @ @

接下来，在第一个排序中，将行分为两类。第一行是那些不包含单词“arch.”、“var.”、“ver.”、“anci”的行。或“家庭”。第 4 列中的任何位置。第二行（排在后面）是包含这些单词的行，例如：

@ a @ @ Does not have one of those words.
@ a @ @ Does not have one of those words.
@ a @ @ Does not have one of those words.
@ a @ @ Does not have one of those words.
@ a @ @ This sentence contains arch.
@ b @ @ Does not have one of those words.
@ b @ @ Has the word ver.
@ c @ @ Does not have one of those words.
@ c @ @ Does not have one of those words.
@ c @ @ Does not have one of those words.
@ c @ @ This sentence contains var.
@ c @ @ This sentence contains fam.
@ c @ @ This sentence contains fam.

最后，仅在第二次排序的单独类别中排序，将行从“包含第 3 列中重复条目最多”到“包含第 3 列中重复条目最少”，例如：

@ a @ fish @ Does not have one of those words.
@ a @ fish @ Does not have one of those words.
@ a @ fish @ Does not have one of those words.
@ a @ tiger @ Does not have one of those words.
@ a @ bear @ This sentence contains arch.
@ b @ fish @ Does not have one of those words.
@ b @ fish @ Has the word ver.
@ c @ bear @ Does not have one of those words.
@ c @ bear @ Does not have one of those words.
@ c @ fish @ Does not have one of those words.
@ c @ tiger @ This sentence contains var.
@ c @ tiger @ This sentence contains fam.
@ c @ bear @ This sentence contains fam.

如何按第 2 列的字母数字、第 4 列中某些关键字的出现以及第 3 列中最常见的重复到最不常见的重复对文件进行排序？

4

5 回答 5

4

TXR：（http://www.nongnu.org/txr）

@(bind special-words ("arch." "var." "ver." "anci." "fam."))
@(bind ahash @(hash :equal-based))
@(repeat)
@id @@ @alpha @@ @animal @@ @words
@  (rebind words @(split-str words " "))
@  (bind record (id alpha animal words))
@  (do (push record [ahash alpha]))
@(end)
@(bind sorted-rec-groups nil)
@(do
   (defun popularity-sort (recs)
     (let ((histogram [group-reduce (hash)
                                    third (do inc @1)
                                    recs 0]))
      [sort recs > [chain third histogram]]))

   (dohash (key records ahash)
     (let (contains does-not combined)
       (each* ((r records)
               (w [mapcar fourth r]))
         (if (isec w special-words)
           (push r contains)
           (push r does-not)))
       (push (append (popularity-sort does-not)                                 
                     (popularity-sort contains))                                
             sorted-rec-groups)))
   (set sorted-rec-groups [sort sorted-rec-groups :
                                [chain first second]]))
@(output)
@  (repeat)
@    (repeat)
@(rep)@{sorted-rec-groups} @@ @(last)@{sorted-rec-groups " "}@(end)
@    (end)
@  (end)
@(end)

数据：

0001 @ b @ fish @ Does not have one of those words.
0002 @ a @ bear @ Does not have one of those words.
0003 @ b @ bear @ Has the word ver.
0004 @ a @ fish @ Does not have one of those words.
0005 @ c @ bear @ Does not have one of those words.
0006 @ c @ bear @ Does not have one of those words.
0007 @ a @ fish @ Does not have one of those words.
0008 @ c @ fish @ Does not have one of those words.
0009 @ a @ fish @ Does not have one of those words.
0010 @ c @ tiger @ This sentence contains var.
0011 @ c @ bear @ This sentence contains fam.
0012 @ a @ fish @ Does not have one of those words.
0013 @ c @ tiger @ This sentence contains fam.

跑：

$ txr sort.txr data.txt 
0004 @ a @ fish @ Does not have one of those words.
0007 @ a @ fish @ Does not have one of those words.
0009 @ a @ fish @ Does not have one of those words.
0012 @ a @ fish @ Does not have one of those words.
0002 @ a @ bear @ Does not have one of those words.
0001 @ b @ fish @ Does not have one of those words.
0003 @ b @ bear @ Has the word ver.
0005 @ c @ bear @ Does not have one of those words.
0006 @ c @ bear @ Does not have one of those words.
0008 @ c @ fish @ Does not have one of those words.
0010 @ c @ tiger @ This sentence contains var.
0013 @ c @ tiger @ This sentence contains fam.
0011 @ c @ bear @ This sentence contains fam.
于 2012-04-07T01:35:39.680 回答
2

这是您第一个问题的答案，可帮助您入门：

sort data -t "@" -k 2,2 -k 3,4

这个怎么运作：

  • -t 指定对您来说是“@”符号的字段分隔符。
  • -k 2,2 表示对字段二进行排序
  • -k 3,4 表示通过在字段 3 上排序，然后在字段 4 上排序来解决平局
于 2012-04-07T01:37:36.200 回答
1

这是Ruby中的解决方案。

#!/usr/bin/env ruby

class Row

  SEPARATOR = " @ "

  attr_accessor :cols

  def initialize(text)
    @cols = text.chomp.split(SEPARATOR)
    @cols.size == 4 or raise "Expected text to have four columns: #{text}"
    duplicate_increment
  end

  def has_words?
    cols[3]=~/arch\.|var\.|ver\.|anci\.|fam\./ ? true : false
  end

  def to_s
    SEPARATOR + 
      @cols[1,3].join(SEPARATOR) +
      " -- id:#{cols[0]} duplicates:#{duplicate_count}"
  end

  ### Comparison

  def <=>(other)
    other or raise "Expected other to exist"
    cmp = self.cols[1] <=> other.cols[1]
    return cmp if cmp !=0
    cmp = (self.has_words? ? 1 : -1) <=> (other.has_words? ? 1 : -1)
    return cmp if cmp !=0
    other.duplicate_count <=> self.duplicate_count 
  end

  ### Track duplicate entries

  @@duplicate_count = Hash.new{|h,k| h[k]=0}

  def duplicate_key
    [cols[1],has_words?]
  end

  def duplicate_count
    @@duplicate_count[duplicate_key]
  end

  def duplicate_increment
    @@duplicate_count[duplicate_key] += 1
  end

end

### Main

lines = ARGF
rows = lines.map{|line| Row.new(line) }
sorted_rows = rows.sort
sorted_rows.each{|row| puts row }

输入：

0001 @ b @ fish @ text
0002 @ a @ bear @ text
0003 @ b @ bear @ ver.
0004 @ a @ fish @ text
0005 @ c @ bear @ text
0006 @ c @ bear @ text
0007 @ a @ fish @ text
0008 @ c @ fish @ text
0009 @ a @ fish @ text
0010 @ c @ lion @ var.
0011 @ c @ bear @ fam.
0012 @ a @ fish @ text
0013 @ c @ lion @ fam.

输出：

$ cat data.txt | ./sorter.rb 
@ a @ fish @ text -- id:0007 duplicates:5
@ a @ bear @ text -- id:0002 duplicates:5
@ a @ fish @ text -- id:0012 duplicates:5
@ a @ fish @ text -- id:0004 duplicates:5
@ a @ fish @ text -- id:0009 duplicates:5
@ b @ fish @ text -- id:0001 duplicates:1
@ b @ bear @ ver. -- id:0003 duplicates:1
@ c @ bear @ text -- id:0005 duplicates:3
@ c @ fish @ text -- id:0008 duplicates:3
@ c @ bear @ text -- id:0006 duplicates:3
@ c @ lion @ var. -- id:0010 duplicates:3
@ c @ bear @ fam. -- id:0011 duplicates:3
@ c @ lion @ fam. -- id:0013 duplicates:3
于 2012-04-07T02:40:08.290 回答
1

 q

首先，我加载“csv”并将其调整为正确的形状。测试数据在我的计算机上称为“蠕虫”，但因为 q 不使用字符串作为文件名“类型”（以防止例如注入攻击），所以我需要使用hsym来创建“文件名”：

t:flip `id`a`b`c!("SSSS";"@")0:hsym`worms;

然后我研究了哪些“第四字段”条目包含你的一个词。我使用like构建了一个位图，并将其应用于每一行（左），然后是每个模式（右），以获得不存在单词的 0 或其中一个存在的 1：

t:update p:any each c like/:\:("*arch.*";"*var.*";"*ver.*";"*anci.*";"*fam.*") from t;

然后我想找到重复的数量。这只是第2 列 (a)、第 3 列 (b) 和当前类别中的行数：

t:update d:neg count i by a,b,p from t;

最后，我因为我否定了计数，所以我的所有值都“走同样的路”，所以我可以简单地按这三列排序：

`a`p`d xasc t
于 2018-04-08T07:38:10.110 回答
0

这可能对你有用（非常不雅！）：

sed 's/[^@]*@\([^@\]*\)@\([^@]*\)/\1\t\2\t&/;h;s/@/&\n/3;s/.*\n//;/\(arch\|var\|ver\|anci\|fam\)\./!ba;s/.*/1/;bb;:a;s/.*/0/;:b;G;s/\(.\)\n\([^\t]*\)/\2\t\1/' file |
sort | 
tee file1 |
sed 's/\(.*\)\t.*/\1/' |
uniq -c |
sed 's|^\s*\(\S*\) \(.*\t.*\t\(.*\)\)|/^\2/s/\3/\1/|' >file.sed
sed -f file.sed file1 |
sort -k1,2 -k3,3nr |
sed 's/\t/\n/3;s/.*\n//'
1 @ a @ fish @ Does not have one of those words.
2 @ a @ fish @ Does not have one of those words.  
3 @ a @ fish @ Does not have one of those words.
4 @ a @ tiger @ Does not have one of those words.
5 @ a @ bear @ This sentence contains arch.
6 @ b @ fish @ Does not have one of those words.
7 @ b @ fish @ Has the word ver.
8 @ c @ bear @ Does not have one of those words.
9 @ c @ bear @ Does not have one of those words.
10 @ c @ fish @ Does not have one of those words.
11 @ c @ tiger @ This sentence contains var.
12 @ c @ tiger @ This sentence contains fam.
13 @ c @ bear @ This sentence contains fam.

解释：

使排序键包括：

  1. 第二场
  2. 0/1: 0 代表没有arch./var./etc的第4个字段。1 代表有。
  3. 对上述 2 进行排序后，第 3 个字段的计数重复。

该文件最终使用上述键排序，然后删除键。

于 2012-04-07T12:04:06.253 回答