0

我有两个文本文件，它们都有索引行。我想比较file1和file2并将相似的行发送到一个新的文本文件。我已经在谷歌上搜索了一段时间，并且一直在尝试各种形式的 grep，但我觉得我已经过头了。我最终想要的是查看 file2 中出现在 file1 中的“Mon-######”，并打印 file1 中对应的行。

（文件要大得多，为了简洁起见，我把它们删掉了）

为了更加清晰：

file1 具有以下形式的条目：

Mon-000101  100.27242   9.608597   11.082   10.034
Mon-000102  100.18012   9.520860   12.296   12.223

file2 具有以下形式的条目：

Mon-000101
Mon-000171

因此，如果 file2 中的标识符（例如 Mon-000101）列在 file1 中，我希望以 Mon-000101 开头的整行打印到单独的文件中。如果它未在 file2 中列出，则可以将其丢弃。

因此，如果文件仅与上述文件一样大，则新生成的文件将具有单个条目

Mon-000101  100.27242   9.608597   11.082   10.034

因为这是两者唯一的共同点。

4

6 回答 6

1 
$ join <(sort file1) <(sort file2) > duplicated-lines
于 2013-05-20T03:13:57.810 回答
1

由于您添加了 python 标记，因此您似乎想要这样的东西：

import csv
f = open('file2')
l = set([l.strip() for l in f.readlines()])
with open('file1', 'rb') as csvfile:
    dialect = csv.Sniffer().sniff(csvfile.read(10024))
    csvfile.seek(0)
    reader = csv.reader(csvfile, dialect)
    cnt = 0
    for item in reader:
        if cnt >0:
           data = item[0]
           if data in l:
               print item
        cnt = cnt + 1
于 2013-05-20T03:24:20.137 回答
0

使用grep和。sed_ bash对于非常大的文件，这可能不是很高效。

grep -f <(sed 's/^/^/' file2.txt) file1.txt
于 2013-05-20T15:21:03.437 回答
0

One way to solve this (provided the files aren't too large) would be to read in file1 and store the data as a dict where each line is keyed by the index (first column) and the data (remaining columns). Then read file2 as a list of keys, which you can then use as a generator to extract matching lines from the data in file1.

A quick and dirty solution:

#!/usr/bin/env python

DATA_FILE='file1.txt'
KEY_FILE='file2.txt'

# Read a list of keys to search for
keys = []
lineno = 1
for line in open(KEY_FILE):
    if lineno > 1:
        keys.append(line.strip())
    lineno += 1

# Read data 
data = {}
lineno = 1
for line in open(DATA_FILE):
    if lineno > 1:
        fields = line.split()
        data[fields[0]] = fields[1:]
    lineno += 1

    # Extract data using keys

extracted_data = [[k, data[k]] for k in keys if k in data]

for k, v in extracted_data:
    print k, ' '.join(v)

There's probably more efficient ways of doing this, but this will do the job, and allow you to put more logic in as required.

于 2013-05-20T03:36:48.250 回答
0

既然从前面的问题中您至少对pandas有点熟悉，那么：

import pandas as pd
df1 = pd.read_csv("file1.csv", sep=r"\s+")
df2 = pd.read_csv("file2.csv", sep=r"\s+")
merged = df1.merge(df2.rename_axis({"Mon-id": "NAME"}))
merged.to_csv("merged.csv", index=False)

一些解释（请注意，我已经修改file2.csv，以便有更多的共同元素）。

首先，读取数据：

>>> import pandas as pd
>>> df1 = pd.read_csv("file1.csv", sep=r"\s+")
>>> df2 = pd.read_csv("file2.csv", sep=r"\s+")
>>> df1.head()
         NAME         RA       DEC  Mean_I1  Mean_I2
0  Mon-000101  100.27242  9.608597   11.082   10.034
1  Mon-000102  100.18012  9.520860   12.296   12.223
2  Mon-000103  100.24811  9.586362    9.429    9.010
3  Mon-000104  100.26741  9.867225   11.811   11.797
4  Mon-000105  100.21005  9.814060   12.087   12.090
>>> df2.head()
       Mon-id
0  Mon-000101
1  Mon-000121
2  Mon-000131
3  Mon-000141
4  Mon-000151

然后，我们可以重命名 df2 中的轴：

>>> df2.rename_axis({"Mon-id": "NAME"}).head()
         NAME
0  Mon-000101
1  Mon-000121
2  Mon-000131
3  Mon-000141
4  Mon-000151

之后，merge将简单地做正确的事情：

>>> merged = df1.merge(df2.rename_axis({"Mon-id": "NAME"}))
>>> merged
         NAME         RA       DEC  Mean_I1  Mean_I2
0  Mon-000101  100.27242  9.608597   11.082   10.034
1  Mon-000121  100.45421  9.685027   11.805   11.777
2  Mon-000131  100.20533  9.397307 -100.000   11.764
3  Mon-000141  100.26134  9.388555 -100.000   12.571

最后，我们可以写出来，告诉它不要添加索引列：

>>> merged.to_csv("output.csv", index=False)

生成一个看起来像的文件

NAME,RA,DEC,Mean_I1,Mean_I2
Mon-000101,100.27242,9.608597,11.082,10.034
Mon-000121,100.45421,9.685027,11.805,11.777
Mon-000131,100.20533,9.397307,-100.0,11.764
Mon-000141,100.26134,9.388555,-100.0,12.571
于 2013-05-20T03:29:43.470 回答
0

由于文件可能很大，这种方法怎么样？它使用 sqlite 来处理文件操作：

import sqlite3
import csv
import os

conn = sqlite3.connect('temp.db')

c = conn.cursor()
c.execute('''CREATE TABLE master
          (id text, ra text, dec text, mean1 text, mean2 text)''')
conn.commit() # Write changes

with open('master.csv') as f:
    reader = csv.reader(f, delimiter=',')
    next(reader) # skips header
    for row in reader:
        c.execute('INSERT INTO master VALUES (?,?,?,?,?)', row)
        conn.commit()

with open('filter.txt') as f, open('diff.txt','w') as out:
    writer = csv.writer(out, delimiter=',')
    writer.writerow(('NAME','RA','DEC','Mean_I1','Mean_I2'))
    for line in f:
         c.execute('SELECT * FROM master WHERE id = ?',(line.strip(),))
         row = c.fetchone()
         if row:
             writer.writerow(row)
conn.close()
os.remove('temp.db')
于 2013-05-20T06:07:50.613 回答