1

我有 CSV 文件，其中数据的格式如下：

文件 1.csv

ID,NAME
001,Jhon
002,Doe

fille2.csv

ID,SCHOOLS_ATTENDED
001,my Nice School
002,His lovely school

文件 3.csv

ID,SALARY
001,25
002,40

ID字段是一种用于获取记录的主键。

读取 3 到 4 个文件并获取相应数据并存储在另一个具有标题（ID、NAME、SCHOOLS_ATTENDED、SALARY）的 CSV 文件中的最有效方法是什么？

文件大小为数百 MB（100、200 Mb）。

4

2 回答 2

3

数百兆字节并不多。为什么不采用简单的方法使用csv模块和collections.defaultdict：

import csv
from collections import defaultdict

result = defaultdict(dict)
fieldnames = {"ID"}

for csvfile in ("file1.csv", "file2.csv", "file3.csv"):
    with open(csvfile, newline="") as infile:
        reader = csv.DictReader(infile)
        for row in reader:
            id = row.pop("ID")
            for key in row:
                fieldnames.add(key) # wasteful, but I don't care enough
                result[id][key] = row[key]

结果defaultdict如下所示：

>>> result
defaultdict(<type 'dict'>, 
{'001': {'SALARY': '25', 'SCHOOLS_ATTENDED': 'my Nice School', 'NAME': 'Jhon'},
'002': {'SALARY': '40', 'SCHOOLS_ATTENDED': 'His lovely school', 'NAME': 'Doe'}})

然后，您可以将其合并到一个 CSV 文件中（这不是我最漂亮的作品，但现在已经足够好了）：

with open("out.csv", "w", newline="") as outfile:
    writer = csv.DictWriter(outfile, sorted(fieldnames))
    writer.writeheader()
    for item in result:
        result[item]["ID"] = item
        writer.writerow(result[item])

out.csv然后包含

ID,NAME,SALARY,SCHOOLS_ATTENDED
001,Jhon,25,my Nice School
002,Doe,40,His lovely school
于 2013-07-23T09:19:53.963 回答
0

以下是将多个 csv 文件与名称中的特定关键字组合成 1 个最终 csv 文件的工作代码。我已将默认关键字设置为“文件”，但如果您想合并文件夹路径中的所有 csv 文件，您可以将其设置为空白。此代码将从您的第一个 csv 文件中获取标题，并将其用作最终组合 csv 文件中的标题。它将忽略所有其他 csv 文件的标题。

import glob,os
@staticmethod
def Combine_multiple_csv_files_thatContainsKeywordInTheirNames_into_one_csv_file(folder_path,keyword='file'):
    #takes header only from 1st csv, all other csv headers are skipped and data is appened to final csv

    fileNames = glob.glob(folder_path + "*" + keyword + "*"+".csv")  # fileNames INCLUDES FOLDER_PATH TOO
    with open(folder_path+"Combined_csv.csv", "w", newline='') as fout:
        print('Combining multiple csv files into 1')
        csv_write_file = csv.writer(fout, delimiter=',')
        # a.writerows(op)
        with open(fileNames[0], mode='rt')  as read_file:  # utf8
            csv_read_file = csv.reader(read_file, delimiter=',')  # CSVREADER READS FILE AS 1 LIST PER ROW. SO WHEN WRITIN TO ANOTHER  CSV FILE WITH FUNCTION WRITEROWS, IT INTRODUCES ANOTHER NEW LINE '\N' CHARACTER. SO TO AVOID DOUBLE NEWLINES , WE SET NEWLINE AS '' WHEN WE OPEN CSV WRITER OBJECT
            csv_write_file.writerows(csv_read_file)

        for num in range(1, len(fileNames)):
            with open(fileNames[num], mode='rt')  as read_file:  # utf8
                csv_read_file = csv.reader(read_file, delimiter=',')  # CSVREADER READS FILE AS 1 LIST PER ROW. SO WHEN WRITIN TO ANOTHER  CSV FILE WITH FUNCTION WRITEROWS, IT INTRODUCES ANOTHER NEW LINE '\N' CHARACTER. SO TO AVOID DOUBLE NEWLINES , WE SET NEWLINE AS '' WHEN WE OPEN CSV WRITER OBJECT
                next(csv_read_file) # ignore header
                csv_write_file.writerows(csv_read_file)
于 2017-01-02T11:01:37.880 回答