0

我有三个 csv 推文数据框，每个推文约 500 万条。以下用于连接它们的代码存在内存不足错误。我的机器有 32GB 内存。如何在 pandas 中为此任务分配更多内存？

df1 = pd.read_csv('tweets.csv')
df2 = pd.read_csv('tweets2.csv')
df3 = pd.read_csv('tweets3.csv')

frames = [df1, df2, df3]
result = pd.concat(frames)

result.to_csv('tweets_combined.csv')

错误是：

$ python concantenate_dataframes.py 
sys:1: DtypeWarning: Columns (0,1,2,3,4,5,6,8,9,10,11,12,13,14,19,22,23,24) have mixed types.Specify dtype option on import or set low_memory=False.
Traceback (most recent call last):
  File "concantenate_dataframes.py", line 19, in <module>
    df2 = pd.read_csv('tweets2.csv')
  File "/home/mona/anaconda3/lib/python3.7/site-packages/pandas/io/parsers.py", line 676, in parser_f
    return _read(filepath_or_buffer, kwds)
  File "/home/mona/anaconda3/lib/python3.7/site-packages/pandas/io/parsers.py", line 454, in _read
    data = parser.read(nrows)
  File "/home/mona/anaconda3/lib/python3.7/site-packages/pandas/io/parsers.py", line 1133, in read
    ret = self._engine.read(nrows)
  File "/home/mona/anaconda3/lib/python3.7/site-packages/pandas/io/parsers.py", line 2037, in read
    data = self._reader.read(nrows)
  File "pandas/_libs/parsers.pyx", line 859, in pandas._libs.parsers.TextReader.read

更新：尝试了答案中的建议，但仍然出现错误

$ python concantenate_dataframes.py 
Traceback (most recent call last):
  File "concantenate_dataframes.py", line 18, in <module>
    df1 = pd.read_csv('tweets.csv', low_memory=False, error_bad_lines=False)
  File "/home/mona/anaconda3/lib/python3.7/site-packages/pandas/io/parsers.py", line 676, in parser_f
    return _read(filepath_or_buffer, kwds)
  File "/home/mona/anaconda3/lib/python3.7/site-packages/pandas/io/parsers.py", line 454, in _read
    data = parser.read(nrows)
  File "/home/mona/anaconda3/lib/python3.7/site-packages/pandas/io/parsers.py", line 1133, in read
    ret = self._engine.read(nrows)
  File "/home/mona/anaconda3/lib/python3.7/site-packages/pandas/io/parsers.py", line 2037, in read
    data = self._reader.read(nrows)
  File "pandas/_libs/parsers.pyx", line 862, in pandas._libs.parsers.TextReader.read
  File "pandas/_libs/parsers.pyx", line 943, in pandas._libs.parsers.TextReader._read_rows
  File "pandas/_libs/parsers.pyx", line 2070, in pandas._libs.parsers.raise_parser_error
pandas.errors.ParserError: Error tokenizing data. C error: Buffer overflow caught - possible malformed input file.

      File "pandas/_libs/parsers.pyx", line 874, in pandas._libs.parsers.TextReader._read_low_memory
      File "pandas/_libs/parsers.pyx", line 928, in pandas._libs.parsers.TextReader._read_rows
      File "pandas/_libs/parsers.pyx", line 915, in pandas._libs.parsers.TextReader._tokenize_rows
      File "pandas/_libs/parsers.pyx", line 2070, in pandas._libs.parsers.raise_parser_error
    pandas.errors.ParserError: Error tokenizing data. C error: Buffer overflow caught - possible malformed input file.

I am running the code on Ubuntu 20.04 OS
4

2 回答 2

0

我认为这是格式错误的数据（某些数据结构不正确tweets2.csv）的问题，因为您可以使用error_bad_lines=False并尝试将引擎从 c 更改为 python，例如engine='python' ex：df2 = pd.read_csv('tweets2.csv', error_bad_lines=False)

或前：df2 = pd.read_csv('tweets2.csv', engine='python')

或者也许是前：df2 = pd.read_csv('tweets2.csv', engine='python', error_bad_lines=False)

但我建议找出那些 revord 并修复它。

而且，如果您想要使用 hacky 的方式来做到这一点，而不是使用

1) https://askubuntu.com/questions/941480/how-to-merge-multiple-files-of-the-same-format-into-a-single-file

2) https://askubuntu.com/questions/656039/concatenate-multiple-files-without-header在此处输入链接描述

于 2020-06-19T05:22:09.147 回答
-1

dtype在导入或设置时指定选项low_memory=False

于 2020-06-19T05:20:25.197 回答