1

我的bq load命令和错误

$ bq load --replace --source_format=CSV --autodetect temp.BQTable ./source.csv
Upload complete.
Waiting on bqjob_123 ... (0s) Current status: DONE
BigQuery error in load operation: Error processing job 'my-project-id-123:bqjob_123': Error
while reading data, error message: Error detected while parsing row starting at position: 333. Error: Missing close
double quote (") character.
Failure details:
- Error while reading data, error message: CSV processing encountered
too many errors, giving up. Rows: 0; errors: 1; max bad: 0; error
percent: 0

Excel 和我的用于 Csv 序列化的 node.js 模块可以很好地处理带有换行符的值的数据。

Papaparse，我的用于序列化的 npm 模块，'\r\n'用作分隔行的默认换行符，因此那里不应该有任何歧义。

这是一些示例 csv 数据，即 source.csv

id,name
1,"this value
includes a newline"

bq 和 Cloud SDK 版本

$ gcloud components update


Your current Cloud SDK version is: 289.0.0
You will be upgraded to version: 337.0.0

┌─────────────────────────────────────────────────────────────────────────────┐
│                      These components will be updated.                      │
├─────────────────────────────────────────────────────┬────────────┬──────────┤
│                         Name                        │  Version   │   Size   │
├─────────────────────────────────────────────────────┼────────────┼──────────┤
│ BigQuery Command Line Tool                          │     2.0.67 │  < 1 MiB │
│ BigQuery Command Line Tool (Platform Specific)      │     2.0.65 │  < 1 MiB │
│ Cloud SDK Core Libraries                            │ 2021.04.16 │ 18.0 MiB │
│ Cloud SDK Core Libraries (Platform Specific)        │ 2021.03.12 │  < 1 MiB │
│ Cloud Storage Command Line Tool                     │       4.61 │  3.9 MiB │
│ Cloud Storage Command Line Tool (Platform Specific) │       4.59 │  < 1 MiB │
│ gcloud cli dependencies                             │ 2021.04.16 │ 10.8 MiB │
│ gcloud cli dependencies                             │ 2021.04.16 │  < 1 MiB │
└─────────────────────────────────────────────────────┴────────────┴──────────┘

我使用的是 289 版，因为他们修补了较新的版本，不允许您使用bq load:(将大小为 100 MB 或更大的文件（CSV、JSON 等）导入 BQ

4

1 回答 1

0

Remove all newline and carriage return characters from all string values in csv.

Javascript

const _ = require('lodash');
const rows = [{ id: 1, name: `this value
  includes a newline` 
}];

rows.forEach(row => {
  return _.mapValues(row, value => typeof value === 'string'
    ? value.replace(/[\r\n]+/g, ' ')
    : value
  );
});

This isn't a good solution since it discards data and assumes those newlines weren't important. Let's hope there aren't newlines in your PK's or foreign fields.

于 2021-04-23T00:19:50.760 回答