google-cloud-dataflow - 跳过标题行 - Cloud DataFlow 可以吗？

Question

我创建了一个管道，它从 GCS 中的文件读取，对其进行转换，最后写入 BQ 表。该文件包含一个标题行（字段）。

有没有办法像在加载时在 BQ 中那样以编程方式设置“要跳过的标题行数”？

Answer 1

This is not currently possible. It sounds like there are two potential requests here:

• Specifying presence and skip behavior for header lines for a BigQuery import.
• Specifying that a GCS text source should skip a header line.

Future work on this is tracked in https://issues.apache.org/jira/browse/BEAM-123.

Also, in the meantime, you could add a simple filter to your ParDo code to skip headers. Something like this:

PCollection<X> rows = ...;
PCollection<X> nonHeaders =
   rows.apply(Filter.by(new MatchIfNonHeader()));