azure-data-lake - How do I ignore invalid rows in U-SQL EXTRACT that don't fit schema?

Question

I'm trying to extract some data from a CSV file using the following U-SQL EXTRACT statement:

EXTRACT SessionId   string,
        Latitude    double,
        Longitude   double,
        Timestamp   int
FROM "wasb://sessions@myaccount.blob.core.windows.net/"
USING Extractors.Csv();

But my job is failing halfway through because there is a row that doesn't fit this schema (common in huge datasets) because it has the wrong number of columns. How do I avoid that this fails the entire extract?

Answer 1

请注意，静默标志将执行以下操作：

1. 忽略列数不匹配的行
2. 如果列类型可以为空，则用 null 替换无效值。

如果出现以下情况，它仍然会出错：

1. 该值不能强制转换为预期的不可为空的类型。
2. 指定编码的字符无效。

Answer 2

使用 Extractors.Csv() ala 的 slient:true 参数：

EXTRACT SessionId   string,
        Latitude    double,
        Longitude   double,
        Timestamp   int
FROM "wasb://sessions@myaccount.blob.core.windows.net/"
USING Extractors.Csv(silent:true);