3

Netezza 外部表的间歇性问题。

外部表因系统本身生成的文件而失败（意味着由外部表生成的文件不是来自其他来源的文件）。但是我们尝试通过 nzload 实用程序将相同的文件加载到另一个表，并且没有任何问题。这个问题不一致，并且大多数时候无法重现。

CREATE EXTERNAL TABLE SP_PORTFOLIO_EXT_DATA_6128_140
(
    CLIENT_ID INTEGER,
    CONFIG_ID INTEGER,
    SCENARIO_ID INTEGER,
    PORTFOLIO_ID INTEGER,
    PORTFOLIO_NAME CHARACTER VARYING(200),
    CUSTOM13 CHARACTER VARYING(600),
    CUSTOM12 CHARACTER VARYING(500),
    CUSTOM11 CHARACTER VARYING(500),
    CUSTOM10 CHARACTER VARYING(500),
    CUSTOM9 CHARACTER VARYING(500),
    CUSTOM8 CHARACTER VARYING(500),
    CUSTOM7 CHARACTER VARYING(500),
    CUSTOM6 CHARACTER VARYING(2000),
    CUSTOM3 CHARACTER VARYING(500),
    CUSTOM2 CHARACTER VARYING(3000),
    CUSTOM1 CHARACTER VARYING(500),
    CREATIVE CHARACTER VARYING(512),
    PLACEMENT CHARACTER VARYING(5000),
    IMPRESSIONS NUMERIC(38,0),
    CLICKS NUMERIC(38,0),
    CONVERSIONS INTEGER,
    TRUE_CONVERSIONS NUMERIC(38,6),
    OPTMETRIC NUMERIC(38,6),
    LASTAD_OPTMETRIC NUMERIC(38,6),
    CURRSPEND NUMERIC(38,6)
)
USING
(
    DATAOBJECT('/san5/Netezza/CAR/CAR_ZEUS/SPBU/test/SP_PORTFOLIO_EXT_DATA_6128_140.csv')
    DELIMITER 254
    ESCAPECHAR '/'
    TIMESTYLE '24HOUR'
    LOGDIR '/tmp'
    Y2BASE 2000
    ENCODING 'internal'
);

命令成功完成

select COUNT(*) from SP_PORTFOLIO_EXT_DATA_6128_140;
ERROR [HY000] ERROR:  External Table : count of bad input rows reached maxerrors limit

NZLOAD 方法

CREATE TABLE TEST_LOAD
(
    CLIENT_ID INTEGER,
    CONFIG_ID INTEGER,
    SCENARIO_ID INTEGER,
    PORTFOLIO_ID INTEGER,
    PORTFOLIO_NAME CHARACTER VARYING(200),
    CUSTOM13 CHARACTER VARYING(600),
    CUSTOM12 CHARACTER VARYING(500),
    CUSTOM11 CHARACTER VARYING(500),
    CUSTOM10 CHARACTER VARYING(500),
    CUSTOM9 CHARACTER VARYING(500),
    CUSTOM8 CHARACTER VARYING(500),
    CUSTOM7 CHARACTER VARYING(500),
    CUSTOM6 CHARACTER VARYING(2000),
    CUSTOM3 CHARACTER VARYING(500),
    CUSTOM2 CHARACTER VARYING(3000),
    CUSTOM1 CHARACTER VARYING(500),
    CREATIVE CHARACTER VARYING(512),
    PLACEMENT CHARACTER VARYING(5000),
    IMPRESSIONS NUMERIC(38,0),
    CLICKS NUMERIC(38,0),
    CONVERSIONS INTEGER,
    TRUE_CONVERSIONS NUMERIC(38,6),
    OPTMETRIC NUMERIC(38,6),
    LASTAD_OPTMETRIC NUMERIC(38,6),
    CURRSPEND NUMERIC(38,6)
)
DISTRIBUTE ON RANDOM;

# Loading data  from the same file using Nzload

nzload -host 10.200.29.30 -u xxxxx -pw xxxxx -db SPBU_REPORT_DB_TEST -t test_load -delim 254 -ctrlChars  -df /san5/Netezza/CAR/CAR_ZEUS/SPBU/test/SP_PORTFOLIO_EXT_DATA_6128_140.csv

Load session of table 'TEST_LOAD' completed successfully

[ja.prod@inet11026 ~]$ cat /san5/Netezza/CAR/CAR_ZEUS/SPBU/test/SP_PORTFOLIO_EXT_DATA_6128_140.csv|wc -l
191322

select count(*) from test_load;
191322

添加 nzlog

File Buffer Size (MB): 8                  Load Replay Region (MB): 0
  Encoding:              INTERNAL           Max errors:            1
  Skip records:          0                  Max rows:              0
  FillRecord:            No                 Truncate String:       No
  Escape Char:           '/'                Accept Control Chars:  No
  Allow CR in string:    No                 Ignore Zero:           No
  Quoted data:           NO                 Require Quotes:        No

  BoolStyle:             1_0                Decimal Delimiter:     '.'

  Disable NFC:           No
  Date Style:            YMD                Date Delim:            '-'
  Y2Base:                2000
  Time Style:            24HOUR             Time Delim:            ':'
  Time extra zeros:      No

Found bad records

bad #: input row #(byte offset to last char examined) [field #, declaration] diagnostic, "text consumed"[last char examined]
----------------------------------------------------------------------------------------------------------------------------
1: 25(184) [21, INT4] expected field delimiter or end of record, "0"[.]

Statistics

  number of records read:      25
  number of bad records:       1
  -------------------------------------------------
  number of records loaded:    0

  Elapsed Time (sec): 0.0

-----------------------------------------------------------------------------
Load completed at: 08-Oct-15 09:59:04 EDT

包含坏行的 .nzbad 数据（管道符号代表实际的分隔符以提高可读性）：

140|1305|6128||NULL|SEO|SEO|test.com/vehicledetail/detail/632888199/overview|SEO|SEO|SEO|SEO Brand|SEO Brand|best Tracking|Google(Seo)|SEO|Impression Tracker|Unknown|0|1|0|0.000000|0.000000|0.000000|0.000000
4

3 回答 3

1

从 nzlog 我们可以看出第 25 行加载失败。具体来说，当它尝试加载第 21 列时，它遇到了一个不是整数的值。

日志显示它遇到一个 0，然后是一个句点。所以数据可能有 0.0 或 0.1234 之类的东西，不能作为整数加载。

bad #: input row #(byte offset to last char examined) [field #, declaration] diagnostic, "text consumed"[last char examined]
----------------------------------------------------------------------------------------------------------------------------
1: 25(184) [21, INT4] expected field delimiter or end of record, "0"[.]

使用您提供的 .nzbad 数据（出于可读性目的，此处使用 '|' 而不是您的实际分隔符）：

140|1305|6128||NULL|SEO|SEO|test.com/vehicledetail/detail/632888199/overview|SEO|SEO|SEO|SEO Brand|SEO Brand|best Tracking|Google(Seo)|SEO|Impression Tracker|Unknown|0|1|0|0.000000|0.000000|0.000000|0.000000

我在这里注意到的一件事是你有一个 varchar 字段，其中包含“/”。外部表和 nzload 方法之间的区别之一是外部表指定了 escapechar '/' 而 nzload 没有。

您会发现您的数据“test.com/vehicledetail/detail/632888199/overview”将被加载为“test.comvehicledetaildetail632888199overview”，因为“/”字符将被删除，因为它们本身没有转义（例如“//”） .

如果“/”直接位于数据中的列分隔符之前，它将指示它将列分隔符视为数据的一部分，并认为数据中的第 22 列实际上是表中匹配的第 21 列我们在这里看到的。

于 2015-10-08T16:47:22.883 回答
0

数据类型必须更改如下： CHARACTER VARYING 替换为 VARCHAR/ NVARCHAR

CREATE TABLE TEST_LOAD
(
CLIENT_ID INTEGER,
CONFIG_ID INTEGER,
SCENARIO_ID INTEGER,
PORTFOLIO_ID INTEGER,
PORTFOLIO_NAME VARCHAR(200),
CUSTOM13 VARCHAR(600),
CUSTOM12 VARCHAR(500),
CUSTOM11 VARCHAR(500),
CUSTOM10 VARCHAR(500),
CUSTOM9 VARCHAR(500),
CUSTOM8 VARCHAR(500),
CUSTOM7 VARCHAR(500),
CUSTOM6 VARCHAR(2000),
CUSTOM3 VARCHAR(500),
CUSTOM2 VARCHAR(3000),
CUSTOM1 VARCHAR(500),
CREATIVE VARCHAR(512),
PLACEMENT VARCHAR(5000),
IMPRESSIONS NUMERIC(38,0),
CLICKS NUMERIC(38,0),
CONVERSIONS INTEGER,
TRUE_CONVERSIONS NUMERIC(38,6),
OPTMETRIC NUMERIC(38,6),
LASTAD_OPTMETRIC NUMERIC(38,6),
CURRSPEND NUMERIC(38,6)
)
DISTRIBUTE ON RANDOM;
于 2020-04-30T06:02:12.453 回答
0

正如你所说的 ScottMcG 我比较了 Nzload 和 External 表生成的 nzlog 文件，发现转义字符是唯一的区别。所以我注释掉了那部分并再次尝试并且事情正常。

CREATE EXTERNAL TABLE SP_PORTFOLIO_EXT_DATA_6128_140

( CLIENT_ID INTEGER, CONFIG_ID INTEGER, SCENARIO_ID INTEGER, PORTFOLIO_ID INTEGER, PORTFOLIO_NAME CHARACTER VARYING(200), CUSTOM13 CHARACTER VARYING(600), CUSTOM12 CHARACTER VARYING(500), CUSTOM11 CHARACTER VARYING(500), CUSTOM10 CHARACTER VARYING(TER500),可变（500），定制8 可变（500），定制7 可变（500），定制6 可变（2000），定制3 可变（500），定制2 可变（3000），定制1 可变（500），创意可变(512), PLACEMENT CHARACTER VARYING(5000), IMPRESSIONS NUMERIC(38,0), CLICKS NUMERIC(38,0), CONVERSIONS INTEGER, TRUE_CONVERSIONS NUMERIC(38,6), OPTMETRIC NUMERIC(38,6), LASTAD_OPTMETRIC NUMERIC(38) ,6), 当前数字 (38,6) ) 使用 ( DATAOBJECT('/san5/Netezza/CAR/CAR_ZEUS/SPBU/test/SP_PORTFOLIO_EXT_DATA_6128_140.csv') DELIMITER 254 TIMESTYLE '24HOUR' LOGDIR '/tmp' Y2BASE 2000 ENCODING 'internal' );

从 SP_PORTFOLIO_EXT_DATA_6128_140 中选择计数（*）；

191322。

于 2015-10-08T17:42:58.280 回答