0

我在使用特定的 UDF jar 时遇到问题,到目前为止还无法弄清楚从哪里开始。我有一种从命令行测试 jar 的方法,它可以工作。如果我在 pig 脚本中注册 jar,pig 无法为作业创建 jar。我可以毫无困难地注册其他罐子,而且这个罐子直到几天前都在工作。这是运行 pig 脚本时的输出:

[michael@hadoop01 logitech-correlation]$ pig -f MatchWithClassifier.pig -param date=20130301 -param siteId=0
2013-05-10 11:20:30,523 [main] INFO  org.apache.pig.Main - Apache Pig version 0.10.0-cdh4.1.2 (rexported) compiled Nov 01 2012, 18:38:58
2013-05-10 11:20:30,524 [main] INFO  org.apache.pig.Main - Logging error messages to: /home/michael/correlation/pig_1368210030521.log
2013-05-10 11:20:30,981 [main] INFO  org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: hdfs://hadoop01/
2013-05-10 11:20:31,346 [main] INFO  org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to map-reduce job tracker at: hadoop01.dev.terapeak.com:8021
2013-05-10 11:20:32,143 [main] INFO  org.apache.pig.tools.pigstats.ScriptState - Pig features used in the script: FILTER
2013-05-10 11:20:32,390 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic? false
2013-05-10 11:20:32,422 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size before optimization: 1
2013-05-10 11:20:32,422 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size after optimization: 1
2013-05-10 11:20:32,508 [main] INFO  org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added to the job
2013-05-10 11:20:32,518 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
2013-05-10 11:20:32,522 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - creating jar file Job5623238576559565298.jar
2013-05-10 11:20:36,398 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 2017: Internal error creating job configuration.
Details at logfile: /home/michael/correlation/pig_1368210030521.log

堆栈跟踪如下:

Pig Stack Trace
---------------
ERROR 2017: Internal error creating job configuration.

org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobCreationException: ERROR 2017: Internal error creating job configuration.
  at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getJob(JobControlCompiler.java:727)
  at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.compile(JobControlCompiler.java:259)
  at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:180)
  at org.apache.pig.PigServer.launchPlan(PigServer.java:1275)
  at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1260)
  at org.apache.pig.PigServer.execute(PigServer.java:1250)
  at org.apache.pig.PigServer.executeBatch(PigServer.java:362)
  at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:132)
  at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:193)
  at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:165)
  at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84)
  at org.apache.pig.Main.run(Main.java:430)
  at org.apache.pig.Main.main(Main.java:111)
  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:616)
  at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
Caused by: java.util.zip.ZipException: invalid distance too far back
  at java.util.zip.InflaterInputStream.read(InflaterInputStream.java:164)
  at java.util.zip.ZipInputStream.read(ZipInputStream.java:163)
  at java.util.jar.JarInputStream.read(JarInputStream.java:194)
  at java.io.FilterInputStream.read(FilterInputStream.java:107)
  at org.apache.pig.impl.util.JarManager.addStream(JarManager.java:242)
  at org.apache.pig.impl.util.JarManager.mergeJar(JarManager.java:216)
  at org.apache.pig.impl.util.JarManager.mergeJar(JarManager.java:206)
  at org.apache.pig.impl.util.JarManager.createJar(JarManager.java:126)
  at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getJob(JobControlCompiler.java:411)
  ... 17 more
================================================================================

基于此,我认为问题出在“java.util.zip.ZipException:无效距离太远”异常。猪读罐子有问题吗?

4

0 回答 0