我正在尝试使用 Pig 和 Twitter 的大象鸟库解析一个相当简单的 json 文件，但它变成了一个非常痛苦的调试过程。
json具有以下结构：
oid_id: (oid:chararray),
bookmarks: {(
oid_id:(oid:chararray),
id:chararray,
creator: chararray,
position:chararray,
creationdate:($ate:chararray)
)},
lastaction:(date:chararray),
settings:(preferredlanguage:chararray),
userid:chararray
行示例：
{"oid_id":{"oid":"573239f905474a686e2333f0"},"bookmarks":[{"id":"LEGONINX106W0079264","creator":"player","position":96,"creationdate":{"date ":"2016-12-26T09:37:36.916Z"},"oid_id":{"oid":"5860e4e0ca6baf9032edc0d0"}},{"id":"ONEPERCENTMW0128677","创作者":"玩家","位置":0.08,"creationdate":{"date":"2018-12-18T15:42:33.956Z"},"oid_id":{"oid":"5c191569faf8474953758930"}}],"lastaction":{"date ":"2018-12-18T15:42:28.107Z"},"settings":{"preferredlanguage":"vf","preferredvideoquality":"hd"},"userid":"ocs_32a6ad6dd242d5e3842f9211fd236723_1461773211"}
这是我的代码（受本教程启发： https ://acadgild.com/blog/determining-popular-hashtags-in-twitter-using-pig ）
register /path/to/json-simple-1.1.1.jar
register /path/to/elephant-bird-core-4.17.jar
register /path/to/elephant-bird-pig-4.17.jar
register /path/to/elephant-bird-hadoop-compat-4.17.jar
define JsonLoaderEB com.twitter.elephantbird.pig.load.JsonLoader;
A = LOAD 'file.json' USING JsonLoaderEB('-nestedLoad=true') as myMap;
describe A;
input_table: { myMap: bytearray }
B = foreach A generate flatten(myMap#'bookmarks') as (bookmark:map[]);
describe B;
B：{书签：地图[]}
当我们转储上述关系时，我们可以看到所有数据都已成功加载。
([{"oid_id":{"oid":"5860e4e0ca6baf9032edc0d0"},"creator":"player","creationdate":{"date":"2016-12-26T09:37:36.916Z"},"id ":"LEGONINX106W0079264","position":96},{"oid_id":{"oid":"5c191569faf8474953758930"},"creator":"player","creationdate":{"date":"2018-12- 18T15:42:33.956Z"},"id":"ONEPERCENTMW0128677","position":0.08}])
现在我们从书签中提取创建日期、创建者、id 和位置。
C = foreach B generate bookmark#'creationdate' as date_fact, bookmark#'creator' as creator, bookmark#'id' as id, bookmark#'position' as position;
C：{ date_fact：bytearray，创建者：bytearray，id：bytearray，位置：bytearray }
转储表会出现以下错误：
猪堆栈跟踪
ERROR 1066: Unable to open iterator for alias C. Backend error : Vertex failed, vertexName=scope-41, vertexId=vertex_1542613138136_6721 88_2_00, diagnostics=[Task failed, taskId=task_1542613138136_672188_2_00_000000, diagnostics=[TaskAttempt=0 错误运行任务时出错（失败）：attempt_1542613138136_672188_2_00_000000_0:org.apache.pig.backend.executionengine.ExecException: ERROR 0: Exception while execution (Name: C: Store(hdfs://sandbox/tmp/temp-1543074195/tmp277240455:org .apache.pig.impl.io.InterStorage) - sc ope-40 Operator Key: scope-40): org.apache.pig.backend.executionengine.ExecException: ERROR 0: Exception while execution [POMapLookUp ( Name: POMapLookUp [bytearray ] - scope-28 Operator Key: scope-28) children: null at [null[4,31]]]: java.lang.ClassCastException: java.lan g.String 无法转换为 java.util。org.apache.pig.backend.hadoop.executionengine.tez.plan.operator.POStoreTez.getNextTuple(POStoreTez) 的 org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.processInput(PhysicalOperator.java:315) .java:123) 在 org.apache.pig.backend.hadoop.executionengine.tez.executionengine.tez.runtime.PigProcessor.runPipeline(PigProcessor.java:376) 在 org.apache.pig.backend.hadoop.executionengine.tez.runtime.PigProcessor .run(PigProcessor.java:241) 在 org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:370) 在 org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)在 org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61) 在 java.security.AccessController.doPrivileged(Native Method) 在 javax.security.auth.Subject.doAs(Subject.java:422 ) 在 org.apache.hadoop。security.UserGroupInformation.doAs(UserGroupInformation.java:1698) at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61) at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable. java:37) 在 org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36) 在 java.util.concurrent.FutureTask.run(FutureTask.java:266) 在 java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1149) 在 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 在 java.lang.Thread.run(Thread.java:748) 引起：org.apache.pig。 backend.executionengine.ExecException：错误0：执行[POMapLookUp [byt earray]时出现异常（名称：POMapLookUp [字节数组]-范围28运算符键：范围28）子项：[null [4,31]]]处为空：java.lang .ClassCastException：java.lang。字符串不能在 org.apache.pig.backend.hadoop.executionengine.physicalLayer 的 org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.getNext(PhysicalOperator.java:364) 的 java.util.Map 中转换.relationalOperators.POForEach.processPlan(POForEach.java:406) 在 org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.getNextTuple(POForEach.java:323) 在 org.apache.pig.backend.hadoop .executionengine.physicalLayer.PhysicalOperator.processInput(PhysicalOperator.java:305) 1,9Toporg.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.processInput(PhysicalOperator.java:305) 1,9Top 上的 relationalOperators.POForEach.getNextTuple(POForEach.java:323)org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.processInput(PhysicalOperator.java:305) 1,9Top 上的 relationalOperators.POForEach.getNextTuple(POForEach.java:323)