我正在尝试使用 MLCP 和摄取时的基本转换脚本将一堆 JSON 文件加载到 MarkLogic 8 中。
我可以按原样加载文件,我在 ML 中获取 JSON 对象。
我想要的是在摄取时从 JSON 转换为 XML,所以我写了一个基本的转换,如下所示:
xquery version "1.0-ml";
module namespace ingest = "http://dikw.com/ingest/linkedin";
import module namespace json="http://marklogic.com/xdmp/json" at "/MarkLogic/json/json.xqy";
import module namespace sem="http://marklogic.com/semantics" at "/MarkLogic/semantics.xqy";
declare namespace basic="http://marklogic.com/xdmp/json/basic";
declare default function namespace "http://www.w3.org/2005/xpath-functions";
declare option xdmp:mapping "false";
declare function ingest:transform(
$content as map:map,
$context as map:map
) as map:map*
{
let $org-doc := map:get($content, "value")
let $jsonxml := json:transform-from-json($org-doc)
let $name := $jsonxml//basic:full__name
let $_ := xdmp:log(concat('Inserting linkedin profile ', $name, '.xml..'))
let $new-doc :=
document {
<json>{
$jsonxml
}</json>
}
return (
map:put($content, "value", $new-doc),
$content
)
};
现在,如果我使用 MLCP 加载没有转换的文档,它可以工作,但如上所述,我在 ML8 中得到 JSON。(我使用Roxy为 MLCP 调用正确的加载环境。)
./ml $ENV mlcp import -input_file_path content/linkedin -input_file_type documents
以上工作正常。
但是像这样使用变换:
./ml $ENV mlcp import -input_file_path content/linkedin -input_file_type documents -transform_module /ingest/linkedin.xqy -output_collections incoming,incoming/linkedin
我收到一个错误:“错误内容泵。多线程映射器:未知内容类型:json”
15/06/22 17:37:12 INFO contentpump.ContentPump: Hadoop library version: 2.0.0-mr1-cdh4.3.0
15/06/22 17:37:12 INFO contentpump.LocalJobRunner: Content type is set to MIXED. The format of the inserted documents will be determined by the MIME type specification configured on MarkLogic Server.
15/06/22 17:37:12 WARN util.KerberosName: Kerberos krb5 configuration not found, setting default realm to empty
15/06/22 17:37:12 INFO input.FileInputFormat: Total input paths to process : 9
15/06/22 17:37:13 ERROR contentpump.MultithreadedMapper: Unknown content type: json
java.lang.IllegalArgumentException: Unknown content type: json
at com.marklogic.mapreduce.ContentType.forName(ContentType.java:107)
at com.marklogic.contentpump.utilities.TransformHelper.getTransformInsertQry(TransformHelper.java:124)
at com.marklogic.contentpump.TransformWriter.write(TransformWriter.java:97)
at com.marklogic.contentpump.TransformWriter.write(TransformWriter.java:46)
at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:85)
at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:106)
at com.marklogic.contentpump.DocumentMapper.map(DocumentMapper.java:46)
at com.marklogic.contentpump.DocumentMapper.map(DocumentMapper.java:32)
at com.marklogic.contentpump.BaseMapper.runThreadSafe(BaseMapper.java:51)
at com.marklogic.contentpump.MultithreadedMapper$MapRunner.run(MultithreadedMapper.java:376)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
在查询控制台中,事情按预期工作,按预期将 JSON 变量转换为 XML 文档......
我在这里想念什么?
发送
雨果