0

I'm trying to run kafka-connect-hdfs using Oozie version: 4.2.0.2.6.5.0-292 via script file sample.sh.
Yes I do know we can run the kafka-hdfs connector directly, but it should happen via oozie.
Kafka has a topic sample and has some data in it.
Trying to push that data to hdfs via oozie.
I have referred a lot of resources before coming here but now luck.

ERROR

Launcher ERROR, reason: Main class [org.apache.oozie.action.hadoop.ShellMain], exit code [1]
2018-07-25 09:54:16,945  INFO ActionEndXCommand:520 - SERVER[nnuat.iot.com] USER[root] GROUP[-] TOKEN[] APP[sample] JOB[0000000-180725094930282-oozie-oozi-W] ACTION[0000000-180725094930282-oozie-oozi-W@shell1] ERROR is considered as FAILED for SLA

I have all the three files inside hdfs and gave permissions to all the files (sample.sh, job.properties, workflow.xml) having all the files inside the location /user/root/sample in hdfs.

Note : Running the oozie in cluster so all the three nodes have the same path and files in it as namenode(/root/oozie-demo) and confluent-kafka(/opt/confluent-4..1.1) too.

job.properties

nameNode=hdfs://171.18.1.192:8020
jobTracker=171.18.1.192:8050
queueName=default
oozie.libpath=${nameNode}/user/oozie/share/lib/lib_20180703063118
oozie.wf.rerun.failnodes=true
oozie.use.system.libpath=true
oozieProjectRoot=${nameNode}/user/${user.name}
oozie.wf.application.path=${nameNode}/user/${user.name}/sample

workflow.xml

<workflow-app xmlns="uri:oozie:workflow:0.3" name="sample">
<start to="shell1"/>
<action name="shell1">
<shell xmlns="uri:oozie:shell-action:0.1">
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<configuration>
<property>
<name>mapred.job.queue.name</name>
<value>${queueName}</value>
</property>
             <property>
                   <name>hadoop.proxyuser.oozie.hosts</name>
                  <value>*</value>
             </property>
             <property>
                   <name>hadoop.proxyuser.oozie.groups</name>
                   <value>*</value>
            </property>
            <property>
                    <name>oozie.launcher.mapreduce.map.java.opts</name>
                   <value>-verbose</value>
            </property>
        </configuration>
    <!--<exec>${myscript}</exec>-->
        <exec>smaple.sh</exec>
         <env-var>HADOOP_USER_NAME=${wf:user()}</env-var>
        <file>hdfs://171.18.1.192:8020/user/root/sample/smaple.sh</file>
         <capture-output/>
    </shell>
    <ok to="end"/>
    <error to="fail"/>
</action>
<kill name="fail">
<message>Shell action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
</kill>
<kill name="fail-output">
    <message>Incorrect output, expected [Hello Oozie] but was [${wf:actionData('shellaction')['my_output']}]</message>
</kill>
<end name="end"/>
</workflow-app>

sample.sh #!/bin/bash

 sudo /opt/confluent-4.1.1/bin/connect-standalone /opt/confluent-4.1.1/etc/schema-registry/connect-avro-standalone.properties /opt/confluent-4.1.1/etc/kafka-connect-hdfs/IOT_DEMO-hdfs.properties

I could not able to find the cause of the Error, I have also tried putting all the jars inside confluent-kafka to oozie/lib directory in hdfs.

link for yarn and oozie error logs.yarn-oozie-error-logs

Thanks!

4

1 回答 1

0

Kafka Connect 旨在完全运行独立进程，而不是通过 Oozie 安排。

它永远不会死，除非发生错误，并且如果 Oozie 重新启动失败的任务，您几乎可以保证在 HDFS 上获得重复的数据，因为除了本地磁盘之外，Connect 偏移量不会永久存储在任何地方（假设 Connect 在单独的机器上重新启动）所以我不明白这一点。

相反，您应该connect-distributed.sh在一组专用机器上作为系统服务独立运行，然后将配置 JSON 发布到 Connect HTTP 端点。然后，任务将作为 Connect 框架的一部分进行分发，并将偏移量持久存储回 Kafka 主题中以实现容错

如果您绝对想使用 Oozie，Confluent 包含 Camus 工具，该工具已被 Connect 弃用，但我一直在维护 Camus+Oozie 流程，它运行良好，只是很难监控失败一旦添加了很多主题。Apache Gobbilin 是该项目的第二次迭代，不由 Confluent 维护

看来您正在运行 HDP，因此 Apache Nifi 应该能够安装在您的集群上以及处理 Kafka 和 HDFS 相关任务

于 2018-07-25T05:45:44.680 回答