2

我正在尝试构建一个远程提交火花作业并监视提交作业状态的应用程序。我发现http://arturmkrtchyan.com/apache-spark-hidden-rest-api描述了一个 REST API 来提交作业和获取状态。但是，当我尝试提交作业时，它也会成功提交并返回提交 ID，但在从远程托管 URL 获取 jar 时会引发 NullPointer 异常。

要求：

    curl -X POST http://<sparkmaster-url>:6066/v1/submissions/create --header "Content-Type:application/json;charset=UTF-8" --data '{
        "action" : "CreateSubmissionRequest",
        "appArgs" : [ "prod", 960 ],
        "appResource" : "http://<jenkins_url>/job/myjob/ws/target/scala-2.11/myapp-assembly-1.0.jar",
        "clientSparkVersion" : "2.0.2",
        "mainClass" : "myclass",
        "sparkProperties" : {
          "spark.jars" : "http://<jenkins_url>/job/myjob/ws/target/scala-2.11/myapp-assembly-1.0.jar",
          "spark.app.name" : "myclass",
          "spark.submit.deployMode" : "cluster",
          "spark.master" : "spark://<spark_master_ip>:6066"
        }
      }'

我在其中一个执行者上收到的错误回溯是这样的：

    2017-03-30 13:25:37,495 INFO  [Worker] - Asked to launch driver driver-20170330132537-0040
    2017-03-30 13:25:37,504 INFO  [DriverRunner] - Copying user jar http://<jenkins_url>/job/myjob/ws/target/scala-2.11/myapp-assembly-1.0.jar to /home/ubuntu/spark/work/driver-20170330132537-0040/myapp-assembly-1.0.jar 
    2017-03-30 13:25:37,850 INFO  [Utils] - Fetching http://<jenkins_url>/job/myjob/ws/target/scala-2.11/myapp-assembly-1.0.jar to /home/ubuntu/spark/work/driver-20170330132537-0040/fetchFileTemp1748077866083410926.tmp
    2017-03-30 13:26:00,821 WARN  [Worker] - Driver driver-20170330132537-0040 failed with unrecoverable exception: java.lang.NullPointerException

但是，通过 spark-submit 命令提交时，此特定作业会成功执行。执行者何时输出日志：

    2017-03-30 15:17:02,884 INFO  [Worker] - Asked to launch driver driver-20170330151702-0054
    2017-03-30 15:17:02,893 INFO  [DriverRunner] - Copying user jar  http://<jenkins_url>/job/myjob/ws/target/scala-2.11/myapp-assembly-1.0.jar to /home/ubuntu/spark/work/driver-20170330151702-0054/myapp-assembly-1.0.jar
    2017-03-30 15:17:03,243 INFO  [Utils] - Fetching http://<jenkins_url>/job/myjob/ws/target/scala-2.11/myapp-assembly-1.0.jar to /home/ubuntu/spark/work/driver-20170330151702-0054/fetchFileTemp4438219504638828769.tmp
    2017-03-30 15:17:22,266 INFO  [DriverRunner] - Launch Command: "/usr/lib/jvm/java-8-oracle/bin/java" "-cp" "/home/ubuntu/spark-statsd/target/scala-2.11/spark-statsd-1.0.0.jar:/home/ubuntu/spark/conf/:/home/ubuntu/spark/jars/*" "-Xmx4096M" "-Dspark.kryo.registrator=serializer.CerebroKryoSerializer" "-Dspark.submit.deployMode=cluster" "-Dspark.executor.memory=15G" "-Dspark.shuffle.consolidateFiles=true" "-Dspark.driver.extraClassPath=/home/ubuntu/spark-statsd/target/scala-2.11/spark-statsd-1.0.0.jar" "-Dspark.app.name=myclass" "-Dspark.executor.instances=16" "-Dspark.master=spark://<spark_master_ip>:7077" "-Dspark.executor.cores=4" "-Dspark.serializer=org.apache.spark.serializer.KryoSerializer" "-Dspark.jars=http://<jenkins_url>/job/myjob/ws/target/scala-2.11/myapp-assembly-1.0.jar" "-Dspark.driver.supervise=false" "-Dspark.shuffle.file.buffer=400" "-Dspark.kryoserializer.buffer=256" "-Dspark.driver.memory=4G" "-Dspark.default.parallelism=48" "org.apache.spark.deploy.worker.DriverWrapper" "spark://Worker@<spark_worker>:39704" "/home/ubuntu/spark/work/driver-20170330151700-0053/myapp-assembly-1.0.jar" "myclass" "prod" "960"
    2017-03-30 15:17:24,047 INFO  [Worker] - Asked to launch executor app-20170330151724-0028/1 for myclass$

你能帮我弄清楚我是否在这里遗漏了什么吗？提前感谢您的所有帮助！

4

0 回答 0