0

我在 Scala ApI 中运行 Livy 客户端应用程序，其中 Livy 为 0.3.0，Spark 版本在 Yarn 模型上为 1.6.0。

这是主要代码。 

def main(args: Array[String]): Unit = {
var scalaClient: LivyScalaClient = null
val livyurl = "http://172.24.4.110:8998"
try {
  scalaClient = new LivyClientBuilder(false).setURI(new URI(livyurl)).build().asScalaClient

 // scalaClient.uploadFile(new File("logs/spath2.jar"))

  val sFuture : ScalaJobHandle[_]= scalaClient.submit(jct => {
    //Spark_Log()
    val conf1 = jct.sc.getConf
    conf1.setAppName("by livy ... ")
    val sc = jct.sc
    val sqlContext = jct.sqlctx
    val dataset: DataFrame = sqlContext.createDataFrame(Seq(
      (1, Vectors.dense(0.0, 0.0, 0.0)),
      (2, Vectors.dense(0.1, 0.1, 0.1)),
      (3, Vectors.dense(0.2, 0.2, 0.2)),
      (4, Vectors.dense(9.0, 9.0, 9.0)),
      (5, Vectors.dense(9.1, 9.1, 9.1)),
      (6, Vectors.dense(9.2, 9.2, 9.2))
    )).toDF("id", "features")

    dataset.write.save("usr/root/livy")
  })
}


finally {
  if (scalaClient != null) {
    scalaClient.stop(true)
    scalaClient = null;
  }
}

和 livy 日志中的异常：

17/08/24 05:50:09 INFO InteractiveSession$: Creating LivyClient for sessionId: 30
17/08/24 05:50:09 WARN RSCConf: Your hostname, master-172-24-4-110.neucloud-bd.org, resolves to a loopback address, but we couldn't find any external IP address!
17/08/24 05:50:09 WARN RSCConf: Set livy.rsc.rpc.server.address if you need to bind to another address.
17/08/24 05:50:09 INFO InteractiveSessionManager: Registering new session 30
17/08/24 05:50:10 INFO InteractiveSession: Stopping InteractiveSession 30...
17/08/24 05:50:10 INFO RSCClient: Failing pending job 5efb67ea-1b76-4b80-a7b1-4f2527bf023b due to shutdown.
17/08/24 05:50:10 WARN ContextLauncher: Waiting thread interrupted, killing child process.
17/08/24 05:50:10 INFO InteractiveSession: Stopped InteractiveSession 30.
17/08/24 05:50:10 ERROR RSCClient: Failed to connect to context.
java.util.concurrent.CancellationException
17/08/24 05:50:10 WARN InteractiveSession: (Fail to get rsc uri,java.util.concurrent.ExecutionException: java.util.concurrent.CancellationException)
17/08/24 05:51:09 ERROR SparkYarnApp: Error whiling refreshing YARN state: java.lang.Exception: No YARN application is found with tag livy-session-30-lkjckxwd in 60 seconds. Please check your cluster status, it is may be very busy.

但是使用Java API可以正常工作

4

1 回答 1

0

可能您应该使用“主机名”而不是“IP”。

于 2017-09-04T10:50:06.470 回答