0

我正在尝试在我正在做的 YARN 上设置 spark 作业服务器

./bin/server_package.sh ec2

这会在/tmp/jobserver 我尝试时创建一个文件夹./sever_start.sh

我收到此错误

Uncaught error from thread [JobServer-akka.actor.default-dispatcher-2] shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[JobServer]
Uncaught error from thread [JobServer-akka.actor.default-dispatcher-3] shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[JobServer]
java.lang.NoClassDefFoundError: ooyala/common/akka/ActorMetrics$$anonfun$receive$1
        at ooyala.common.akka.ActorMetrics$class.receive(ActorMetrics.scala:20)
        at ooyala.common.akka.InstrumentedActor.receive(InstrumentedActor.scala:8)
        at akka.actor.ActorCell.newActor(ActorCell.scala:558)
        at akka.actor.ActorCell.create(ActorCell.scala:578)
        at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:456)
        at akka.actor.ActorCell.systemInvoke(ActorCell.scala:478)
        at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:263)
        at akka.dispatch.Mailbox.run(Mailbox.scala:219)
        at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
        at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
        at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
        at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
        at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Uncaught error from thread [JobServer-akka.actor.default-dispatcher-5] shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[JobServer]
java.lang.NoClassDefFoundError: ooyala/common/akka/ActorMetrics$$anonfun$receive$1
        at ooyala.common.akka.ActorMetrics$class.receive(ActorMetrics.scala:20)
        at ooyala.common.akka.InstrumentedActor.receive(InstrumentedActor.scala:8)
        at akka.actor.ActorCell.newActor(ActorCell.scala:558)
        at akka.actor.ActorCell.create(ActorCell.scala:578)
        at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:456)
        at akka.actor.ActorCell.systemInvoke(ActorCell.scala:478)
        at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:263)
        at akka.dispatch.Mailbox.run(Mailbox.scala:219)
        at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
        at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
        at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
        at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
        at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
java.lang.NoClassDefFoundError: scala/runtime/AbstractPartialFunction$mcVL$sp
        at java.lang.ClassLoader.defineClass1(Native Method)
        at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
        at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)

试过缩小版的akka​​没有效果有什么建议吗?

4

3 回答 3

1

问题似乎在于使用的 scala 版本冲突。Spark 在 2.11.7 上,而作业服务器在 2.10.4 上

于 2016-01-27T18:39:20.863 回答
0

您是否使用与构建 spark jobserver 的版本相同的 sbt 版本来构建?

于 2016-01-27T17:08:03.923 回答
0

始终尝试匹配 Spark 的 scala 版本和 spark 作业服务器。ec2.sh/local.sh 中的版本应该与您在 spark-shell 中看到的 scala 版本相同。

于 2016-03-16T10:50:38.067 回答