0

我试图使用spark-testing-base对我的 spark 流代码运行单元测试。而且我在运行他们的示例代码时遇到了麻烦。

这是我复制的代码片段

进口 com.holdenkarau.spark.testing.SharedSparkContext 进口 org.scalatest.FunSuite

class SampleTest extends FunSuite with SharedSparkContext {
  test("test initializing spark context") {
    val list = List(1, 2, 3, 4)
    val rdd = sc.parallelize(list)

    assert(rdd.count === list.length)
  }
}

这是错误的堆栈跟踪。

18/10/19 02:08:01 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

An exception or error caused a run to abort. 
java.lang.ExceptionInInitializerError
    at org.apache.spark.SparkContext.withScope(SparkContext.scala:701)
    at org.apache.spark.SparkContext.parallelize(SparkContext.scala:718)
    at com.myproject.SampleTest$$anonfun$1.apply(DStreamTransformSpec.scala:11)
    at com.myproject.analytic.SampleTest$$anonfun$1.apply(DStreamTransformSpec.scala:9)
    at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)
    at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)

我包括我的 build.sbt 不确定这是否有帮助。

version := "0.1"

scalaVersion := "2.11.12"

libraryDependencies += "org.apache.spark" %% "spark-streaming" % "2.2.0"
libraryDependencies += "org.apache.bahir" %% "spark-streaming-pubsub" % "2.2.0"
libraryDependencies += "com.typesafe.play" %% "play-json" % "2.6.7"
libraryDependencies += "com.google.cloud" % "google-cloud-datastore" % "1.40.0"
// For test
libraryDependencies += "org.scalactic" %% "scalactic" % "3.0.5"
libraryDependencies += "org.scalatest" %% "scalatest" % "3.0.5" % "test"
libraryDependencies += "com.holdenkarau" %% "spark-testing-base" % "2.2.0_0.10.0" % "test"

fork in Test := true
javaOptions ++= Seq("-Xms512M", "-Xmx2048M", "-XX:MaxPermSize=2048M", "-XX:+CMSClassUnloadingEnabled")

assemblyMergeStrategy in assembly := {
  case PathList("META-INF", xs @ _*) => MergeStrategy.discard
  case x => MergeStrategy.first
}

assemblyShadeRules in assembly := Seq(
  ShadeRule.rename("com.google.common.**" -> "repackaged.com.google.common.@1").inAll,
  ShadeRule.rename("com.google.protobuf.**" -> "repackaged.com.google.protobuf.@1").inAll
)

是否有任何其他工具，建议使用 DStream 进行测试？

4

1 回答 1

0

也许你已经忘记了？我在 build.sbt 中没有看到

parallelExecution in Test := false
于 2018-10-18T19:10:57.967 回答