2

我正在使用 Maven 的FailSafe插件和Maven-Docker-Plugin来实现 spark 作业的自动化集成测试。

我的项目在这里签入

https://github.com/abhitechdojo/maven-scala-spark

我启动 spark 作业的 docker 文件如下所示

FROM cloudera/quickstart

MAINTAINER abhishek "http://www.foobar.com"

# COPY THE UBER JAR IN THE / OF THE FILE SYSTEM

ADD /SparkIntegrationTestsAssembly.jar /

# CREATE INPUT DIRECTORY
CMD ["hadoop", "fs", "-mkdir", "-p", "/input"]

# COPY INPUT FILES
CMD ["hadoop", "fs", "-put", "/twitter.avro", "/input/twitter.avro"]

EXPOSE 8020:8020
EXPOSE 50070:50070
EXPOSE 50010:50010
EXPOSE 50020:50020
EXPOSE 50075:50075
EXPOSE 8030:8030
EXPOSE 8031:8031
EXPOSE 8032:8032
EXPOSE 8033:8033
EXPOSE 8088:8088
EXPOSE 8040:8040
EXPOSE 8042:8042
EXPOSE 10020:10020
EXPOSE 19888:19888
EXPOSE 11000:11000
EXPOSE 8888:8888
EXPOSE 18080:18080
EXPOSE 7077:7077

# RUN ALL SERVICES
ENTRYPOINT ["/usr/bin/docker-quickstart"]

# RUN SPARK JOB
CMD ["spark-submit", "--class", "com.abhi.HelloWorld", "--master", "local[2]", "SparkIntegrationTestsAssembly.jar", "/input", "/output"]

但是当我尝试基于这个图像启动一个容器时......它只是继续说

[INFO] Waiting for container 'cloudera/quickstart' to finish startup (max 10000 sec.)
[INFO] Waiting for container 'cloudera/quickstart' to finish startup (max 10000 sec.)
[INFO] Waiting for container 'cloudera/quickstart' to finish startup (max 10000 sec.)
[INFO] Waiting for container 'cloudera/quickstart' to finish startup (max 10000 sec.)
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 07:50 min
[INFO] Finished at: 2016-04-15T01:45:02-05:00
[INFO] Final Memory: 37M/831M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal net.wouterdanes.docker:docker-maven-plugin:5.0.0:start-containers (start) on project SparkIntegrationTest: Execution start of goal net.wouterdanes.docker:docker-maven-plugin:5.0.0:start-containers failed: java.net.SocketException: Socket closed -> [Help 1]

我的 POM.xml 太大，无法在此处发布，但让我在这里提供链接

https://github.com/abhitechdojo/maven-scala-spark/blob/master/pom.xml

我希望 maven-docker-plugin 会根据我的 dockerfile 运行图像。该文件将通过入口点启动所有服务，然后使用 spark-submit 运行 spark 作业。该插件将等待看到“停止火花上下文”行。

看到这条线后，将运行集成测试，这将验证 spark 作业生成的数据。

但不知何故，插件只是永远等待。我尝试将超时增加到 2000，但仍然是相同的行为。

4

0 回答 0