0

我正在尝试在 Amazon EMR 集群（AMI 版本 3.2.0/Hadoop 版本 2.4.0）上安装 sqoop 2（版本 1.99.3）。当我启动 sqoop 服务器时，我在 localhost.log 中看到了这个错误：

Sep 10, 2014 4:55:56 PM org.apache.catalina.core.StandardContext listenerStart
SEVERE: Exception sending context initialized event to listener instance of class org.apache.sqoop.server.ServerInitializer
java.lang.RuntimeException: Failure in server initialization
    at org.apache.sqoop.core.SqoopServer.initialize(SqoopServer.java:57)
    at org.apache.sqoop.server.ServerInitializer.contextInitialized(ServerInitializer.java:36)
    at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4206)
    at org.apache.catalina.core.StandardContext.start(StandardContext.java:4705)
    at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:799)
    at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:779)
    at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:601)
    at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:943)
    at org.apache.catalina.startup.HostConfig.deployWARs(HostConfig.java:778)
    at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:504)
    at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1317)
    at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:324)
    at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142)
    at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1065)
    at org.apache.catalina.core.StandardHost.start(StandardHost.java:840)
    at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1057)
    at org.apache.catalina.core.StandardEngine.start(StandardEngine.java:463)
    at org.apache.catalina.core.StandardService.start(StandardService.java:525)
    at org.apache.catalina.core.StandardServer.start(StandardServer.java:754)
    at org.apache.catalina.startup.Catalina.start(Catalina.java:595)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)
    at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414)
Caused by: org.apache.sqoop.common.SqoopException: MAPREDUCE_0002:Failure on submission engine initialization
    at org.apache.sqoop.submission.mapreduce.MapreduceSubmissionEngine.initialize(MapreduceSubmissionEngine.java:115)
    at org.apache.sqoop.framework.JobManager.initialize(JobManager.java:215)
    at org.apache.sqoop.core.SqoopServer.initialize(SqoopServer.java:53)
    ... 25 more
Caused by: java.io.IOException: Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name and the correspond server addresses.
    at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:120)
    at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:82)
    at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:75)
    at org.apache.hadoop.mapred.JobClient.init(JobClient.java:470)
    at org.apache.hadoop.mapred.JobClient.<init>(JobClient.java:449)
    at org.apache.sqoop.submission.mapreduce.MapreduceSubmissionEngine.initialize(MapreduceSubmissionEngine.java:113)
    ... 27 more

根据安装说明，这是我所做的。请注意，对于 EMR，$HADOOP_HOME 是 /home/hadoop。

  • 我从Apache下载了 sqoop-1.99.3-bin-hadoop200.tar.gz并将其解压到 $HADOOP_HOME/sqoop 中。
  • 在 catalina.properties 中的 common.loader 属性中添加了以下内容：/home/hadoop/share/hadoop/common/*.jar,/home/hadoop/share/hadoop/common/lib/*.jar,/home/hadoop /share/hadoop/mapreduce/*.jar,/home/hadoop/share/hadoop/yarn/*.jar
  • 在 sqoop.properties 中：
    • 我用 /home/hadoop/sqoop/log 替换了@LOGDIR@
    • 我用 /home/hadoop/sqoop 替换了@BASEDIR@
    • 对于 org.apache.sqoop.submission.engine.mapreduce.configuration.directory 属性，我将 /etc/hadoop/conf/ 替换为 /home/hadoop/conf/
  • 然后我启动了服务器：bin/sqoop.sh server start

据我从错误中可以看出，问题的根源是这一行：

Caused by: java.io.IOException: Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name and the correspond server addresses.

此属性在 mapred-site.xml 中设置：

<property><name>mapreduce.framework.name</name><value>yarn</value></property>

这就是我想要的。

我觉得我在 sqoop 的安装/配置中缺少一些参数设置，并且我尝试了其他一些东西（例如尝试 AMI 3.0.4 / Hadoop 2.2.0）但无法启动 sqoop服务器。

供参考。我已经阅读了Kyle Mulka 博客上的这篇文章，但它引用了不同版本的 Hadoop 和 sqoop，并且似乎无法深入了解我的配置。我已经阅读了该站点上的其他一些页面，但还没有找到引用我正在使用的 Hadoop 和 sqoop 版本的页面。而且我已经看到这个配置在 Cloudera 上运行（sqoop2 和 Hadoop 2 和 yarn），虽然还没有弄清楚这将如何转化为 EMR 安装。

4

1 回答 1

1

感谢对这篇文章的反馈，我更改了 catalina.properties 中的 common.loader 属性，而不是以前的：

common.loader=${catalina.base}/lib,${catalina.base}/lib/*.jar,${catalina.home}/lib,${catalina.home}/lib/*.jar,${catalina.home}/../lib/*.jar,/home/hadoop/share/hadoop/common/*.jar,/home/hadoop/share/hadoop/common/lib/*.jar,/home/hadoop/share/hadoop/mapreduce/*.jar,/home/hadoop/share/hadoop/yarn/*.jar

对此：

common.loader=${catalina.base}/lib,${catalina.base}/lib/*.jar,${catalina.home}/lib,${catalina.home}/lib/*.jar,${catalina.home}/../lib/*.jar,/home/hadoop/share/hadoop/common/*.jar,/home/hadoop/share/hadoop/common/lib/*.jar,/home/hadoop/share/hadoop/hdfs/*.jar,/home/hadoop/share/hadoop/hdfs/lib/*.jar,/home/hadoop/share/hadoop/mapreduce/*.jar,/home/hadoop/share/hadoop/mapreduce/lib/*.jar,/home/hadoop/share/hadoop/tools/lib/*.jar,/home/hadoop/share/hadoop/yarn/*.jar,/home/hadoop/share/hadoop/yarn/lib/*.jar

之后，sqoop服务器启动成功。

于 2014-09-12T16:45:57.203 回答