0

请帮助了解在 Hadoop 上运行不成功的 Scalding 的输出。

我从 git 获得了最新的 Scalding 发行版:git clone https://github.com/twitter/scalding.git

sbt assembly从目录之后,scalding我尝试使用命令运行教程:

scripts/scald.rb --hdfs tutorial/Tutorial0.scala

结果我得到了以下错误:

scripts/scald.rb:194: warning: already initialized constant SCALA_LIB_DIR
rsyncing 19.8M from scalding-core-assembly-0.10.0.jar to my.host.here in background...
downloading hadoop-core-1.1.2.jar from http://repo1.maven.org/maven2/org/apache/hadoop/hadoop-core/1.1.2/hadoop-core-1.1.2.jar...
ssh: Could not resolve hostname my.host.here: Name or service not known
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(600) [sender=3.0.6]
Successfully downloaded hadoop-core-1.1.2.jar!
downloading commons-codec-1.8.jar from http://repo1.maven.org/maven2/commons-codec/commons-codec/1.8/commons-codec-1.8.jar...
Successfully downloaded commons-codec-1.8.jar!
downloading commons-configuration-1.9.jar from http://repo1.maven.org/maven2/commons-configuration/commons-configuration/1.9/commons-configuration-1.9.jar...
Successfully downloaded commons-configuration-1.9.jar!
downloading jackson-asl-0.9.5.jar from http://repo1.maven.org/maven2/org/codehaus/jackson/jackson-asl/0.9.5/jackson-asl-0.9.5.jar...
Successfully downloaded jackson-asl-0.9.5.jar!
downloading jackson-mapper-asl-1.9.13.jar from http://repo1.maven.org/maven2/org/codehaus/jackson/jackson-mapper-asl/1.9.13/jackson-mapper-asl-1.9.13.jar...
Successfully downloaded jackson-mapper-asl-1.9.13.jar!
downloading commons-lang-2.6.jar from http://repo1.maven.org/maven2/commons-lang/commons-lang/2.6/commons-lang-2.6.jar...
Successfully downloaded commons-lang-2.6.jar!
downloading slf4j-log4j12-1.6.6.jar from http://repo1.maven.org/maven2/org/slf4j/slf4j-log4j12/1.6.6/slf4j-log4j12-1.6.6.jar...
Successfully downloaded slf4j-log4j12-1.6.6.jar!
downloading log4j-1.2.15.jar from http://repo1.maven.org/maven2/log4j/log4j/1.2.15/log4j-1.2.15.jar...
Successfully downloaded log4j-1.2.15.jar!
downloading commons-httpclient-3.1.jar from http://repo1.maven.org/maven2/commons-httpclient/commons-httpclient/3.1/commons-httpclient-3.1.jar...
Successfully downloaded commons-httpclient-3.1.jar!
downloading commons-cli-1.2.jar from http://repo1.maven.org/maven2/commons-cli/commons-cli/1.2/commons-cli-1.2.jar...
Successfully downloaded commons-cli-1.2.jar!
downloading commons-logging-1.1.1.jar from http://repo1.maven.org/maven2/commons-logging/commons-logging/1.1.1/commons-logging-1.1.1.jar...
Successfully downloaded commons-logging-1.1.1.jar!
downloading zookeeper-3.3.4.jar from http://repo1.maven.org/maven2/org/apache/zookeeper/zookeeper/3.3.4/zookeeper-3.3.4.jar...
Successfully downloaded zookeeper-3.3.4.jar!
compiling tutorial/Tutorial0.scala
scalac -classpath /tmp/temp_scala_home_2.9.3_654763/scala-library-2.9.3.jar:/tmp/temp_scala_home_2.9.3_654763/scala-compiler-2.9.3.jar:/home/test/Cascading/scalding/scalding-core/target/scala-2.9.3/scalding-core-assembly-0.10.0.jar:/tmp/maven/hadoop-core-1.1.2.jar:/tmp/maven/commons-codec-1.8.jar:/tmp/maven/commons-configuration-1.9.jar:/tmp/maven/jackson-asl-0.9.5.jar:/tmp/maven/jackson-mapper-asl-1.9.13.jar:/tmp/maven/commons-lang-2.6.jar:/tmp/maven/slf4j-log4j12-1.6.6.jar:/tmp/maven/log4j-1.2.15.jar:/tmp/maven/commons-httpclient-3.1.jar:/tmp/maven/commons-cli-1.2.jar:/tmp/maven/commons-logging-1.1.1.jar:/tmp/maven/zookeeper-3.3.4.jar -d /tmp/script-build tutorial/Tutorial0.scala
ssh: Could not resolve hostname my.host.here: Name or service not known
rsyncing 1.5K from job-jars/Tutorial0.jar to my.host.here in background...
Waiting for 2 background threads...
ssh: Could not resolve hostname my.host.here: Name or service not known
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: unexplained error (code 255) at io.c(600) [sender=3.0.6]
Could not rsync: /home/test/Cascading/scalding/scalding-core/target/scala-2.9.3/scalding-core-assembly-0.10.0.jar to my.host.here:scalding-core-assembly-0.10.0.jar
Could not rsync: /tmp/Tutorial0.jar to my.host.here:job-jars/Tutorial0.jar

* 更新 *

在更改主机后,scald.rb我得到以下身份验证问题:

$ scripts/scald.rb --hdfs tutorial/Tutorial0.scala
scripts/scald.rb:194: warning: already initialized constant SCALA_LIB_DIR
rsyncing 19.8M from scalding-core-assembly-0.10.0.jar to node7.test.net in background...
The authenticity of host 'node7.test.net (10.1.21.32)' can't be established.
RSA key fingerprint is fa:41:31:ab:b0:46:08:8f:2b:75:0a:18:24:f9:d5:ec.
Are you sure you want to continue connecting (yes/no)? The authenticity of host        'node7.test.net (10.1.21.32)' can't be established.

RSA 密钥指纹为 fa:41:31:ab:b0:46:08:8f:2b:75:0a:18:24:f9:d5:ec。您确定要继续连接(是/否)?是 警告:将“node7.test.net”(RSA)永久添加到已知主机列表中。test@node7.test.net 的密码:请输入“是”或“否”:权限被拒绝,请重试。test@node7.test.net 的密码:

我输入了正确的路径词,但身份验证错误仍然存​​在。我应该如何配置rsync

4

1 回答 1

1

你确实改变了这个 https://github.com/twitter/scalding/blob/develop/scripts/scald.rb#l27 对吧?

默认主机是:my.host.here。

于 2014-07-03T21:59:05.437 回答