hadoop - 如何在 Amazon S3 上使用存储配置 hadoop 1.2.1？

Question

我已尝试按照此 wiki http://wiki.apache.org/hadoop/AmazonS3将以下内容放入我的 core-site.xml 文件中，但我的 hadoop 无法与 S3n 建立连接。我在 EC2 集群上部署了 hadoop 1.2.1。使用 S3 存储配置 Hadoop 的正确方法是什么？多谢！

 <property>
  <name>fs.default.name</name>
  <value>s3://BUCKET</value>
</property>

<property>
  <name>fs.s3.awsAccessKeyId</name>
  <value>ID</value>
</property>

<property>
  <name>fs.s3.awsSecretAccessKey</name>
  <value>SECRET</value>
</property>

Answer 1

Did you try to replace s3 with s3n in your configuration?

Try this:

    <property>
      <name>fs.default.name</name>
      <value>s3n://BUCKET</value>
    </property>

    <property>
      <name>fs.s3n.awsAccessKeyId</name>
      <value>ID</value>
    </property>

    <property>
      <name>fs.s3n.awsSecretAccessKey</name>
      <value>SECRET</value>
    </property>

By the way there're great explanations of the difference between s3 and s3n here

Additionally you may need to start and stop HDFS cluster (not sure that there is a need to start it, as we don't use it anymore).