1

我尝试按照本教程运行 mapreduce。

我在文件夹 ~/hello_mapreduce 中的 EC2 实例上上传了文件 mrjob.conf、readme.txt 和 word_count.py 并尝试运行命令：

python word_count.py -r emr README.txt

它返回以下内容：

no configs found; falling back on auto-configuration
no configs found; falling back on auto-configuration
Traceback (most recent call last):
  File "word_count.py", line 21, in <module>
    MRWordFrequencyCount.run()
  File "/usr/local/lib/python2.7/dist-packages/mrjob/job.py", line 494, in run
    mr_job.execute()
  File "/usr/local/lib/python2.7/dist-packages/mrjob/job.py", line 512, in execute
    super(MRJob, self).execute()
  File "/usr/local/lib/python2.7/dist-packages/mrjob/launch.py", line 147, in execute
    self.run_job()
  File "/usr/local/lib/python2.7/dist-packages/mrjob/launch.py", line 207, in run_job
    with self.make_runner() as runner:
  File "/usr/local/lib/python2.7/dist-packages/mrjob/job.py", line 535, in make_runner
    return super(MRJob, self).make_runner()
  File "/usr/local/lib/python2.7/dist-packages/mrjob/launch.py", line 162, in make_runner
    return EMRJobRunner(**self.emr_job_runner_kwargs())
  File "/usr/local/lib/python2.7/dist-packages/mrjob/emr.py", line 579, in __init__
    self._fix_s3_scratch_and_log_uri_opts()
  File "/usr/local/lib/python2.7/dist-packages/mrjob/emr.py", line 664, in _fix_s3_scratch_and_log_uri_opts
    s3_conn = self.make_s3_conn()
  File "/usr/local/lib/python2.7/dist-packages/mrjob/fs/s3.py", line 226, in make_s3_conn
    host=self._s3_endpoint)
  File "/usr/lib/python2.7/dist-packages/boto/__init__.py", line 108, in connect_s3
    return S3Connection(aws_access_key_id, aws_secret_access_key, **kwargs)
  File "/usr/lib/python2.7/dist-packages/boto/s3/connection.py", line 154, in __init__
    suppress_consec_slashes=suppress_consec_slashes)
  File "/usr/lib/python2.7/dist-packages/boto/connection.py", line 486, in __init__
    host, config, self.provider, self._required_auth_capability())
  File "/usr/lib/python2.7/dist-packages/boto/auth.py", line 406, in get_auth_handler
    'Check your credentials' % (len(names), str(names)))
boto.exception.NoAuthHandlerFound: No handler was ready to authenticate. 1 handlers were checked. ['HmacAuthV1Handler'] Check your credentials

我可能错了，但我猜可能/主要问题是文件 mrjob.conf 不可见但不知道为什么。

mrjob.conf 的内容如下：

runners:
  emr:
    aws_access_key_id: xxxxx
    aws_secret_access_key: xxxxxxxxxx

    ec2_key_pair: EMR
    ec2_key_pair_file: ~/.ssh/EMR.pem # ~/ and $ENV_VARS allowed here
    ssh_tunnel_to_job_tracker: true

作为 AWS 用户新手，如果有人能帮助解决问题以便使用 mrjob 和 AWS 成功运行 mapreduce，我将非常感激。

谢谢！

4

1 回答 1

2

解决了。

我使用命令（明确定义的位置）运行了这项工作：

 python word_count.py -r emr readme.txt --conf-path mrjob.conf

并且从 .conf 文件中删除了建立 ssh 连接的行：

runners:
  emr:
    aws_access_key_id: xxxxx
    aws_secret_access_key: xxxxxxxxxx
于 2014-08-10T14:04:32.230 回答