1

问题陈述：

我正在尝试使用 python MRJob 库在 Amazon EMR 中运行 map-reduce 作业，但在使用必要的库和包引导节点时遇到了问题。

细节：

我的示例python mrjob代码：

    import re
    from mrjob.job import MRJob
    from sentClassifier import sentClassify
    import nltk

    .. do something ..

有一些像 NLTK 这样的库需要导入，还有一些我正在导入的本地模块，比如from sentClassifier import sentClassify

我想知道引导 EMR 节点以使这些方法和包可用的最佳方法是什么。该代码在我的本地机器上运行良好。

我的示例mrjob.conf文件：

    runners:
      emr:
        aws_access_key_id: ***
        aws_secret_access_key: ***
        ec2_core_instance_type: m1.large
        ec2_key_pair: mykey
        ec2_key_pair_file: mykey.pem
        num_ec2_core_instances: 5
        pool_wait_minutes: 2
        pool_emr_job_flows: true
        ssh_tunnel_is_open: true
        ssh_tunnel_to_job_tracker: true
      hadoop:
        setup:
          - virtualenv venv
          - . venv/bin/activate
          - pip install mr3po simplejson
          - sudo easy_install https://code.google.com/p/nltk/downloads/detail?name=nltk-2.0b9-py2.6.egg&can=2&q=

但是工作失败了。

我已阅读以下参考资料并尝试了所有各种方法，但仍然没有运气：

错误日志：

    Scanning SSH logs for probable cause of failure
    Probable cause of failure (from ssh://ec2-54-86-50-115.compute-1.amazonaws.com!172.31.19.60/mnt/var/log/hadoop/userlogs/job_201405030101_0006/attempt_201405030101_0006_m_000002_3/stderr):
    Traceback (most recent call last):
    File "obidroidMR.py", line 5, in <module>
       import nltk
       ImportError: No module named nltk
       (while reading from s3://mrjob-   51b9493c1a467671/tmp/obidroidMR.shreyas.20140503.012933.336228/files/STDIN)
       Attempting to terminate job...
       Job appears to have already been terminated
       Killing our SSH tunnel (pid 12909)
       Traceback (most recent call last):
         File "obidroidMR.py", line 107, in <module>
         ObidroidReview.run()
         File "/Users/shreyas/anaconda/envs/obidroid/lib/python2.7/site-packages/mrjob/job.py", line 494, in run
         mr_job.execute()
         File "/Users/shreyas/anaconda/envs/obidroid/lib/python2.7/site-packages/mrjob/job.py", line 512, in execute
super(MRJob, self).execute()
         File "/Users/shreyas/anaconda/envs/obidroid/lib/python2.7/site-packages/mrjob/launch.py", line 147, in execute
         self.run_job()
         File "/Users/shreyas/anaconda/envs/obidroid/lib/python2.7/site-packages/mrjob/launch.py", line 208, in run_job
runner.run()
         File "/Users/shreyas/anaconda/envs/obidroid/lib/python2.7/site-packages/mrjob/runner.py", line 458, in run
self._run()
         File "/Users/shreyas/anaconda/envs/obidroid/lib/python2.7/site-packages/mrjob/emr.py", line 809, in _run
         self._wait_for_job_to_complete()
         File "/Users/shreyas/anaconda/envs/obidroid/lib/python2.7/site-packages/mrjob/emr.py", line 1599, in _wait_for_job_to_complete
         raise Exception(msg)
         Exception: Job on job flow j-2R8G1Q3RIE9ED failed with status WAITING: Waiting after step failed
         Probable cause of failure (from ssh://ec2-54-86-50-115.compute-1.amazonaws.com!172.31.19.60/mnt/var/log/hadoop/userlogs/job_201405030101_0006/attempt_201405030101_0006_m_000002_3/stderr):
         Traceback (most recent call last):
         File "obidroidMR.py", line 5, in <module>
         import nltk
         ImportError: No module named nltk

任何帮助将非常感激

4

2 回答 2

2

在mrjob.conf安装软件包所需的行中可能不在它们应该在的位置。应该为在 EMR 上运行的作业应用的东西应该列在下面emr:而不是hadoop:（这是在本地 Hadoop 安装上运行作业时的配置。

如果它是一个简单的 Linux 命令，例如pipor apt-get，那么您应该能够像这样安装软件包：

runners:
  emr:
    aws_access_key_id: ***
    ... all the other stuff ...
    bootstrap_cmds:
    - sudo apt-get install -y python-boto
    - sudo pip install simplejson

我从来没有尝试过专门安装 NLTK，所以我无法帮助你，但你应该可以按照这条线安装。

对于可能更复杂的安装，我建议ssh使用 EMR CLI 安装到您的主节点上：

$ ./elastic-mapreduce -j JOB_FLOW_ID --ssh

并实际尝试安装软件包。如果您找到一系列成功安装软件包的 shell 命令，那么您只需将其复制并粘贴到您的mrjob.conf.

于 2014-05-03T11:48:25.970 回答
0

鉴于 Amazon Elastic Map Reduce 使用基于 Amazon Linux 的 AMI，我验证我可以nltk使用以下命令在 Amazon Linux AMI 2014.03.1 - ami-fb8e9292（64 位）上安装

sudo easy_install -U pip
sudo easy_install -U distribute
sudo pip install -U pyyaml nltk

您可以尝试将这 3 行合并到您的 mrjob.conf

于 2014-05-03T18:51:49.783 回答