2

我正在创建一个 bash 来提供执行气流的环境,但由于某种原因,该脚本不能正常工作。

如果我先提供 EMR,然后执行脚本,它就可以了。但是,如果我使用脚本在自定义活动中执行不起作用。我试图更改命令以像 sudo 一样执行,但仍然无法正常工作。

#!/bin/bash

# check for master node
IS_MASTER=true
if [ -f /mnt/var/lib/info/instance.json ]
then
    IS_MASTER=`cat /mnt/var/lib/info/instance.json | tr -d '\n ' | sed -n 's|.*\"isMaster\":\([^,]*\).*|\1|p'`
fi


if [ "$IS_MASTER" = "true}" ]; 
    then
    # install mysql jdbc driver on sqoop
    wget -qN -O ~/mysql-connector-java-5.1.39.tar.gz "https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.39.tar.gz"
    tar -zxvf ~/mysql-connector-java-5.1.39.tar.gz && rm ~/mysql-connector-java-5.1.39.tar.gz

    sudo mv ~/mysql-connector-java-5.1.39/mysql-connector-java-5.1.39-bin.jar /usr/lib/sqoop/lib

    sudo chmod 744 /usr/lib/sqoop/lib/mysql-connector-java-5.1.39-bin.jar

    aws s3 cp s3://monet-datapipeline/scripts/emr_boostrap_scripts/airflow_boostrap ~/ --recursive --exclude "*.sh"

    #create enviroment for airflow
    virtualenv airflowenv -p python3 
    source ~/airflowenv/bin/activate

    pip install --upgrade pip
    pip install airflow
    pip install boto3

    airflow initdb

    mv ~/carriola/airflow.cfg ~/airflow

    airflow webserver -p 9030

    airflow scheduler
fi

这是代码错误。

来自主节点的标准错误。

mv: cannot stat ‘/home/hadoop/mysql-connector-java-5.1.39/mysql-connector-java-5.1.39-bin.jar’: No such file or directory
chmod: cannot access ‘/usr/lib/sqoop/lib/mysql-connector-java-5.1.39-bin.jar’: No such file or directory
/emr/instance-controller/lib/bootstrap-actions/1/airflow_bootstrap.sh: line 25: /home/hadoop/airflowenv/bin/activate: No such file or directory
You are using pip version 6.1.1, however version 8.1.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
      Exception:
      Traceback (most recent call last):
        File "/usr/lib/python2.7/dist-packages/pip/basecommand.py", line 246, in main
          status = self.run(options, args)
        File "/usr/lib/python2.7/dist-packages/pip/commands/install.py", line 352, in run
          root=options.root_path,
        File "/usr/lib/python2.7/dist-packages/pip/req/req_set.py", line 687, in install
          requirement.uninstall(auto_confirm=True)
        File "/usr/lib/python2.7/dist-packages/pip/req/req_install.py", line 730, in uninstall
          paths_to_remove.remove(auto_confirm)
        File "/usr/lib/python2.7/dist-packages/pip/req/req_uninstall.py", line 126, in remove
          renames(path, new_path)
        File "/usr/lib/python2.7/dist-packages/pip/utils/__init__.py", line 292, in renames
          shutil.move(old, new)
        File "/usr/lib64/python2.7/shutil.py", line 303, in move
          os.unlink(src)
      OSError: [Errno 13] Permission denied: '/usr/bin/pip'

You are using pip version 6.1.1, however version 8.1.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
    Command "/usr/bin/python2.7 -c "import setuptools, tokenize;__file__='/mnt/tmp/pip-build-rmGy3J/sqlalchemy/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-J6Ft9n-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /mnt/tmp/pip-build-rmGy3J/sqlalchemy
You are using pip version 6.1.1, however version 8.1.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
  Exception:
  Traceback (most recent call last):
    File "/usr/lib/python2.7/dist-packages/pip/basecommand.py", line 246, in main
      status = self.run(options, args)
    File "/usr/lib/python2.7/dist-packages/pip/commands/install.py", line 352, in run
      root=options.root_path,
    File "/usr/lib/python2.7/dist-packages/pip/req/req_set.py", line 693, in install
      **kwargs
    File "/usr/lib/python2.7/dist-packages/pip/req/req_install.py", line 817, in install
      self.move_wheel_files(self.source_dir, root=root)
    File "/usr/lib/python2.7/dist-packages/pip/req/req_install.py", line 1018, in move_wheel_files
      isolated=self.isolated,
    File "/usr/lib/python2.7/dist-packages/pip/wheel.py", line 237, in move_wheel_files
      clobber(source, lib_dir, True)
    File "/usr/lib/python2.7/dist-packages/pip/wheel.py", line 208, in clobber
      os.makedirs(destdir)
    File "/usr/lib64/python2.7/os.py", line 157, in makedirs
      mkdir(name, mode)
  OSError: [Errno 13] Permission denied: '/usr/local/lib/python2.7/site-packages/s3transfer'

/emr/instance-controller/lib/bootstrap-actions/1/airflow_bootstrap.sh: line 31: airflow: command not found
mv: cannot stat ‘/home/hadoop/carriola/airflow.cfg’: No such file or directory
/emr/instance-controller/lib/bootstrap-actions/1/airflow_bootstrap.sh: line 35: airflow: command not found
/emr/instance-controller/lib/bootstrap-actions/1/airflow_bootstrap.sh: line 37: airflow: command not found
4

1 回答 1

0

您可以使用:

cat /mnt/var/lib/info/instance.json | jq .isMaster

而是查找 isMaster 是否为真。

于 2017-01-31T21:56:29.863 回答