17

我最近airflow为我的工作流程安装了。在创建我的项目时，我执行了以下命令：

airflow initdb

返回以下错误：

[2016-08-15 11:17:00,314] {__init__.py:36} INFO - Using executor SequentialExecutor
DB: sqlite:////Users/mikhilraj/airflow/airflow.db
[2016-08-15 11:17:01,319] {db.py:222} INFO - Creating tables
INFO  [alembic.runtime.migration] Context impl SQLiteImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
ERROR [airflow.models.DagBag] Failed to import: /usr/local/lib/python2.7/site-packages/airflow/example_dags/example_twitter_dag.py
Traceback (most recent call last):
    File "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 247, in process_file
       m = imp.load_source(mod_name, file path)
    File "/usr/local/lib/python2.7/site-packages/airflow/example_dags/example_twitter_dag.py", line 26, in <module>
       from airflow.operators import BashOperator, HiveOperator, PythonOperator
ImportError: cannot import name HiveOperator
Done.

我在网上查了一些类似的问题，建议我安装airflow[hive]，pyhs2但它似乎不起作用。

4

3 回答 3

24

你在用HiveOperator吗？您收到的错误似乎是由于 1 个示例 dags 造成的。在生产中，您可能应该仅在使用 HiveOperator 时设置load_examples并安装False。airflow[hive]

话虽如此，不知道为什么airflow[hive]对你来说还不够。您可以尝试安装airflow[hive,hdfs,jdbc]，但气流 [hive] 应该足以摆脱HiveOperator导入错误。您能否添加您遇到的其他错误？

于 2016-08-15T22:01:19.740 回答
3 
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/airflow/models.py", line 247, in process_file
    m = imp.load_source(mod_name, filepath)
  File "/usr/local/lib/python2.7/dist-packages/airflow/example_dags/example_twitter_dag.py", line 26, in <module>
    from airflow.operators import BashOperator, HiveOperator, PythonOperator
ImportError: cannot import name HiveOperator

如果您仍想继续安装示例数据...对于 Ubuntu 14.04，请将此方法与最新的 python 2.7 一起使用。（在 DO 测试）

1.apt-get更新

2.apt-get install python-pip python-dev build-essential

3.pip install --upgrade pip

3a.which pip #/usr/local/bin/pip

3b.pip -V #pip 9.0.1 来自 /usr/local/lib/python2.7/dist-packages (python 2.7)

4.pip install --upgrade virtualenv

（任务 5 是可选的）

5.apt-get install sqlite3 libsqlite3-dev

https://askubuntu.com/questions/683601/how-to-upgrade-python-setuptools-12-2-on-ubuntu-15-04

6.apt-get 移除 python-setuptools

7.pip install -U pip setuptools

8.export AIRFLOW_HOME=~/气流

9.pip安装气流

10.pip安装气流[蜂巢]

11.气流初始化数据库

您将在下方收到此回复

[2017-02-01 12:04:28,289] {__init__.py:36} INFO - Using executor SequentialExecutor
[2017-02-01 12:04:28,350] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/Grammar.txt
[2017-02-01 12:04:28,376] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
DB: sqlite:////root/airflow/airflow.db
[2017-02-01 12:04:28,522] {db.py:222} INFO - Creating tables
INFO  [alembic.runtime.migration] Context impl SQLiteImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
Done.

注意：如果适用，请应用必要的 sudo 命令

于 2017-02-01T04:10:00.583 回答
0

检查dag文件中是否导入了hive操作符？如果没有，您可以执行以下操作：

from airflow.operators.hive_operator import HiveOperator

于 2018-07-09T19:22:11.370 回答