我尝试了几个步骤,但遇到了问题。
下面是我的项目结构
GE-ToolProject
---great_expectations
--checkpoints
--expectations
-.ge_store_backend_id
-demo-suite.json
-Expectations.py
--notebooks
--plugins
--uncommitted
--great_expectations.yml
---dockerfile
---requirement.txt
在上面的结构中,GE-ToolProject 是父文件夹,它包含 great_expectations 文件夹结构,它在运行 great_expectations init 命令时自动创建,我添加的是 Expectations.py 文件,我在其中编写了一个 python 代码来运行期望并生成一个博士。
这里我的数据源是 redshift 数据库,我正在运行自定义查询来获取数据集。
dockerfile 包含:
FROM python:alpine3.7
COPY . /app
WORKDIR /app
RUN pip install -r requirements.txt
ENTRYPOINT [ "python" ]
CMD [ "Expectations.py" ]
requirements.txt 包含
Flask-SQLAlchemy==0.16
sqlalchemy==1.3.24
sqlalchemy-redshift
great_expectations
我想从 docker 运行 Expectations.py 文件。
当我尝试构建一个低于错误的 docker 图像时
[+] Building 75.2s (9/9) FINISHED
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 179B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load metadata for docker.io/library/python:alpine3.7 5.7s
=> [auth] library/python:pull token for registry-1.docker.io 0.0s
=> [internal] load build context 0.4s
=> => transferring context: 18.95kB 0.4s
=> CACHED [1/4] FROM docker.io/library/python:alpine3.7@sha256:35f6f83ab08f98c727dbefd53738e3b3174a48b4571ccb191 0.0s
=> [2/4] COPY . /app 0.1s
=> [3/4] WORKDIR /app 0.1s
=> ERROR [4/4] RUN pip install -r requirements.txt 68.8s
------
> [4/4] RUN pip install -r requirements.txt:
#9 1.462 Collecting Flask-SQLAlchemy==0.16 (from -r requirements.txt (line 1))
#9 4.117 Downloading https://files.pythonhosted.org/packages/92/31/0a7347379af42d211c6c2dc8a4e33f1855264220aed024902baa6f4ddaa2/Flask-SQLAlchemy-0.16.tar.gz (97kB)
#9 5.562 Collecting sqlalchemy==1.3.24 (from -r requirements.txt (line 2))
#9 6.920 Downloading https://files.pythonhosted.org/packages/c5/ab/81bef2f960abf3cdaf32fbf1994f0c6f5e6a5f1667b5713ed6ebf162b6a2/SQLAlchemy-1.3.24.tar.gz (6.4MB)
#9 18.41 Collecting sqlalchemy-redshift (from -r requirements.txt (line 3))
#9 18.82 Downloading https://files.pythonhosted.org/packages/9f/60/47bd9ab38ad5f0b1b20e8ccb1605dc173f75e962ef40deb5d794e0354352/sqlalchemy_redshift-0.8.2-py2.py3-none-any.whl
#9 18.85 Collecting great_expectations (from -r requirements.txt (line 4))
#9 19.39 Downloading https://files.pythonhosted.org/packages/36/9e/49c98831f32d4bf0fae61ffc83c2af39bb3522454b35b21963c193f4c27c/great_expectations-0.13.17-py3-none-any.whl (4.7MB)
#9 25.17 Requirement already satisfied: setuptools in /usr/local/lib/python3.7/site-packages (from Flask-SQLAlchemy==0.16->-r requirements.txt (line 1)) (40.7.1)
#9 25.18 Collecting Flask (from Flask-SQLAlchemy==0.16->-r requirements.txt (line 1))
#9 25.60 Downloading https://files.pythonhosted.org/packages/f2/28/2a03252dfb9ebf377f40fba6a7841b47083260bf8bd8e737b0c6952df83f/Flask-1.1.2-py2.py3-none-any.whl (94kB)
#9 25.73 Collecting packaging (from sqlalchemy-redshift->-r requirements.txt (line 3))
#9 26.15 Downloading https://files.pythonhosted.org/packages/3e/89/7ea760b4daa42653ece2380531c90f64788d979110a2ab51049d92f408af/packaging-20.9-py2.py3-none-any.whl (40kB)
#9 26.19 Collecting numpy<1.20.0,>=1.14.1 (from great_expectations->-r requirements.txt (line 4))
#9 27.78 Downloading https://files.pythonhosted.org/packages/51/60/3f0fe5b7675a461d96b9d6729beecd3532565743278a9c3fe6dd09697fa7/numpy-1.19.5.zip (7.3MB)
#9 41.58 Installing build dependencies: started
#9 48.57 Installing build dependencies: finished with status 'done'
#9 48.57 Getting requirements to build wheel: started
#9 48.86 Getting requirements to build wheel: finished with status 'done'
#9 48.86 Preparing wheel metadata: started
#9 68.57 Preparing wheel metadata: finished with status 'error'
#9 68.57 Complete output from command /usr/local/bin/python /usr/local/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /tmp/tmpzb00tid0:
#9 68.57 Processing numpy/random/_bounded_integers.pxd.in
#9 68.57 Processing numpy/random/_pcg64.pyx
#9 68.57 Processing numpy/random/_philox.pyx
#9 68.57 Processing numpy/random/mtrand.pyx
#9 68.57 Processing numpy/random/_generator.pyx
#9 68.57 Processing numpy/random/_common.pyx
#9 68.57 Processing numpy/random/_mt19937.pyx
#9 68.57 Processing numpy/random/_sfc64.pyx
#9 68.57 Processing numpy/random/_bounded_integers.pyx.in
#9 68.57 Processing numpy/random/bit_generator.pyx
#9 68.57 Cythonizing sources
#9 68.57 blas_opt_info:
#9 68.57 blas_mkl_info:
#9 68.57 customize UnixCCompiler
#9 68.57 libraries mkl_rt not found in ['/usr/local/lib', '/usr/lib']
#9 68.57 NOT AVAILABLE
#9 68.57
#9 68.57 blis_info:
#9 68.57 libraries blis not found in ['/usr/local/lib', '/usr/lib']
#9 68.57 NOT AVAILABLE
#9 68.57
#9 68.57 openblas_info:
#9 68.57 libraries openblas not found in ['/usr/local/lib', '/usr/lib']
#9 68.57 NOT AVAILABLE
#9 68.57
#9 68.57 atlas_3_10_blas_threads_info:
#9 68.57 Setting PTATLAS=ATLAS
#9 68.57 libraries tatlas not found in ['/usr/local/lib', '/usr/lib']
#9 68.57 NOT AVAILABLE
#9 68.57
#9 68.57 atlas_3_10_blas_info:
#9 68.57 libraries satlas not found in ['/usr/local/lib', '/usr/lib']
#9 68.57 NOT AVAILABLE
#9 68.57
#9 68.57 atlas_blas_threads_info:
#9 68.57 Setting PTATLAS=ATLAS
#9 68.57 libraries ptf77blas,ptcblas,atlas not found in ['/usr/local/lib', '/usr/lib']
#9 68.57 NOT AVAILABLE
#9 68.57
#9 68.57 atlas_blas_info:
#9 68.57 libraries f77blas,cblas,atlas not found in ['/usr/local/lib', '/usr/lib']
#9 68.57 NOT AVAILABLE
#9 68.57
#9 68.57 accelerate_info:
#9 68.57 NOT AVAILABLE
#9 68.57
#9 68.57 blas_info:
#9 68.57 libraries blas not found in ['/usr/local/lib', '/usr/lib']
#9 68.57 NOT AVAILABLE
#9 68.57
#9 68.57 blas_src_info:
#9 68.57 NOT AVAILABLE
#9 68.57
#9 68.57 NOT AVAILABLE
#9 68.57
#9 68.57 non-existing path in 'numpy/distutils': 'site.cfg'
#9 68.57 lapack_opt_info:
#9 68.57 lapack_mkl_info:
#9 68.57 libraries mkl_rt not found in ['/usr/local/lib', '/usr/lib']
#9 68.57 NOT AVAILABLE
#9 68.57
#9 68.57 openblas_lapack_info:
#9 68.57 libraries openblas not found in ['/usr/local/lib', '/usr/lib']
#9 68.57 NOT AVAILABLE
#9 68.57
#9 68.57 openblas_clapack_info:
#9 68.57 libraries openblas,lapack not found in ['/usr/local/lib', '/usr/lib']
#9 68.57 NOT AVAILABLE
#9 68.57
#9 68.57 flame_info:
#9 68.57 libraries flame not found in ['/usr/local/lib', '/usr/lib']
#9 68.57 NOT AVAILABLE
#9 68.57
#9 68.57 atlas_3_10_threads_info:
#9 68.57 Setting PTATLAS=ATLAS
#9 68.57 libraries lapack_atlas not found in /usr/local/lib
#9 68.57 libraries tatlas,tatlas not found in /usr/local/lib
#9 68.57 libraries lapack_atlas not found in /usr/lib
#9 68.57 libraries tatlas,tatlas not found in /usr/lib
#9 68.57 <class 'numpy.distutils.system_info.atlas_3_10_threads_info'>
#9 68.57 NOT AVAILABLE
#9 68.57
#9 68.57 atlas_3_10_info:
#9 68.57 libraries lapack_atlas not found in /usr/local/lib
#9 68.57 libraries satlas,satlas not found in /usr/local/lib
#9 68.57 libraries lapack_atlas not found in /usr/lib
#9 68.57 libraries satlas,satlas not found in /usr/lib
#9 68.57 <class 'numpy.distutils.system_info.atlas_3_10_info'>
#9 68.57 NOT AVAILABLE
#9 68.57
#9 68.57 atlas_threads_info:
#9 68.57 Setting PTATLAS=ATLAS
#9 68.57 libraries lapack_atlas not found in /usr/local/lib
#9 68.57 libraries ptf77blas,ptcblas,atlas not found in /usr/local/lib
#9 68.57 libraries lapack_atlas not found in /usr/lib
#9 68.57 libraries ptf77blas,ptcblas,atlas not found in /usr/lib
#9 68.57 <class 'numpy.distutils.system_info.atlas_threads_info'>
#9 68.57 NOT AVAILABLE
#9 68.57
#9 68.57 atlas_info:
#9 68.57 libraries lapack_atlas not found in /usr/local/lib
#9 68.57 libraries f77blas,cblas,atlas not found in /usr/local/lib
#9 68.57 libraries lapack_atlas not found in /usr/lib
#9 68.57 libraries f77blas,cblas,atlas not found in /usr/lib
#9 68.57 <class 'numpy.distutils.system_info.atlas_info'>
#9 68.57 NOT AVAILABLE
#9 68.57
#9 68.57 lapack_info:
#9 68.57 libraries lapack not found in ['/usr/local/lib', '/usr/lib']
#9 68.57 NOT AVAILABLE
#9 68.57
#9 68.57 lapack_src_info:
#9 68.57 NOT AVAILABLE
#9 68.57
#9 68.57 NOT AVAILABLE
#9 68.57
#9 68.57 numpy_linalg_lapack_lite:
#9 68.57 FOUND:
#9 68.57 language = c
#9 68.57 define_macros = [('HAVE_BLAS_ILP64', None), ('BLAS_SYMBOL_SUFFIX', '64_')]
#9 68.57
#9 68.57 running dist_info
#9 68.57 running build_src
#9 68.57 build_src
#9 68.57 building py_modules sources
#9 68.57 creating build
#9 68.57 creating build/src.linux-x86_64-3.7
#9 68.57 creating build/src.linux-x86_64-3.7/numpy
#9 68.57 creating build/src.linux-x86_64-3.7/numpy/distutils
#9 68.57 building library "npymath" sources
#9 68.57 Could not locate executable gfortran
#9 68.57 Could not locate executable f95
#9 68.57 Could not locate executable ifort
#9 68.57 Could not locate executable ifc
#9 68.57 Could not locate executable lf95
#9 68.57 Could not locate executable pgfortran
#9 68.57 Could not locate executable nvfortran
#9 68.57 Could not locate executable f90
#9 68.57 Could not locate executable f77
#9 68.57 Could not locate executable fort
#9 68.57 Could not locate executable efort
#9 68.57 Could not locate executable efc
#9 68.57 Could not locate executable g77
#9 68.57 Could not locate executable g95
#9 68.57 Could not locate executable pathf95
#9 68.57 Could not locate executable nagfor
#9 68.57 don't know how to compile Fortran code on platform 'posix'
#9 68.57 Running from numpy source directory.
#9 68.57 setup.py:480: UserWarning: Unrecognized setuptools command, proceeding with generating Cython sources and expanding templates
#9 68.57 run_build = parse_setuppy_commands()
#9 68.57 /tmp/pip-install-c4ba9ifm/numpy/numpy/distutils/system_info.py:1914: UserWarning:
#9 68.57 Optimized (vendor) Blas libraries are not found.
#9 68.57 Falls back to netlib Blas library which has worse performance.
#9 68.57 A better performance should be easily gained by switching
#9 68.57 Blas library.
#9 68.57 if self._calc_info(blas):
#9 68.57 /tmp/pip-install-c4ba9ifm/numpy/numpy/distutils/system_info.py:1914: UserWarning:
#9 68.57 Blas (http://www.netlib.org/blas/) libraries not found.
#9 68.57 Directories to search for the libraries can be specified in the
#9 68.57 numpy/distutils/site.cfg file (section [blas]) or by setting
#9 68.57 the BLAS environment variable.
#9 68.57 if self._calc_info(blas):
#9 68.57 /tmp/pip-install-c4ba9ifm/numpy/numpy/distutils/system_info.py:1914: UserWarning:
#9 68.57 Blas (http://www.netlib.org/blas/) sources not found.
#9 68.57 Directories to search for the sources can be specified in the
#9 68.57 numpy/distutils/site.cfg file (section [blas_src]) or by setting
#9 68.57 the BLAS_SRC environment variable.
#9 68.57 if self._calc_info(blas):
#9 68.57 /tmp/pip-install-c4ba9ifm/numpy/numpy/distutils/system_info.py:1748: UserWarning:
#9 68.57 Lapack (http://www.netlib.org/lapack/) libraries not found.
#9 68.57 Directories to search for the libraries can be specified in the
#9 68.57 numpy/distutils/site.cfg file (section [lapack]) or by setting
#9 68.57 the LAPACK environment variable.
#9 68.57 return getattr(self, '_calc_info_{}'.format(name))()
#9 68.57 /tmp/pip-install-c4ba9ifm/numpy/numpy/distutils/system_info.py:1748: UserWarning:
#9 68.57 Lapack (http://www.netlib.org/lapack/) sources not found.
#9 68.57 Directories to search for the sources can be specified in the
#9 68.57 numpy/distutils/site.cfg file (section [lapack_src]) or by setting
#9 68.57 the LAPACK_SRC environment variable.
#9 68.57 return getattr(self, '_calc_info_{}'.format(name))()
#9 68.57 /usr/local/lib/python3.7/distutils/dist.py:274: UserWarning: Unknown distribution option: 'define_macros'
#9 68.57 warnings.warn(msg)
#9 68.57 Traceback (most recent call last):
#9 68.57 File "/usr/local/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py", line 207, in <module>
#9 68.57 main()
#9 68.57 File "/usr/local/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py", line 197, in main
#9 68.57 json_out['return_val'] = hook(**hook_input['kwargs'])
#9 68.57 File "/usr/local/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py", line 69, in prepare_metadata_for_build_wheel
#9 68.57 return hook(metadata_directory, config_settings)
#9 68.57 File "/tmp/pip-build-env-sn8bmpub/overlay/lib/python3.7/site-packages/setuptools/build_meta.py", line 157, in prepare_metadata_for_build_wheel
#9 68.57 self.run_setup()
#9 68.57 File "/tmp/pip-build-env-sn8bmpub/overlay/lib/python3.7/site-packages/setuptools/build_meta.py", line 142, in run_setup
#9 68.57 exec(compile(code, __file__, 'exec'), locals())
#9 68.57 File "setup.py", line 508, in <module>
#9 68.57 setup_package()
#9 68.57 File "setup.py", line 500, in setup_package
#9 68.57 setup(**metadata)
#9 68.57 File "/tmp/pip-install-c4ba9ifm/numpy/numpy/distutils/core.py", line 169, in setup
#9 68.57 return old_setup(**new_attr)
#9 68.57 File "/tmp/pip-build-env-sn8bmpub/overlay/lib/python3.7/site-packages/setuptools/__init__.py", line 165, in setup
#9 68.57 return distutils.core.setup(**attrs)
#9 68.57 File "/usr/local/lib/python3.7/distutils/core.py", line 148, in setup
#9 68.57 dist.run_commands()
#9 68.57 File "/usr/local/lib/python3.7/distutils/dist.py", line 966, in run_commands
#9 68.57 self.run_command(cmd)
#9 68.57 File "/usr/local/lib/python3.7/distutils/dist.py", line 985, in run_command
#9 68.57 cmd_obj.run()
#9 68.57 File "/tmp/pip-build-env-sn8bmpub/overlay/lib/python3.7/site-packages/setuptools/command/dist_info.py", line 31, in run
#9 68.57 egg_info.run()
#9 68.57 File "/tmp/pip-install-c4ba9ifm/numpy/numpy/distutils/command/egg_info.py", line 24, in run
#9 68.57 self.run_command("build_src")
#9 68.57 File "/usr/local/lib/python3.7/distutils/cmd.py", line 313, in run_command
#9 68.57 self.distribution.run_command(command)
#9 68.57 File "/usr/local/lib/python3.7/distutils/dist.py", line 985, in run_command
#9 68.57 cmd_obj.run()
#9 68.57 File "/tmp/pip-install-c4ba9ifm/numpy/numpy/distutils/command/build_src.py", line 144, in run
#9 68.57 self.build_sources()
#9 68.57 File "/tmp/pip-install-c4ba9ifm/numpy/numpy/distutils/command/build_src.py", line 155, in build_sources
#9 68.57 self.build_library_sources(*libname_info)
#9 68.57 File "/tmp/pip-install-c4ba9ifm/numpy/numpy/distutils/command/build_src.py", line 288, in build_library_sources
#9 68.57 sources = self.generate_sources(sources, (lib_name, build_info))
#9 68.57 File "/tmp/pip-install-c4ba9ifm/numpy/numpy/distutils/command/build_src.py", line 378, in generate_sources
#9 68.57 source = func(extension, build_dir)
#9 68.57 File "numpy/core/setup.py", line 663, in get_mathlib_info
#9 68.57 raise RuntimeError("Broken toolchain: cannot link a simple C program")
#9 68.57 RuntimeError: Broken toolchain: cannot link a simple C program
#9 68.57
#9 68.57 ----------------------------------------
#9 68.68 Command "/usr/local/bin/python /usr/local/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /tmp/tmpzb00tid0" failed with error code 1 in /tmp/pip-install-c4ba9ifm/numpy
#9 68.71 You are using pip version 19.0.1, however version 21.0.1 is available.
#9 68.71 You should consider upgrading via the 'pip install --upgrade pip' command.
------
executor failed running [/bin/sh -c pip install -r requirements.txt]: exit code: 1```
what I am missing here.
can someone please help me to resolve this.