0

我尝试了几个步骤,但遇到了问题。

下面是我的项目结构

GE-ToolProject
---great_expectations
         --checkpoints
         --expectations
             -.ge_store_backend_id
             -demo-suite.json
             -Expectations.py
         --notebooks
         --plugins
         --uncommitted
         --great_expectations.yml
---dockerfile
---requirement.txt

在上面的结构中,GE-ToolProject 是父文件夹,它包含 great_expectations 文件夹结构,它在运行 great_expectations init 命令时自动创建,我添加的是 Expectations.py 文件,我在其中编写了一个 python 代码来运行期望并生成一个博士。

这里我的数据源是 redshift 数据库,我正在运行自定义查询来获取数据集。

dockerfile 包含:

FROM python:alpine3.7 
COPY . /app
WORKDIR /app
RUN pip install -r requirements.txt 
ENTRYPOINT [ "python" ] 
CMD [ "Expectations.py" ]

requirements.txt 包含

Flask-SQLAlchemy==0.16
sqlalchemy==1.3.24
sqlalchemy-redshift
great_expectations

我想从 docker 运行 Expectations.py 文件。

当我尝试构建一个低于错误的 docker 图像时

[+] Building 75.2s (9/9) FINISHED
 => [internal] load build definition from Dockerfile                                                               0.0s
 => => transferring dockerfile: 179B                                                                               0.0s
 => [internal] load .dockerignore                                                                                  0.0s
 => => transferring context: 2B                                                                                    0.0s
 => [internal] load metadata for docker.io/library/python:alpine3.7                                                5.7s
 => [auth] library/python:pull token for registry-1.docker.io                                                      0.0s
 => [internal] load build context                                                                                  0.4s
 => => transferring context: 18.95kB                                                                               0.4s
 => CACHED [1/4] FROM docker.io/library/python:alpine3.7@sha256:35f6f83ab08f98c727dbefd53738e3b3174a48b4571ccb191  0.0s
 => [2/4] COPY . /app                                                                                              0.1s
 => [3/4] WORKDIR /app                                                                                             0.1s
 => ERROR [4/4] RUN pip install -r requirements.txt                                                               68.8s
------
 > [4/4] RUN pip install -r requirements.txt:
#9 1.462 Collecting Flask-SQLAlchemy==0.16 (from -r requirements.txt (line 1))
#9 4.117   Downloading https://files.pythonhosted.org/packages/92/31/0a7347379af42d211c6c2dc8a4e33f1855264220aed024902baa6f4ddaa2/Flask-SQLAlchemy-0.16.tar.gz (97kB)
#9 5.562 Collecting sqlalchemy==1.3.24 (from -r requirements.txt (line 2))
#9 6.920   Downloading https://files.pythonhosted.org/packages/c5/ab/81bef2f960abf3cdaf32fbf1994f0c6f5e6a5f1667b5713ed6ebf162b6a2/SQLAlchemy-1.3.24.tar.gz (6.4MB)
#9 18.41 Collecting sqlalchemy-redshift (from -r requirements.txt (line 3))
#9 18.82   Downloading https://files.pythonhosted.org/packages/9f/60/47bd9ab38ad5f0b1b20e8ccb1605dc173f75e962ef40deb5d794e0354352/sqlalchemy_redshift-0.8.2-py2.py3-none-any.whl
#9 18.85 Collecting great_expectations (from -r requirements.txt (line 4))
#9 19.39   Downloading https://files.pythonhosted.org/packages/36/9e/49c98831f32d4bf0fae61ffc83c2af39bb3522454b35b21963c193f4c27c/great_expectations-0.13.17-py3-none-any.whl (4.7MB)
#9 25.17 Requirement already satisfied: setuptools in /usr/local/lib/python3.7/site-packages (from Flask-SQLAlchemy==0.16->-r requirements.txt (line 1)) (40.7.1)
#9 25.18 Collecting Flask (from Flask-SQLAlchemy==0.16->-r requirements.txt (line 1))
#9 25.60   Downloading https://files.pythonhosted.org/packages/f2/28/2a03252dfb9ebf377f40fba6a7841b47083260bf8bd8e737b0c6952df83f/Flask-1.1.2-py2.py3-none-any.whl (94kB)
#9 25.73 Collecting packaging (from sqlalchemy-redshift->-r requirements.txt (line 3))
#9 26.15   Downloading https://files.pythonhosted.org/packages/3e/89/7ea760b4daa42653ece2380531c90f64788d979110a2ab51049d92f408af/packaging-20.9-py2.py3-none-any.whl (40kB)
#9 26.19 Collecting numpy<1.20.0,>=1.14.1 (from great_expectations->-r requirements.txt (line 4))
#9 27.78   Downloading https://files.pythonhosted.org/packages/51/60/3f0fe5b7675a461d96b9d6729beecd3532565743278a9c3fe6dd09697fa7/numpy-1.19.5.zip (7.3MB)
#9 41.58   Installing build dependencies: started
#9 48.57   Installing build dependencies: finished with status 'done'
#9 48.57   Getting requirements to build wheel: started
#9 48.86   Getting requirements to build wheel: finished with status 'done'
#9 48.86     Preparing wheel metadata: started
#9 68.57     Preparing wheel metadata: finished with status 'error'
#9 68.57     Complete output from command /usr/local/bin/python /usr/local/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /tmp/tmpzb00tid0:
#9 68.57     Processing numpy/random/_bounded_integers.pxd.in
#9 68.57     Processing numpy/random/_pcg64.pyx
#9 68.57     Processing numpy/random/_philox.pyx
#9 68.57     Processing numpy/random/mtrand.pyx
#9 68.57     Processing numpy/random/_generator.pyx
#9 68.57     Processing numpy/random/_common.pyx
#9 68.57     Processing numpy/random/_mt19937.pyx
#9 68.57     Processing numpy/random/_sfc64.pyx
#9 68.57     Processing numpy/random/_bounded_integers.pyx.in
#9 68.57     Processing numpy/random/bit_generator.pyx
#9 68.57     Cythonizing sources
#9 68.57     blas_opt_info:
#9 68.57     blas_mkl_info:
#9 68.57     customize UnixCCompiler
#9 68.57       libraries mkl_rt not found in ['/usr/local/lib', '/usr/lib']
#9 68.57       NOT AVAILABLE
#9 68.57
#9 68.57     blis_info:
#9 68.57       libraries blis not found in ['/usr/local/lib', '/usr/lib']
#9 68.57       NOT AVAILABLE
#9 68.57
#9 68.57     openblas_info:
#9 68.57       libraries openblas not found in ['/usr/local/lib', '/usr/lib']
#9 68.57       NOT AVAILABLE
#9 68.57
#9 68.57     atlas_3_10_blas_threads_info:
#9 68.57     Setting PTATLAS=ATLAS
#9 68.57       libraries tatlas not found in ['/usr/local/lib', '/usr/lib']
#9 68.57       NOT AVAILABLE
#9 68.57
#9 68.57     atlas_3_10_blas_info:
#9 68.57       libraries satlas not found in ['/usr/local/lib', '/usr/lib']
#9 68.57       NOT AVAILABLE
#9 68.57
#9 68.57     atlas_blas_threads_info:
#9 68.57     Setting PTATLAS=ATLAS
#9 68.57       libraries ptf77blas,ptcblas,atlas not found in ['/usr/local/lib', '/usr/lib']
#9 68.57       NOT AVAILABLE
#9 68.57
#9 68.57     atlas_blas_info:
#9 68.57       libraries f77blas,cblas,atlas not found in ['/usr/local/lib', '/usr/lib']
#9 68.57       NOT AVAILABLE
#9 68.57
#9 68.57     accelerate_info:
#9 68.57       NOT AVAILABLE
#9 68.57
#9 68.57     blas_info:
#9 68.57       libraries blas not found in ['/usr/local/lib', '/usr/lib']
#9 68.57       NOT AVAILABLE
#9 68.57
#9 68.57     blas_src_info:
#9 68.57       NOT AVAILABLE
#9 68.57
#9 68.57       NOT AVAILABLE
#9 68.57
#9 68.57     non-existing path in 'numpy/distutils': 'site.cfg'
#9 68.57     lapack_opt_info:
#9 68.57     lapack_mkl_info:
#9 68.57       libraries mkl_rt not found in ['/usr/local/lib', '/usr/lib']
#9 68.57       NOT AVAILABLE
#9 68.57
#9 68.57     openblas_lapack_info:
#9 68.57       libraries openblas not found in ['/usr/local/lib', '/usr/lib']
#9 68.57       NOT AVAILABLE
#9 68.57
#9 68.57     openblas_clapack_info:
#9 68.57       libraries openblas,lapack not found in ['/usr/local/lib', '/usr/lib']
#9 68.57       NOT AVAILABLE
#9 68.57
#9 68.57     flame_info:
#9 68.57       libraries flame not found in ['/usr/local/lib', '/usr/lib']
#9 68.57       NOT AVAILABLE
#9 68.57
#9 68.57     atlas_3_10_threads_info:
#9 68.57     Setting PTATLAS=ATLAS
#9 68.57       libraries lapack_atlas not found in /usr/local/lib
#9 68.57       libraries tatlas,tatlas not found in /usr/local/lib
#9 68.57       libraries lapack_atlas not found in /usr/lib
#9 68.57       libraries tatlas,tatlas not found in /usr/lib
#9 68.57     <class 'numpy.distutils.system_info.atlas_3_10_threads_info'>
#9 68.57       NOT AVAILABLE
#9 68.57
#9 68.57     atlas_3_10_info:
#9 68.57       libraries lapack_atlas not found in /usr/local/lib
#9 68.57       libraries satlas,satlas not found in /usr/local/lib
#9 68.57       libraries lapack_atlas not found in /usr/lib
#9 68.57       libraries satlas,satlas not found in /usr/lib
#9 68.57     <class 'numpy.distutils.system_info.atlas_3_10_info'>
#9 68.57       NOT AVAILABLE
#9 68.57
#9 68.57     atlas_threads_info:
#9 68.57     Setting PTATLAS=ATLAS
#9 68.57       libraries lapack_atlas not found in /usr/local/lib
#9 68.57       libraries ptf77blas,ptcblas,atlas not found in /usr/local/lib
#9 68.57       libraries lapack_atlas not found in /usr/lib
#9 68.57       libraries ptf77blas,ptcblas,atlas not found in /usr/lib
#9 68.57     <class 'numpy.distutils.system_info.atlas_threads_info'>
#9 68.57       NOT AVAILABLE
#9 68.57
#9 68.57     atlas_info:
#9 68.57       libraries lapack_atlas not found in /usr/local/lib
#9 68.57       libraries f77blas,cblas,atlas not found in /usr/local/lib
#9 68.57       libraries lapack_atlas not found in /usr/lib
#9 68.57       libraries f77blas,cblas,atlas not found in /usr/lib
#9 68.57     <class 'numpy.distutils.system_info.atlas_info'>
#9 68.57       NOT AVAILABLE
#9 68.57
#9 68.57     lapack_info:
#9 68.57       libraries lapack not found in ['/usr/local/lib', '/usr/lib']
#9 68.57       NOT AVAILABLE
#9 68.57
#9 68.57     lapack_src_info:
#9 68.57       NOT AVAILABLE
#9 68.57
#9 68.57       NOT AVAILABLE
#9 68.57
#9 68.57     numpy_linalg_lapack_lite:
#9 68.57       FOUND:
#9 68.57         language = c
#9 68.57         define_macros = [('HAVE_BLAS_ILP64', None), ('BLAS_SYMBOL_SUFFIX', '64_')]
#9 68.57
#9 68.57     running dist_info
#9 68.57     running build_src
#9 68.57     build_src
#9 68.57     building py_modules sources
#9 68.57     creating build
#9 68.57     creating build/src.linux-x86_64-3.7
#9 68.57     creating build/src.linux-x86_64-3.7/numpy
#9 68.57     creating build/src.linux-x86_64-3.7/numpy/distutils
#9 68.57     building library "npymath" sources
#9 68.57     Could not locate executable gfortran
#9 68.57     Could not locate executable f95
#9 68.57     Could not locate executable ifort
#9 68.57     Could not locate executable ifc
#9 68.57     Could not locate executable lf95
#9 68.57     Could not locate executable pgfortran
#9 68.57     Could not locate executable nvfortran
#9 68.57     Could not locate executable f90
#9 68.57     Could not locate executable f77
#9 68.57     Could not locate executable fort
#9 68.57     Could not locate executable efort
#9 68.57     Could not locate executable efc
#9 68.57     Could not locate executable g77
#9 68.57     Could not locate executable g95
#9 68.57     Could not locate executable pathf95
#9 68.57     Could not locate executable nagfor
#9 68.57     don't know how to compile Fortran code on platform 'posix'
#9 68.57     Running from numpy source directory.
#9 68.57     setup.py:480: UserWarning: Unrecognized setuptools command, proceeding with generating Cython sources and expanding templates
#9 68.57       run_build = parse_setuppy_commands()
#9 68.57     /tmp/pip-install-c4ba9ifm/numpy/numpy/distutils/system_info.py:1914: UserWarning:
#9 68.57         Optimized (vendor) Blas libraries are not found.
#9 68.57         Falls back to netlib Blas library which has worse performance.
#9 68.57         A better performance should be easily gained by switching
#9 68.57         Blas library.
#9 68.57       if self._calc_info(blas):
#9 68.57     /tmp/pip-install-c4ba9ifm/numpy/numpy/distutils/system_info.py:1914: UserWarning:
#9 68.57         Blas (http://www.netlib.org/blas/) libraries not found.
#9 68.57         Directories to search for the libraries can be specified in the
#9 68.57         numpy/distutils/site.cfg file (section [blas]) or by setting
#9 68.57         the BLAS environment variable.
#9 68.57       if self._calc_info(blas):
#9 68.57     /tmp/pip-install-c4ba9ifm/numpy/numpy/distutils/system_info.py:1914: UserWarning:
#9 68.57         Blas (http://www.netlib.org/blas/) sources not found.
#9 68.57         Directories to search for the sources can be specified in the
#9 68.57         numpy/distutils/site.cfg file (section [blas_src]) or by setting
#9 68.57         the BLAS_SRC environment variable.
#9 68.57       if self._calc_info(blas):
#9 68.57     /tmp/pip-install-c4ba9ifm/numpy/numpy/distutils/system_info.py:1748: UserWarning:
#9 68.57         Lapack (http://www.netlib.org/lapack/) libraries not found.
#9 68.57         Directories to search for the libraries can be specified in the
#9 68.57         numpy/distutils/site.cfg file (section [lapack]) or by setting
#9 68.57         the LAPACK environment variable.
#9 68.57       return getattr(self, '_calc_info_{}'.format(name))()
#9 68.57     /tmp/pip-install-c4ba9ifm/numpy/numpy/distutils/system_info.py:1748: UserWarning:
#9 68.57         Lapack (http://www.netlib.org/lapack/) sources not found.
#9 68.57         Directories to search for the sources can be specified in the
#9 68.57         numpy/distutils/site.cfg file (section [lapack_src]) or by setting
#9 68.57         the LAPACK_SRC environment variable.
#9 68.57       return getattr(self, '_calc_info_{}'.format(name))()
#9 68.57     /usr/local/lib/python3.7/distutils/dist.py:274: UserWarning: Unknown distribution option: 'define_macros'
#9 68.57       warnings.warn(msg)
#9 68.57     Traceback (most recent call last):
#9 68.57       File "/usr/local/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py", line 207, in <module>
#9 68.57         main()
#9 68.57       File "/usr/local/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py", line 197, in main
#9 68.57         json_out['return_val'] = hook(**hook_input['kwargs'])
#9 68.57       File "/usr/local/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py", line 69, in prepare_metadata_for_build_wheel
#9 68.57         return hook(metadata_directory, config_settings)
#9 68.57       File "/tmp/pip-build-env-sn8bmpub/overlay/lib/python3.7/site-packages/setuptools/build_meta.py", line 157, in prepare_metadata_for_build_wheel
#9 68.57         self.run_setup()
#9 68.57       File "/tmp/pip-build-env-sn8bmpub/overlay/lib/python3.7/site-packages/setuptools/build_meta.py", line 142, in run_setup
#9 68.57         exec(compile(code, __file__, 'exec'), locals())
#9 68.57       File "setup.py", line 508, in <module>
#9 68.57         setup_package()
#9 68.57       File "setup.py", line 500, in setup_package
#9 68.57         setup(**metadata)
#9 68.57       File "/tmp/pip-install-c4ba9ifm/numpy/numpy/distutils/core.py", line 169, in setup
#9 68.57         return old_setup(**new_attr)
#9 68.57       File "/tmp/pip-build-env-sn8bmpub/overlay/lib/python3.7/site-packages/setuptools/__init__.py", line 165, in setup
#9 68.57         return distutils.core.setup(**attrs)
#9 68.57       File "/usr/local/lib/python3.7/distutils/core.py", line 148, in setup
#9 68.57         dist.run_commands()
#9 68.57       File "/usr/local/lib/python3.7/distutils/dist.py", line 966, in run_commands
#9 68.57         self.run_command(cmd)
#9 68.57       File "/usr/local/lib/python3.7/distutils/dist.py", line 985, in run_command
#9 68.57         cmd_obj.run()
#9 68.57       File "/tmp/pip-build-env-sn8bmpub/overlay/lib/python3.7/site-packages/setuptools/command/dist_info.py", line 31, in run
#9 68.57         egg_info.run()
#9 68.57       File "/tmp/pip-install-c4ba9ifm/numpy/numpy/distutils/command/egg_info.py", line 24, in run
#9 68.57         self.run_command("build_src")
#9 68.57       File "/usr/local/lib/python3.7/distutils/cmd.py", line 313, in run_command
#9 68.57         self.distribution.run_command(command)
#9 68.57       File "/usr/local/lib/python3.7/distutils/dist.py", line 985, in run_command
#9 68.57         cmd_obj.run()
#9 68.57       File "/tmp/pip-install-c4ba9ifm/numpy/numpy/distutils/command/build_src.py", line 144, in run
#9 68.57         self.build_sources()
#9 68.57       File "/tmp/pip-install-c4ba9ifm/numpy/numpy/distutils/command/build_src.py", line 155, in build_sources
#9 68.57         self.build_library_sources(*libname_info)
#9 68.57       File "/tmp/pip-install-c4ba9ifm/numpy/numpy/distutils/command/build_src.py", line 288, in build_library_sources
#9 68.57         sources = self.generate_sources(sources, (lib_name, build_info))
#9 68.57       File "/tmp/pip-install-c4ba9ifm/numpy/numpy/distutils/command/build_src.py", line 378, in generate_sources
#9 68.57         source = func(extension, build_dir)
#9 68.57       File "numpy/core/setup.py", line 663, in get_mathlib_info
#9 68.57         raise RuntimeError("Broken toolchain: cannot link a simple C program")
#9 68.57     RuntimeError: Broken toolchain: cannot link a simple C program
#9 68.57
#9 68.57     ----------------------------------------
#9 68.68 Command "/usr/local/bin/python /usr/local/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /tmp/tmpzb00tid0" failed with error code 1 in /tmp/pip-install-c4ba9ifm/numpy
#9 68.71 You are using pip version 19.0.1, however version 21.0.1 is available.
#9 68.71 You should consider upgrading via the 'pip install --upgrade pip' command.
------
executor failed running [/bin/sh -c pip install -r requirements.txt]: exit code: 1```


what I am missing here.

can someone please help me to resolve this.

4

1 回答 1

0

问题是 docker 容器无法编译。利用

RUN apk add make automake gcc g++

或者alpine-sdk那是等效的包build-essential

RUN apk add alpine-sdk
于 2021-07-01T13:47:16.210 回答