在docker Alpine中安装熊猫

Question

我很难在docker中安装稳定的数据科学包配置。有了这样的主流相关工具，这应该更容易实现。

下面是Dockerfile，它使用了一些技巧，从包核心中删除pandas，并单独安装，指定pandas

    FROM alpine:3.6

    ENV PACKAGES="\
    dumb-init \
    musl \
    libc6-compat \
    linux-headers \
    build-base \
    bash \
    git \
    ca-certificates \
    freetype \
    libgfortran \
    libgcc \
    libstdc++ \
    openblas \
    tcl \
    tk \
    libssl1.0 \
    "

ENV PYTHON_PACKAGES="\
    numpy \
    matplotlib \
    scipy \
    scikit-learn \
    nltk \
    " 

RUN apk add --no-cache --virtual build-dependencies python3 \
    && apk add --virtual build-runtime \
    build-base python3-dev openblas-dev freetype-dev pkgconfig gfortran \
    && ln -s /usr/include/locale.h /usr/include/xlocale.h \
    && python3 -m ensurepip \
    && rm -r /usr/lib/python*/ensurepip \
    && pip3 install --upgrade pip setuptools \
    && ln -sf /usr/bin/python3 /usr/bin/python \
    && ln -sf pip3 /usr/bin/pip \
    && rm -r /root/.cache \
    && pip install --no-cache-dir $PYTHON_PACKAGES \
    && pip3 install 'pandas<0.21.0' \    #<---------- PANDAS
    && apk del build-runtime \
    && apk add --no-cache --virtual build-dependencies $PACKAGES \
    && rm -rf /var/cache/apk/*

# set working directory
WORKDIR /usr/src/app

# add and install requirements
COPY ./requirements.txt /usr/src/app/requirements.txt # other than data science packages go here
RUN pip install -r requirements.txt

# add entrypoint.sh
COPY ./entrypoint.sh /usr/src/app/entrypoint.sh

RUN chmod +x /usr/src/app/entrypoint.sh

# add app
COPY . /usr/src/app

# run server
CMD ["/usr/src/app/entrypoint.sh"]

上面的配置用于工作。现在发生的情况是，构建确实完成了，但导入时pandas失败，出现以下错误：

ImportError: Missing required dependencies ['numpy']

由于安装了numpy 1.16.1，我不知道哪个numpypandas正在尝试查找更多。。。

有人知道如何获得稳定的解决方案吗？

注意：由数据科学的交钥匙docker映像中的至少上面提到的包组成的解决方案也将非常受欢迎。

编辑1：

如果我将数据包的安装移到要求中。txt，如注释中所示：

要求。文本

(...)
numpy==1.16.1 # or numpy==1.16.0
scikit-learn==0.20.2
scipy==1.2.1
nltk==3.4   
pandas==0.24.1 # or pandas== 0.23.4
matplotlib==3.0.2 
(...)

和Dockerfile：

# add and install requirements
COPY ./requirements.txt /usr/src/app/requirements.txt
RUN pip install -r requirements.txt

它在pandas处再次中断，抱怨numpy。

Collecting numpy==1.16.1 (from -r requirements.txt (line 61))
  Downloading https://files.pythonhosted.org/packages/2b/26/07472b0de91851b6656cbc86e2f0d5d3a3128e7580f23295ef58b6862d6c/numpy-1.16.1.zip (5.1MB)
Collecting scikit-learn==0.20.2 (from -r requirements.txt (line 62))
  Downloading https://files.pythonhosted.org/packages/49/0e/8312ac2d7f38537361b943c8cde4b16dadcc9389760bb855323b67bac091/scikit-learn-0.20.2.tar.gz (10.3MB)
Collecting scipy==1.2.1 (from -r requirements.txt (line 63))
  Downloading https://files.pythonhosted.org/packages/a9/b4/5598a706697d1e2929eaf7fe68898ef4bea76e4950b9efbe1ef396b8813a/scipy-1.2.1.tar.gz (23.1MB)
Collecting nltk==3.4 (from -r requirements.txt (line 64))
  Downloading https://files.pythonhosted.org/packages/6f/ed/9c755d357d33bc1931e157f537721efb5b88d2c583fe593cc09603076cc3/nltk-3.4.zip (1.4MB)
Collecting pandas==0.24.1 (from -r requirements.txt (line 65))
  Downloading https://files.pythonhosted.org/packages/81/fd/b1f17f7dc914047cd1df9d6813b944ee446973baafe8106e4458bfb68884/pandas-0.24.1.tar.gz (11.8MB)
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 359, in get_provider
        module = sys.modules[moduleOrReq]
    KeyError: 'numpy'

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-_e5z6o6_/pandas/setup.py", line 732, in <module>
        ext_modules=maybe_cythonize(extensions, compiler_directives=directives),
      File "/tmp/pip-install-_e5z6o6_/pandas/setup.py", line 475, in maybe_cythonize
        numpy_incl = pkg_resources.resource_filename('numpy', 'core/include')
      File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 1144, in resource_filename
        return get_provider(package_or_requirement).get_resource_filename(
      File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 361, in get_provider
        __import__(moduleOrReq)
    ModuleNotFoundError: No module named 'numpy'

Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-install-_e5z6o6_/pandas/

编辑2：

这似乎是一个公开的问题。有关更多详细信息，请参阅：

熊猫开发github

“不幸的是，这意味着requirements.txt文件不足以设置安装了pandas的新环境（如docker容器中）”。

  **ImportError**:

  IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!

  Importing the multiarray numpy extension module failed.  Most
  likely you are trying to import a failed build of numpy.
  Here is how to proceed:
  - If you're working with a numpy git repository, try `git clean -xdf`
    (removes all files not under version control) and rebuild numpy.
  - If you are simply trying to use the numpy version that you have installed:
    your installation is broken - please reinstall numpy.
  - If you have already reinstalled and that did not fix the problem, then:
    1. Check that you are using the Python you expect (you're using /usr/local/bin/python),
       and that you have no directories in your PATH or PYTHONPATH that can
       interfere with the Python and numpy versions you're trying to use.
    2. If (1) looks fine, you can open a new issue at
       https://github.com/numpy/numpy/issues.  Please include details on:
       - how you installed Python
       - how you installed numpy
       - your operating system
       - whether or not you have multiple versions of Python installed
       - if you built from source, your compiler versions and ideally a build log

编辑3

requirements.txt---

编辑4

自20年12月1日起，已接受的解决方案开始不再有效。现在，在构建scipy的控制盘时，构建中断不是在pandas处，而是在scipy处，而是在numpy之后。这是日志：

  ----------------------------------------
  ERROR: Failed building wheel for scipy
  Running setup.py clean for scipy
  ERROR: Command errored out with exit status 1:
   command: /usr/bin/python3.6 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-s6nahssd/scipy/setup.py'"'"'; __file__='"'"'/tmp/pip-install-s6nahssd/scipy/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' clean --all
       cwd: /tmp/pip-install-s6nahssd/scipy
  Complete output (9 lines):

  `setup.py clean` is not supported, use one of the following instead:

    - `git clean -xdf` (cleans all files)
    - `git clean -Xdf` (cleans all versioned files, doesn't touch
                        files that aren't checked into the git repo)

  Add `--force` to your command to use it anyway if you must (unsupported).

  ----------------------------------------
  ERROR: Failed cleaning build dir for scipy
Successfully built numpy
Failed to build scipy
ERROR: Could not build wheels for scipy which use PEP 517 and cannot be installed directly

从错误中可以看出，构建过程使用的是python3。6，而我使用的是来自alpine:3.7的。

这里有完整的日志-

这是当前的Dockerfile：

https://pastebin.com/3SftEufx




Answer 1

这可能并不完全相关，因为这是在阿尔卑斯山搜索numpy/pandas安装失败时弹出的第一个答案，我添加了这个答案。

以下修复对我有效（但安装pandas/numpy需要更长的时间）

apk update
apk --no-cache add curl gcc g++
ln -s /usr/include/locale.h /usr/include/xlocale.h

现在尝试安装pandas/numpy

Answer 2

尝试将此添加到您的requirements.txt文件：

numpy==1.16.0
pandas==0.23.4

从昨天开始，我一直面临着同样的错误，这个变化为我解决了这个问题。

Answer 3

匿名用户

如果您没有绑定到Alpine 3.6，那么使用Alpine 3.7（或更高版本）应该可以。

在Alpine 3.6上，我安装matplotlib失败，原因如下：

Collecting matplotlib
  Downloading https://files.pythonhosted.org/packages/26/04/8b381d5b166508cc258632b225adbafec49bbe69aa9a4fa1f1b461428313/matplotlib-3.0.3.tar.gz (36.6MB)
    Complete output from command python setup.py egg_info:
    Download error on https://pypi.org/simple/numpy/: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:833) -- Some packages may not be found!
    Couldn't find index page for 'numpy' (maybe misspelled?)
    Download error on https://pypi.org/simple/: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:833) -- Some packages may not be found!
    No local packages or working download links found for numpy>=1.10.0

然而，在阿尔卑斯山3.7上，它成功了。这可能是由于numpy版本控制问题造成的（请参见此处），但我无法确定。解决了这个问题后，成功地构建和安装了软件包——花了很长时间，大约30分钟（由于Alpine的musl libc与Python的Wheels格式不兼容，所有安装了pip的软件包都必须从源代码构建）。

请注意，需要进行一个重要的更改：您应该只删除pip安装之后的构建-运行时虚拟包（apk del构建-运行时）。此外，如果适用的话，您可以将Numpy1.16.1替换为1.16.2，这是附带的版本（否则1.16.2将被卸载，并且1.16.1从源代码构建，进一步增加构建时间）-我没有不过，试过这个。

作为参考，这是我稍微修改的Dockerfile和docker构建输出。

注:

通常选择Alpine作为最小化图像大小的基础（Alpine在其他方面也非常灵活，但由于glibc/musl，与大陆Linux应用程序存在兼容性问题）。必须从源代码构建Python包有点超出了这个目的，因为在进行任何清理之前，您会得到一个非常臃肿的映像—900MB，这也需要花费很长时间来构建。通过删除所有中间编译工件、构建依赖项等，可以极大地压缩映像，但仍然可以。

如果你不能得到你需要在Alpine上工作的Python包版本，而不必从源代码中构建它们，我建议尝试其他更小的和更兼容的基本映像，如debian-slm，甚至ubuntu。

编辑：

在添加了需求的“编辑3”之后，这里是更新的Dockerfile和Docker生成输出。为满足生成依赖关系，添加了以下包：

postgresql-dev libffi-dev libressl-dev libxml2 libxml2-dev libxslt libxslt-dev libjpeg-turbo-dev zlib-dev

对于由于特定的标题而无法构建的包，我使用Alpin的包内容搜索来查找丢失的包。具体到cffi，ffi. h头丢失，需要libffi-dev包：https://pkgs.alpinelinux.org/contents?file=ffi.h

或者，当包构建失败不是很清楚时，例如，可以参考特定包的安装说明。

在任何压缩之前，新的图像大小为1.04GB。为了减少一点，你可以删除Python和pip缓存：

RUN apk del build-runtime && \
    find -type d -name __pycache__ -prune -exec rm -rf {} \; && \
    rm -rf ~/.cache/pip

当使用docker build--squash时，这将使图像大小降至661MB。