m1芯片Mac安装Apple版TensorFlow(conda-forge)+编译安装sklearn
写在前面
折腾了一晚上,终于搞定了用conda-forge安装Apple优化之后的TensorFlow啦!真的太不容易了…一开始我是不想折腾的,后来才发现直接用bash安装的Python虚拟环境使用起来有很多不方便的地方,而且就算很多软件包可以通过源码编译的方式安装到虚拟环境,但是还是有那么几个不配合的软件包安装不了…就比如sklearn,人家官网上明确说了:
For Apple Silicon M1 hardware, only the conda-forge method below is known to work at the time of writing (January 2021). You can install the macos/arm64 distribution of conda using the miniforge installer
对于Apple Silicon M1硬件，在撰写本文时（2021年1月），只有下面的conda-forge方法有效。 您可以使用miniforge安装程序安装conda的macos / arm64发行版。
那就只好硬着头皮找方法了,在Apple的GitHub主页issue3中,还真的有用miniforge安装TensorFlow的方法,安装起来没啥难度,除了那烦人的网速…
安装好TensorFlow之后,就是sklearn这个包的安装了,除了要特别指定一下brew安装的libomp包的位置,其他的根据官网的步骤一路走下来就能安装好了.下面具体操作.
s
k
l
e
a
r
n
可
以
不
编
译
安
装
.
.
我
傻
了
\color{red}sklearn可以不编译安装..我傻了
sklearn可以不编译安装..我傻了
conda安装优化版TensorFlow
这里需要先创建虚拟环境,激活该环境,
conda create -n tf24 python=3.8
conda activate tf24
然后就是指定一下TensorFlow_macos这个下载好的软件包的位置,
libs="/Users/xxx/tensorflow_macos/arm64/"
env="/opt/homebrew/Caskroom/miniforge/base/envs/tf24/"
安装相关依赖包,这里也要注意一下版本号,建议复制一下安装的文件名,要不可能会找不到文件(版本更新很快的):
conda install -c conda-forge pip setuptools cached-property six
conda upgrade -c conda-forge pip setuptools cached-property six
pip install --upgrade -t "$env/lib/python3.8/site-packages/" --no-dependencies --force "$libs/grpcio-1.33.2-cp38-cp38-macosx_11_0_arm64.whl"
pip install --upgrade -t "$env/lib/python3.8/site-packages/" --no-dependencies --force "$libs/h5py-2.10.0-cp38-cp38-macosx_11_0_arm64.whl"
pip install --upgrade -t "$env/lib/python3.8/site-packages/" --no-dependencies --force "$libs/tensorflow_addons_macos-0.1a3-cp38-cp38-macosx_11_0_arm64.whl"
接下来是跟TensorFlow有关的一些包:
conda install -c conda-forge -y absl-py
conda install -c conda-forge -y astunparse
conda install -c conda-forge -y gast
conda install -c conda-forge -y opt_einsum
conda install -c conda-forge -y termcolor
conda install -c conda-forge -y typing_extensions
conda install -c conda-forge -y wheel
conda install -c conda-forge -y typeguard
TensorBoard:
pip install tensorboard
TensorFlow相关组件:
pip install wrapt flatbuffers tensorflow_estimator google_pasta keras_preprocessing protobuf
最后就是重头戏TensorFlow本尊了(同样需要注意版本号):
pip install --upgrade -t "$env/lib/python3.8/site-packages/" --no-dependencies --force "$libs/tensorflow_macos-0.1a3-cp38-cp38-macosx_11_0_arm64.whl"
安装其他数据科学软件包
这个比起来直接创建的tf_venv环境下安装软件包(各种编译安装),简直是简单到爆了,直接conda install走天下…就是指定一下channel为conda-forge就可以了.
pandas&pytables
conda install pandas -c conda-forge
conda install pytables -c conda-forge
matplotlib
conda install matplotlib -c conda-forge
ipython
conda install ipython -c conda-forge
20210314下午更新
后来找到一篇medium博客的文章:tensorflow-2-4-on-apple-silicon-m1-installation-under-conda-environment(需要富强上网).
直接一句
conda install -c conda-forge scikit-learn
就解决的问题,我非给源码编译了一遍??希望大家别学我…尽量找到最方便快捷的方法.
源码编译安装sklearn
由于sklearn不支持直接用homebrew编译安装,那就只好硬着头皮编译了.
安装相关依赖
conda activate tf24
conda install -c conda-forge scipy cython \
joblib threadpoolctl pytest compilers llvm-openmp
brew install libomp
需要记录一下libomp的位置,一会导入路径时候要修改一下:
/opt/homebrew/Cellar/libomp/11.1.0/lib
/opt/homebrew/Cellar/libomp/11.1.0/include
直接运行安装命令的话又是一大片的红色报错,根据官网的提示:
The compilers meta-package will automatically set custom environment variables:
echo $CC
echo $CXX
echo $CFLAGS
echo $CXXFLAGS
echo $LDFLAGS
They point to files and folders from your sklearn-dev conda environment (in particular in the bin/, include/ and lib/ subfolders). For instance -L/path/to/conda/envs/sklearn-dev/lib should appear in LDFLAGS.
For instance -L/path/to/conda/envs/sklearn-dev/lib should appear in LDFLAGS.
只需要修改编译路径为虚拟环境的路径即可,但是我修改后编译还是没有成功,后来抱着试一试的态度再看一下下面介绍的brew安装的方法,就是把环境变量导入为libomp包的位置,然后再运行,竟然成功了!!!
首先看一下官网给出的(Intel芯片的brew位置),
export CC=/usr/bin/clang
export CXX=/usr/bin/clang++
export CPPFLAGS="$CPPFLAGS -Xpreprocessor -fopenmp"
export CFLAGS="$CFLAGS -I/usr/local/opt/libomp/include"
export CXXFLAGS="$CXXFLAGS -I/usr/local/opt/libomp/include"
export LDFLAGS="$LDFLAGS -Wl,-rpath,/usr/local/opt/libomp/lib -L/usr/local/opt/libomp/lib -lomp"
需要修改为(上一步没记录的话,需要brew info libomp看一下安装位置)本地编译版本的brew的包安装位置:
export CC=/usr/bin/clang
export CXX=/usr/bin/clang++
export CPPFLAGS="$CPPFLAGS -Xpreprocessor -fopenmp"
export CFLAGS="$CFLAGS -I/opt/homebrew/Cellar/libomp/11.1.0/include"
export CXXFLAGS="$CXXFLAGS -I/opt/homebrew/Cellar/libomp/11.1.0/include"
export LDFLAGS="$LDFLAGS -Wl,-rpath,/opt/homebrew/Cellar/libomp/11.1.0/lib -L/opt/homebrew/Cellar/libomp/11.1.0/lib -lomp"
然后再执行编译,
make clean
pip install --verbose --no-build-isolation --editable .
successfully这个单词我印象深刻…
然后用
conda list
查看一下包信息:
scikit-learn 0.24.1 dev_0 <develop>
scipy 1.6.0 py38hdf044fb_0 conda-forge
setuptools 49.6.0 py38h10201cd_3 conda-forge
six 1.15.0 pyh9f0ad1d_0 conda-forge
sqlite 3.34.0 h6d56c25_0 conda-forge
tapi 1100.0.11 he4954df_0 conda-forge
tensorboard 2.4.1 pypi_0 pypi
tensorboard-plugin-wit 1.8.0 pypi_0 pypi
tensorflow 2.4.0rc0 pypi_0 pypi
完成!
但是这里还不能急着庆祝,查看一下sklearn包的安装位置:
In [6]: sklearn._config
Out[6]: <module 'sklearn._config' from '/Users/xxx/Downloads/scikit-learn-0.24.1/sklearn/_config.py'>
其实是被安装在了源码文件夹,需要再进行操作才可以安装到虚拟环境中,
- 拷贝安装文件以及egg-info文件:
(tf24) ~/Downloads/scikit-learn-0.24.1 cp -r sklearn /opt/homebrew/Caskroom/miniforge/base/envs/tf24/lib/python3.8/site-packages/ (tf24) ~/Downloads/scikit-learn-0.24.1 cp -r scikit_learn.egg-info /opt/homebrew/Caskroom/miniforge/base/envs/tf24/lib/python3.8/site-packages/ - 删除链接文件(可选):
(tf24) ~ cd /opt/homebrew/Caskroom/miniforge/base/envs/tf24/lib/python3.8/site-packages/ (tf24) /opt/homebrew/Caskroom/miniforge/base/envs/tf24/lib/python3.8/site-packages rm scikit-learn.egg-link
这样就可以完美安装啦!(并且可以删除sklearn的源码文件了~)
验证
~ conda activate tf24
(tf24) ~ ipython
Python 3.8.8 | packaged by conda-forge | (default, Feb 20 2021, 15:50:57)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.21.0 -- An enhanced Interactive Python. Type '?' for help.
In [1]: import sklearn
In [2]: sklearn._config
Out[2]: <module 'sklearn._config' from '/opt/homebrew/Caskroom/miniforge/base/envs/tf24/lib/python3.8/site-packages/sklearn/_config.py'>