一、 install_deps:安装系统级依赖
【Ubuntu环境】
- curl: Get a file from an HTTP, HTTPS or FTP server
- libxml2-dev: Development files for the GNOME XML library
- libxslt-dev:
- libgl1-mesa-dev: free implementation of the OpenGL API – GLX development files
- libgl1-mesa-glx: free implementation of the OpenGL API – GLX runtime
- libglapi-mesa: free implementation of the GL API – shared library
- libgl1-mesa-dri: free implementation of the OpenGL API – DRI modules
- nginx: small, but very powerful and efficient web server and mail proxy
- python-dev: header files and a static library for Python (default)
- python-mysql.connector: pure Python implementation of MySQL Client/Server protocol
- python-numpy: Numerical Python adds a fast array facility to the Python language
- python-openssl: Python wrapper around the OpenSSL library
- python-pip: alternative Python package installer
- python-software-properties: manage the repositories that you install software from
【CentOS环境】
http://centos-packages.com/7m
- yum groupinstall “C Development Tools and Libraries”
- yum groupinstall ‘Development Tools’
- yum install python-devel
- yum安装:
- curl
- libxml2, libxml2-devel
- libxslt, libxslt-devel
- mesa-libGL-devel: Mesa libGL development package
- mesa-libGL: Mesa libGL runtime libraries and DRI drivers
- mesa-libglapi: Mesa shared glapi
- mesa-dri-drivers: Mesa-based DRI drivers
- nginx
- MySQL-python, mysql-connector-*
- numpy
- 用yum安装可自动安装依赖blas,lapack,atlas,python-nose
- openssl, openssl-devel
- python-devel
- bzip2 bzip2-devel
- 安装python2.7时
- 路径设置
- /usr/local/python/Python-2.7.10 安装目录
- /usr/local/python/2.7.10/lib 生成库目录
- https://github.com/docker-library/python/issues/21
- ./configure --enable-unicode=ucs4 --enable-shared --prefix=/usr/local/python/2.7.10 LDFLAGS="-Wl,-rpath /usr/local/python/2.7.10/lib"
- make
- make altinstall
- cd /usr/bin
- ln -fs python2.7 python (原python2.7.5)
- ln -fs /usr/local/python/2.7.10/bin/python2.7 python2.7.10 (当前python2.7.10)
- ln -fs python2.7.10 python2 (当前python2.7.10)
- 安装setuptool, pip, virtualenv, virtualenvwrapper,并设置/etc/profile,然后创建一个python虚拟环境
- 方案一:
- 方案二:
- pip install virtualenv
- pip install virtualenvwrapper
- mkdir /usr/local/pyenv
- 编辑/etc/profile, /root/.bashrc
- export WORKON_HOME=/usr/local/pyenv
- source /usr/bin/virtualenvwrapper.sh
- source /etc/profile
- 创建python虚拟环境:py2_portia
- mkvirtualenv -p python2.7.10 py2_portia
- workon py2_portia
二、 install_splash: 安装splash
– install_builddeps()
【Ubuntu环境】
- python3-dev: header files and a static library for Python (default)
- python3-pip: alternative Python package installer - Python 3 version of the package
- build-essential: Informational list of build-essential packages
- libre2-dev: efficient, principled regular expression library, development kit
- liblua5.2-dev: Development files for the Lua language version 5.2
- libsqlite3-dev: SQLite 3 development file
- zlib1g-dev: compression library - development
- libjpeg-turbo8-dev: Development files for the IJG JPEG library
- libgl1-mesa-dev: free implementation of the OpenGL API – GLX development files
- libglu1-mesa-dev: Mesa OpenGL utility library – development files
- mesa-common-dev: Developer documentation for Mesa
【CentOS环境】
- yum install python34-devel
- yum install python34-setuptools
- easy_install-3.4 pip
- 安装re2, re2-devel
- yumdownloader --urls re2 re2-devel
- 得到两个urls
- wget …
- rpm -ivh …
- yum install lua lua-devel ???
- yum install sqlite sqlite-devel
- yum install zlib zlib-devel
- yum install libjpeg-turbo-devel
- yum install mesa-*
- yum install mesa-libGLU mesa-libGLU-devel
- yum install libffi libffi-devel
– install_deps()
【Ubuntu环境】
*
|「netbase: Basic TCP/IP networking system
- ca-certificates: Common CA certificates
- xvfb: Virtual Framebuffer ‘fake’ X server
- pkg-config: manage compile and link flags for libraries
- python3
- qt55base
- qt55webkit
- qt55svg
- libre2
- libicu52 (libicu55: International Components for Unicode)
- liblua5.2-0
- zlib1g
【CentOS环境】
- yum install ca-certificates
- yum install xorg-x11-server-Xvfb
- yum install pkgconfig
- yum install python34
- yum install qt5-qt* (时间过长)或者只安装:
- yum install qt5-qtbase qt5-qtbase-devel
- yum install qt5-qtwebkit qt5-qtwebkit-devel
- yum install qt5-qtsvg qt5-qtsvg-devel
- yum install libicu
– install_pyqt5()
- 安装sip
- cd sip-4.18.1
- 【Python2】
- 【Python3】
- 安装Qt5
- ./qt-opensource-linux-x64-5.4.0.run
- 指定安装目录: /opt/qt54
- 设置环境变量: vim /etc/profile
- QTDIR=/opt/qt54/Tools/QtCreator
- QMAKEDIR=/opt/qt54/5.4/gcc_64/bin
- PATH=
Q
M
A
K
E
D
I
R
:
QMAKEDIR:
QMAKEDIR:QTDIR/bin:$PATH
- MANPATH=
Q
T
D
I
R
/
m
a
n
:
QTDIR/man:
QTDIR/man:MANPATH
- LD_LIBRARY_PATH=
Q
T
D
I
R
/
l
i
b
:
QTDIR/lib:
QTDIR/lib:LD_LIBRARY_PATH
- export QTDIR MANPATH PATH LD_LIBRARY_PATH
- source /etc/profile
- 安装PyQt5
- cd PyQt5_gpl-5.7
- 【Python2】
- 【Python3】
- python configure.py -c --qmake /opt/qt57/5.7/gcc_64/bin/qmake --verbose --confirm-license --no-designer-plugin -e QtCore -e QtGui -e QtWidgets -e QtNetwork -e QtWebKit -e QtWebKitWidgets -e QtSvg -e QtPrintSupport
- make
- make install
- 安装lua 5.3.3
- tar zxvf lua-5.3.3.tar.gz
- cd lua-5.3.3
- make clean
- make linux
- make install
- 安装lupa
- tar zxvf lupa-1.3.tar.gz
- cd lupa-1.3
- tar zxvf LuaJIT-2.0.4.tar.gz
- cd LuaJIT-2.0.4/
- make clean
- make CFLAGS=-fPIC
- cd …
- python setup.py install
– install_python_deps()
- 【Python2】
- pip install --upgrade pip
- pip install six (当前最新版本==1.10.1)
- pip install qt5reactor (当前最新版本==0.3)
- pip install psutil (当前最新版本4.3.0,保守4.1.0)
- pip install Twisted16.1.1 (当前最新版本16.4.0,保守==16.1.1)
- pip install adblockparser (当前最新版本==0.5)
- pip install xvfbwrapper (当前最新版本==0.2.8)
- pip install funcparserlib (当前最新版本==0.3.6)
- pip install Pillow (当前最新版本3.3.1,保守3.3.0)
- pip install lupa (当前最新版本==1.3)
- 安装python-re2
- git clone git://github.com/axiak/pyre2.git
- cd pyre2
- python setup.py install
– requirements-dev.txt
- 【Python2】
- pip install pytest (当前最新版本==3.0.1)
- pip install pyOpenSSL (当前最新版本==16.1.0)
- pip install cryptography (当前最新版本==1.5)
- pip install requests (当前最新版本==2.11.1)
- pip install jsonschema(当前最新版本==2.5.1)
- pip install strict-rfc3339(当前最新版本==0.7)
- pip install jupyter_kernel_test(当前最新版本==0.2.2)
- pip install nose(当前最新版本==1.3.7)
- pip install ipython[notebook](ipython当前最新版本5.1.0, notebook当前最新版本4.2.2)
- pip install service-identity (当前最新版本==16.0.0)
– splash源码
cd /root/git
pip install -e splash
三、 install_python_deps:安装python依赖
- 【slyd/requirements.txt】
- pip install six (已装)
- pip install twisted (已装,当前最新版本16.4.0,保守15.4.0)
- pip install service-identity (已装,当前最新版本=16.0.0,保守==14.0.0)
- pip install requests (当前最新版本2.11.1,保守2.7.0)
- pip install autobahn0.10.4(当前最新版本0.16.0,保守==0.10.4)
- pip install chardet (当前最新版本==2.3.0)
- pip install monotonic (当前最新版本1.2,保守0.3)
- pip install parse (当前最新版本==1.6.6)
- pip install ndg-httpsclient (当前最新版本0.4.2,保守0.4.0)
- Dulwich’s MySQL backend
- Splash dev
- 先把setup.py中的Twisted后面的版本要求注释掉
- pip install -e /root/git/splash
- pip install -e /root/git/portia/slyd
- 【slybot/requirements.txt】
- pip install scrapy (当前最新版本1.1.2,保守1.1.0)
- pip install scrapely (当前最新版本==0.12.0)
- pip install loginform (当前最新版本1.2.0,保守1.0)
- pip install lxml (当前最新版本3.6.4,保守3.6.0)
- pip install dateparser (当前最新版本0.4.0,保守0.3.5)
- pip install python-dateutil (当前最新版本2.5.3,保守2.4.2)
- pip install jsonschema (当前最新版本2.5.1,保守2.4.0)
- pip install six (当前最新版本==1.10.0)
- pip install scrapyjs (当前最新版本0.2,保守0.1.1)
- pip install page_finder (当前最新版本==0.1.2)
- pip install page_clustering (当前最新版本==0.0.1)
- pip install -e /root/git/portia/slybot
四、 安装ember.js
- 安装node.js
- 下载https://nodejs.org/dist/v4.5.0/node-v4.5.0-linux-x64.tar.xz
- 解压压缩包
- 复制到cp -r node-v4.5.0-linux-x64 /usr/local/node
- 配置环境变量 vi /etc/profile
- 在文件最后添加export PATH="/usr/local/node/bin:$PATH"
- source /etc/profile
- 输入node -v 和 npm -v检测是否安装成功
- 更新npm
- 安装bower
- 安装ember.js
- 初始化portia/slyd
五、 安装并配置nginx
使用已安装的Python,创建python虚拟环境:py2_portia_nui
- mkvirtualenv -p python2.7.10 py2_portia_nui
- workon py2_portia_nui
portia/provision.sh
- install_deps:安装系统级依赖
- yum install python-devel
- yum install curl
- yum install libxml2 libxml2-devel
- yum install libxslt libxslt-devel
- yum install mesa-*
- yum install mysql mysql-devel mysql-libs
- yum install nginx
- yum install MySQL-python mysql-connector-*
- yum install numpy
- yum install openssl openssl-devel pyOpenSSL
-
install_splash: 安装splash
同上
-
install_python_deps:安装python依赖
- 【slyd/requirements.txt】
- pip install six (已装,当前最新版本1.10.1,保守1.10.0) #python2和python3的兼容
- pip install twisted16.1.1 (已装,当前最新版本16.4.0,保守==16.1.1)
- pip install service-identity (已装,当前最新版本=16.0.0) #pyopenssl ->secure sockets layer
- pip install requests (已装,当前最新版本2.11.1,保守2.9.1)
- pip install autobahn0.13.0(当前最新版本0.16.0,保守==0.13.0) #WebSocket client & server library, WAMP real-time framework
- pip install chardet (当前最新版本==2.3.0) #Universal encoding detector for Python 2 and 3
- pip install marshmallow2.8.0 (当前最新版本2.10.0,保守==2.8.0) #serializer
- pip install marshmallow_jsonapi (当前最新版本0.8.0,保守0.8.0) #jsonapi serializer
- pip install parse (当前最新版本==1.6.6) #format的反过程
- pip install ndg-httpsclient (当前最新版本0.4.2,保守0.4.0) #enhanced HTTPS support for httplib and urllib
- pip install retrying (当前最新版本1.3.3,保守1.3.3) # retry装饰器,可以指定时间多次执行函数
- pip install django1.9.4 (当前最新版本1.10.1,保守==1.9.4)
- pip install toposort (当前最新版本1.4,保守1.4) #Implements a topological sort algorithm.
- pip install mock (当前最新版本2.0.0,保守2.0.0) #using for test
- Dulwich’s MySQL backend
- pip install mysql-python==1.2.5
- pip install dulwich==0.13.0
- pip install portia2code==0.0.6 #portia产生的json转换为scrape的spider
- pip install -e /root/git/portia_nui/slyd
- 【slybot/requirements.txt】
- pip install scrapy (当前最新版本1.1.2,保守1.1.0)
- pip install scrapely (当前最新版本==0.12.0)
- pip install loginform (当前最新版本1.2.0,保守1.0)
- pip install lxml (当前最新版本3.6.4,保守3.6.0)
- pip install dateparser (当前最新版本0.4.0,保守0.3.5)
- pip install python-dateutil2.4.2 (当前最新版本2.5.3,保守==2.4.2)
- pip install jsonschema (已装)
- pip install six (已装)
- pip install scrapyjs (当前最新版本0.2,保守0.1.1) #JavaScript support for Scrapy using Splash
- pip install page_finder (当前最新版本==0.1.2)
- pip install page_clustering (当前最新版本==0.0.1) #Online k-means clustering of web pages
- pip install -e /root/git/portia_nui/slybot
六、 ember.js依赖安装
- cd /root/git/portia/portiaui
- npm install
- cd node_modules/ember-cli
- npm install
- cd …/…
- bower install