scrapydweb

安装python3.6.5

1. 安装依赖
   yum -y install python-devel openssl-devel bzip2-devel zlib-devel expat-devel ncurses-devel sqlite-devel gdbm-devel xz-devel tk-devel readline-devel gcc

2. 下载python3.6.5.tgz
   /usr/local/soft> wget https://www.python.org/ftp/python/3.6.5/Python-3.6.5.tgz

3. 解压
   tar -xzvf Python3.6.5.tgz

4. 建立一个空文件夹，用于存放python3程序
   mkdir /usr/local/python3

5. 执行配置文件，编译，安装

/usr/local/soft> cd Python-3.6.5
/usr/local/soft/Python-3.6.5> ./configure --prefix=/usr/local/python3
make && make install

5. 建立软连接

ln -s /usr/local/python3/bin/python3.6 /usr/bin/python3
ln -s /usr/local/python3/bin/pip3.6 /usr/bin/pip3

6. 大功告成

安装scrapyd-scrapydweb

scrapyd用于部署scrapy，scrapydweb用户监控爬虫程序。scrapyd运行于爬虫服务器，scrapydweb一般运行于开发机

安装scrapyd

1. 直接通过pip3 install scrapyd -i "https://pypi.douban.com/simple"即可安装成功

2. vi /usr/local/python3/lib/python3.6/site-packages/scrapyd/default_scrapyd.conf修改配置文件以便远程客户端访问

[scrapyd]
eggs_dir    = eggs
logs_dir    = logs
items_dir   =
jobs_to_keep = 5
dbs_dir     = dbs
max_proc    = 0
max_proc_per_cpu = 4
finished_to_keep = 100
poll_interval = 5.0
bind_address = 0.0.0.0  # 只需要修改此行就好
http_port   = 6800
debug       = off
runner      = scrapyd.runner
application = scrapyd.app.application
launcher    = scrapyd.launcher.Launcher
webroot     = scrapyd.website.Root

3. 此时可以尝试前台运行/usr/local/python3/bin> ./scrapyd

4. 上一步中如果没有报错，基本说明没有问题，同时也可以在客户端浏览器中输入地址0.0.0.0:6800验证能否访问

5. 由于是在前台运行，服务器窗口关闭scrapyd就会关闭，现在来配置后台运行

新建文件
touch /etc/init.d/scrapyd

授可执行权限
sudo chmod 755 /etc/init.d/scrapyd

编辑`vi scrapyd`

PORT=6800
HOME="/var/scrapyd"
BIN="/usr/local/python3.5.2/bin/scrapyd"
pid=`netstat -lnopt | grep :$PORT | awk '/python/{gsub(/\/python/,"",$7);print $7;}'`
start() {
   if [ -n "$pid" ]; then
      echo "server already start,pid:$pid"
      return 0
   fi
   cd $HOME
   nohup $BIN >> $HOME/scrapyd.log 2>&1 &
   echo "start at port:$PORT"
}
stop() {
   if [ -z "$pid" ]; then
      echo "not find program on port:$PORT"
      return 0
   fi
   #结束程序，使用讯号2，如果不行可以尝试讯号9强制结束
   kill -9 $pid
   echo "kill program use signal 9,pid:$pid"
}
status() {
   if [ -z "$pid" ]; then
      echo "not find program on port:$PORT"
   else
      echo "program is running,pid:$pid"
   fi
}
case $1 in
   start)
      start
   ;;
   stop)
      stop
   ;;
   status)
      status
   ;;
   *)
      echo "Usage: {start|stop|status}"
   ;;
esac
exit 0

6. 新建目录mkdir /var/scrapyd

7. 命令操作
   service scrapyd {start|stop|status}

8. 设置为系统启动项
   sysv-rc-conf scrapyd on/off
   或者
   chkconfig --add/del scrapyd

9. 大功告成