支持nginx_tcp_proxy_module的nginx安装(1.9.9版本)

安装背景

1、nginx服务器需要支持tcp的负载均衡的配置；

2、高版本的nginx(1.10以上?)不支持第三方模块nginx_tcp_proxy_module的安装；

3、高版本的nginx有自带tcp负载均衡配置(stream模块)，但是health_check功能只供商业用户使用，第三方tcp心跳检测模块还没有适配上(2018.5)；

4、低版本的nginx可以用第三方模块nginx_tcp_proxy_module实现tcp负载均衡并且支持用check进行心跳检测，打补丁安装第三方模块nginx_tcp_proxy_module来实现tcp负载均衡和心跳检测；

高版本tcp(NO)(第三方模块不支持) --> 高版本stream(NO)(health_check暂不可用，未见适配的第三方心跳检测模块) --> 低版本的nginx(安装第三方模块实现tcp负载均衡)

5、安装的nginx版本：1.9.9

nginx安装

1、准备环境

yum install vim

yum install lsof

yum install ntpdate

yum -y install gcc-c++

yum install -y unzip zip

yum -y install patch 

2、安装pcre

获取tar包

由于服务器不能连接外网，所以需要下载到本地PC，然后再传上服务器

去官网(http://www.pcre.org/)下载对应版本的tar包到本地PC

在服务器中，进入到对应目录(/opt/app_install)下面，输入rz，在本地找到对应的tar包，上传到服务器

注意这里不要在本地windows解压然后用Xftp上传,否则安装的时候会报错装不了(踩了很久的坑)

解压并安装

tar -xvf pcre-8.37.tar.gz

cd pcre-8.37

./configure

make & make install

3、下载nginx及安装时需要的相应模块

nginx

去官网下载(http://nginx.org/download/)下载nginx对应版本的压缩包并上传，步骤同前

目录(/opt/app_install)下解压

tar -xvf nginx-1.9.9.tar.gz

相应模块

在github上搜索然后下载相应的项目压缩包，然后同样方式传到服务器上并解压(对这些模块进行统一的路径管理)

nginx_tcp_proxy_module-master
nginx_upstream_check_module-master
nginx-upstream-consistent-master
headers-more-nginx-module-master

目录(/opt/app_install)下解压

unzip file

4、为nginx_tcp_proxy_module打补丁

直接安装nginx会报错误，查阅知道有两个问题

打补丁

安装错误提示

../nginx_tcp_proxy_module-master/ngx_tcp_upstream.c: In function 'ngx_tcp_upstream_connect':
../nginx_tcp_proxy_module-master/ngx_tcp_upstream.c:374: error: 'ngx_peer_connection_t' has no member named 'check_index'
../nginx_tcp_proxy_module-master/ngx_tcp_upstream.c:374: error: 'NGX_INVALID_CHECK_INDEX' undeclared (first use in this function)
../nginx_tcp_proxy_module-master/ngx_tcp_upstream.c:374: error: (Each undeclared identifier is reported only once
../nginx_tcp_proxy_module-master/ngx_tcp_upstream.c:374: error: for each function it appears in.)
../nginx_tcp_proxy_module-master/ngx_tcp_upstream.c:375: error: 'ngx_peer_connection_t' has no member named 'check_index'
../nginx_tcp_proxy_module-master/ngx_tcp_upstream.c: In function 'ngx_tcp_upstream_next':
../nginx_tcp_proxy_module-master/ngx_tcp_upstream.c:520: error: 'ngx_peer_connection_t' has no member named 'check_index'
../nginx_tcp_proxy_module-master/ngx_tcp_upstream.c:520: error: 'NGX_INVALID_CHECK_INDEX' undeclared (first use in this function)
../nginx_tcp_proxy_module-master/ngx_tcp_upstream.c:521: error: 'ngx_peer_connection_t' has no member named 'check_index'
../nginx_tcp_proxy_module-master/ngx_tcp_upstream.c:522: error: 'ngx_peer_connection_t' has no member named 'check_index'
../nginx_tcp_proxy_module-master/ngx_tcp_upstream.c: In function 'ngx_tcp_upstream_finalize_session':
../nginx_tcp_proxy_module-master/ngx_tcp_upstream.c:576: error: 'ngx_peer_connection_t' has no member named 'check_index'
../nginx_tcp_proxy_module-master/ngx_tcp_upstream.c:576: error: 'NGX_INVALID_CHECK_INDEX' undeclared (first use in this function)
../nginx_tcp_proxy_module-master/ngx_tcp_upstream.c:577: error: 'ngx_peer_connection_t' has no member named 'check_index'
../nginx_tcp_proxy_module-master/ngx_tcp_upstream.c:578: error: 'ngx_peer_connection_t' has no member named 'check_index'
make[1]: *** [objs/addon/nginx_tcp_proxy_module-master/ngx_tcp_upstream.o] Error 1
make[1]: Leaving directory `/opt/apps_install/nginx-1.9.9'
make: *** [build] Error 2

原因

由于未按照github上nginx_tcp_proxy_module模块的安装提示打补丁导致(以后应该认真阅读README文件)

解决办法

进入nginx解压目录

cd /opt/apps_intsall/nginx-1.9.9

打补丁

patch -p1 < /opt/app_install/nginx_tcp_proxy_module-master/tcp.patch

补丁位置要定位到nginx_tcp_proxy_module-master解压文件下的补丁tcp.patch

补丁结果

patching file src/core/ngx_log.c
Hunk #1 FAILED at 66.
1 out of 1 hunk FAILED -- saving rejects to file src/core/ngx_log.c.rej
patching file src/core/ngx_log.h
Hunk #1 FAILED at 29.
Hunk #2 FAILED at 37.
2 out of 2 hunks FAILED -- saving rejects to file src/core/ngx_log.h.rej
patching file src/event/ngx_event_connect.h
Hunk #1 succeeded at 33 (offset 1 line).
Hunk #2 succeeded at 44 with fuzz 2 (offset 1 line).

解决链接

https://github.com/yaoweibin/nginx_tcp_proxy_module/blob/master/README.md  
 

修改头文件

安装错误提示

In file included from ../nginx_tcp_proxy_module-master/ngx_tcp.h:32,
                 from ../nginx_tcp_proxy_module-master/ngx_tcp.c:5:
../nginx_tcp_proxy_module-master/ngx_tcp_upstream.h:144: error: expected specifier-qualifier-list before 'ngx_resolver_addr_t'
make[1]: *** [objs/addon/nginx_tcp_proxy_module-master/ngx_tcp.o] Error 1
make[1]: Leaving directory `/opt/apps_install/nginx-1.9.9'
make: *** [build] Error 2

原因

nginx 1.9版本起addrs的结构体由ngx_addr_t 改为ngx_resolver_addr_t
定义文件位置：/opt/apps_install/nginx-1.9.9/src/core/ngx_resolver.h(但是查看nginx1.9.9的头文件发现这个版本并没有改，还是ngx_addr_t)
在nginx_tcp_proxy_module第三方模块的头文件ngx_tcp_upstream.h中写法仍然是ngx_addr_t，由于版本原因不匹配

解决办法

官方说新的nginx_tcp_proxy_module的master分支已经解决了这个问题，在头文件中加入了判断

#if (nginx_version) >= 1005008
     #if (nginx_version) >= 1009001
         ngx_resolver_addr_t *addrs;
     #else
         ngx_addr_t *addrs;
     #endif
     #else
         in_addr_t *addrs;
#endif

但是这里还是不正确，感觉是版本号判断不准确(因为1.9.9版本还没有改,可能对于1.9版本判断弄错了?这里我也不知道nginx_version的具体数值是什么意思)，这里判断之后应该是用了ngx_resolver_addr_t，最后造成了不匹配

所以这里直接把ngx_resolver_addr_t改成ngx_addr_t，即不管啥版本都用ngx_addr_t吧，总之对于1.9.9现在的安装版本是适用的。

步骤

修改第三方模块包里的头文件

vim /opt/apps_install/nginx_tcp_proxy_module-master/ngx_tcp_upstream.h

144行：ngx_resolver_addr_t -->  ngx_addr_t

解决链接

https://github.com/yaoweibin/nginx_tcp_proxy_module/issues/138

5、安装nginx

cd /opt/app_install/nginx-1.9.9/

./configure --prefix=/opt/apps/nginx-1.9.9 --conf-path=/data/conf/nginx/nginx.conf --with-stream --with-mail --with-mail_ssl_module --with-http_realip_module --with-http_stub_status_module --with-http_gunzip_module --with-http_sub_module --with-http_ssl_module --with-pcre=../pcre-8.37 --add-module=../nginx_tcp_proxy_module-master --add-module=../nginx_upstream_check_module-master --add-module=../nginx-upstream-consistent-master --add-module=../headers-more-nginx-module-master

make & make install

6、软连接配置及验证

/opt/apps/nginx-1.9.9目录下

ln -s /data/conf/nginx /opt/apps/nginx-1.9.9/conf

/opt/apps目录下

ln -s /opt/apps/nginx-1.9.9 /opt/apps/nginx

/opt目录下(先建好/data下的相应目录)

ln -s /data/conf /opt/conf
ln -s /data/logs /opt/logs
ln -s /opt/apps/nginx /opt/nginx

软连接效果

[@bx16-77-219 ~]# ll /opt/
total 12
drwxr-xr-x  3 root root 4096 May 28 12:00 apps
drwxr-xr-x  7 root root 4096 May 28 11:49 apps_install
lrwxrwxrwx  1 root root   10 May 28 12:14 conf -> /data/conf
lrwxrwxrwx  1 root root   10 May 28 12:14 logs -> /data/logs
lrwxrwxrwx  1 root root   15 May 28 12:08 nginx -> /opt/apps/nginx
drwxr-xr-x. 7 root root 4096 Apr 20 18:38 nofalcon-agent-deploy

[@bx16-77-219 ~]# ll /opt/apps
total 4
lrwxrwxrwx  1 root root   21 May 28 12:00 nginx -> /opt/apps/nginx-1.9.9
drwxr-xr-x 10 root root 4096 May 28 12:21 nginx-1.9.9

[@bx16-77-219 ~]# ll /opt/nginx/
total 32
drwx------ 2 nobody root 4096 May 28 11:56 client_body_temp
lrwxrwxrwx 1 root   root   16 May 28 12:21 conf -> /data/conf/nginx
drwx------ 2 nobody root 4096 May 28 11:56 fastcgi_temp
drwxr-xr-x 2 root   root 4096 May 28 11:54 html
drwxr-xr-x 2 root   root 4096 May 28 11:59 logs
drwx------ 2 nobody root 4096 May 28 11:56 proxy_temp
drwxr-xr-x 2 root   root 4096 May 28 11:54 sbin
drwx------ 2 nobody root 4096 May 28 11:56 scgi_temp
drwx------ 2 nobody root 4096 May 28 11:56 uwsgi_temp