nginx-upsync-module使用

编译模块到nginx

安装nginx依赖以及其他工具

apt-get install git
apt-get build-dep nginx

下载nginx-upsync-module源码

git clone https://github.com/weibocom/nginx-upsync-module.git

下载nginx源码

wget 'http://nginx.org/download/nginx-1.8.0.tar.gz'
tar -xzvf nginx-1.8.0.tar.gz
cd nginx-1.8.0/

开始编译

./configure --add-module=/path/to/nginx-upsync-module
make
make install

启动服务

测试例子中consul与nginx在同一服务器

启动consul

wget https://releases.hashicorp.com/consul/0.6.4/consul_0.6.4_linux_amd64.zip
unzip consul_0.6.4_linux_amd64.zip
./consul agent -advertise=x.x.x.x -client=0.0.0.0 -dev

创建nginx配置文件

mkdir -p /usr/local/nginx/conf/servers
/usr/local/nginx/conf/nginx.conf
events {
  worker_connections  4096;  ## Default: 1024
}

http {
    upstream test {
        # fake server otherwise ngx_http_upstream will report error when startup
        server 127.0.0.1:11111;

        # all backend server will pull from consul when startup and will delete fake server
        upsync 127.0.0.1:8500/v1/kv/upstreams/test upsync_timeout=6m upsync_interval=500ms upsync_type=consul strong_dependency=off;
        upsync_dump_path /usr/local/nginx/conf/servers/servers_test.conf;
    }

    server {
        listen 8080;

        location = / {
            proxy_pass http://test;
        }

        location = /upstream_show {
            upstream_show;
        }

    }
}

测试

fedora 虚拟机中[172.18.18.10]使用docker创建nginx服务

docker run --name=nginx1 -p 1234:80 -d nginx
docker run --name=nginx2 -p 1235:80 -d nginx
docker run --name=nginx3 -p 1236:80 -d nginx
docker run --name=nginx4 -p 1237:80 -d nginx
docker run --name=nginx5 -p 1238:80 -d nginx
docker run --name=nginx6 -p 1239:80 -d nginx

添加服务

curl -X PUT -d '{"weight":1, "max_fails":2, "fail_timeout":10}' http://127.0.0.1:8500/v1/kv/upstreams/test/172.18.18.10:1234
curl -X PUT -d '{"weight":1, "max_fails":2, "fail_timeout":10}' http://127.0.0.1:8500/v1/kv/upstreams/test/172.18.18.10:1235
curl -X PUT -d '{"weight":1, "max_fails":2, "fail_timeout":10}' http://127.0.0.1:8500/v1/kv/upstreams/test/172.18.18.10:1236
curl -X PUT -d '{"weight":1, "max_fails":2, "fail_timeout":10}' http://127.0.0.1:8500/v1/kv/upstreams/test/172.18.18.10:1237
curl -X PUT -d '{"weight":1, "max_fails":2, "fail_timeout":10}' http://127.0.0.1:8500/v1/kv/upstreams/test/172.18.18.10:1238
curl -X PUT -d '{"weight":1, "max_fails":2, "fail_timeout":10}' http://127.0.0.1:8500/v1/kv/upstreams/test/172.18.18.10:1239

并修改每个nginx的默认显示页面，以方便展示负载轮训情况

docker exec -it nginx1 sh
echo "nginx1">>/usr/share/nginx/html/index.html
docker exec -it nginx2 sh
echo "nginx2">>/usr/share/nginx/html/index.html
docker exec -it nginx3 sh
echo "nginx3">>/usr/share/nginx/html/index.html
docker exec -it nginx4 sh
echo "nginx4">>/usr/share/nginx/html/index.html
docker exec -it nginx5 sh
echo "nginx5">>/usr/share/nginx/html/index.html

回到nginx负载服务器。将服务添加到consul之后查/usr/local/nginx/conf/servers/servers_test.conf内容,或者通过浏览器查看服务情况

cat /usr/local/nginx/conf/servers/servers_test.conf
server 172.18.18.10:1235 weight=1 max_fails=2 fail_timeout=10s;
server 172.18.18.10:1236 weight=1 max_fails=2 fail_timeout=10s;
server 172.18.18.10:1234 weight=1 max_fails=2 fail_timeout=10s;
server 172.18.18.10:1237 weight=1 max_fails=2 fail_timeout=10s;
server 172.18.18.10:1238 weight=1 max_fails=2 fail_timeout=10s;
server 172.18.18.10:1239 weight=1 max_fails=2 fail_timeout=10s;

或者浏览器打开http://172.18.21.2:8080/upstream_show?test
显示内容如下：

Upstream name: test; Backend server count: 5
        server 172.18.18.10:1235 weight=1 max_fails=2 fail_timeout=10s;
        server 172.18.18.10:1236 weight=1 max_fails=2 fail_timeout=10s;
        server 172.18.18.10:1234 weight=1 max_fails=2 fail_timeout=10s;
        server 172.18.18.10:1237 weight=1 max_fails=2 fail_timeout=10s;
        server 172.18.18.10:1238 weight=1 max_fails=2 fail_timeout=10s;

总结

此模块只修改upstream 中的缓存信息，不能修改或添加其他配置

测试中遇到的问题

在添加服务时出现如下错误，导致服务添加不能实时进行，大约需要3分钟左右时间。

consul日志：

...
2016/03/22 05:34:42 [DEBUG] http: Request GET /v1/kv/upstreams/test?recurse&index=169 (149.023µs) from=127.0.0.1:38853
    2016/03/22 05:34:43 [DEBUG] http: Request GET /v1/kv/upstreams/test?recurse&index=169 (146.759µs) from=127.0.0.1:38854
    2016/03/22 05:34:45 [DEBUG] http: Request GET /v1/kv/upstreams/test?recurse&index=169 (149.853µs) from=127.0.0.1:38855
    2016/03/22 05:34:46 [DEBUG] http: Request GET /v1/kv/upstreams/test?recurse&index=169 (111.46µs) from=127.0.0.1:38856
    2016/03/22 05:34:48 [DEBUG] http: Request GET /v1/kv/upstreams/test?recurse&index=169 (142.696µs) from=127.0.0.1:38857
    2016/03/22 05:34:48 [DEBUG] http: Request GET /v1/kv/upstreams/test?recurse&index=169 (112.089µs) from=127.0.0.1:38858
    2016/03/22 05:34:49 [DEBUG] http: Request GET /v1/kv/upstreams/test?recurse&index=169 (114.29µs) from=127.0.0.1:38859
    2016/03/22 05:34:50 [DEBUG] http: Request GET /v1/kv/upstreams/test?recurse&index=169 (148.245µs) from=127.0.0.1:38860
...

nginx日志

...
2016/03/22 05:35:09 [error] 18879#0: recv() failed (104: Connection reset by peer)
2016/03/22 05:35:09 [error] 18879#0: upsync_recv: recv error with upsync_server: 127.0.0.1:8500
2016/03/22 05:35:10 [error] 18879#0: recv() failed (104: Connection reset by peer)
2016/03/22 05:35:10 [error] 18879#0: upsync_recv: recv error with upsync_server: 127.0.0.1:8500
2016/03/22 05:35:11 [error] 18879#0: recv() failed (104: Connection reset by peer)
2016/03/22 05:35:11 [error] 18879#0: upsync_recv: recv error with upsync_server: 127.0.0.1:8500
2016/03/22 05:35:13 [error] 18879#0: recv() failed (104: Connection reset by peer)
2016/03/22 05:35:13 [error] 18879#0: upsync_recv: recv error with upsync_server: 127.0.0.1:8500
2016/03/22 05:35:13 [error] 18879#0: recv() failed (104: Connection reset by peer)
2016/03/22 05:35:13 [error] 18879#0: upsync_recv: recv error with upsync_server: 127.0.0.1:8500
2016/03/22 05:35:14 [error] 18879#0: recv() failed (104: Connection reset by peer)
...

问题现象

当添加一个服务时，出现此问题，新增服务不能及时添加到负载中，不影响运行正常的服务。
此时再往consul继续添加一个服务时，可能会导致此错误终止，并能成功添加当前两条服务记录。