使用socket.io和Docker不能获得超过980个连接

Question

我无法使用Docker将我的简单Socket.io应用程序扩展到大约980个并发连接。但是，如果我在macOS Sierra 10.12.6上本地运行它，我可以获得3000多个连接。我已经包含了一个简单的SocketIO应用程序的回购，我正在用它进行测试：https://github.com/gsccheng/simple-socketIo-app

我的Docker-for Mac配置为4个CPU和5 GB内存。版本是

Version 17.09.0-ce-mac35 (19611)
Channel: stable
a98b7c1b7c

我正在使用火炮版本1.6.0-9对其进行加载测试

$ artillery run load-test.yaml

我展示了一些设置的冗余配置（以显示它们已经被考虑过）。以下是我复制的步骤。

$ docker build . -t socket-test
$ docker run -p 8000:8000 -c 1024 -m 4096M --privileged --ulimit nofile=9000:9000 -it test-socket:latest /bin/sh
#> DEBUG=* npm start

最多约980个连接，我将获得如下日志：

Connected to Socket!
  socket.io:client writing packet {"type":2,"data":["news",{"hello":"world"}],"nsp":"/"} +0ms
  socket.io-parser encoding packet {"type":2,"data":["news",{"hello":"world"}],"nsp":"/"} +0ms
  socket.io-parser encoded {"type":2,"data":["news",{"hello":"world"}],"nsp":"/"} as 2["news",{"hello":"world"}] +0ms
  engine:socket sending packet "message" (2["news",{"hello":"world"}]) +0ms
  socket.io:socket joined room 0ohCcHMWYASnfRgJAAPS +0ms
  engine:ws received "2" +5ms
  engine:socket packet +0ms
  engine:socket got ping +0ms
  engine:socket sending packet "pong" (undefined) +0ms
  engine:socket flushing buffer to transport +0ms
  engine:ws writing "3" +0ms
  engine upgrading existing transport +2ms
  engine:socket might upgrade socket transport from "polling" to "websocket" +0ms
  engine intercepting request for path "/socket.io/" +2ms
  engine handling "GET" http request "/socket.io/?EIO=3&transport=polling&t=Ly8pfqL&b64=1&sid=0ohCcHMWYASnfRgJAAPS" +0ms
  engine setting new request for existing client +0ms
  engine:polling setting request +0ms
  engine:socket flushing buffer to transport +0ms
  engine:polling writing "28:42["news",{"hello":"world"}]" +0ms
  engine:socket executing batch send callback +1ms
  engine:ws received "2probe" +4ms
  engine:ws writing "3probe" +0ms
  engine intercepting request for path "/socket.io/" +4ms
  engine handling "GET" http request "/socket.io/?EIO=3&transport=polling&t=Ly8pfqV&b64=1&sid=0ohCcHMWYASnfRgJAAPS" +0ms
  engine setting new request for existing client +0ms
  engine:polling setting request +0ms
  engine:socket writing a noop packet to polling for fast upgrade +10ms
  engine:polling writing "1:6" +0ms
  engine:ws received "5" +2ms
  engine:socket got upgrade packet - upgrading +0ms
  engine:polling closing +0ms
  engine:polling transport discarded - closing right away +1ms
  engine:ws received "2" +20ms
  engine:socket packet +0ms
  engine:socket got ping +0ms
  engine:socket sending packet "pong" (undefined) +0ms
  engine:socket flushing buffer to transport +1ms
  engine:ws writing "3" +0ms
  engine intercepting request for path "/socket.io/" +1ms
  engine handling "GET" http request "/socket.io/?EIO=3&transport=polling&t=Ly8pfr1&b64=1" +0ms
  engine handshaking client "6ccAiZwbvrchxZEiAAPT" +0ms
  engine:socket sending packet "open" ({"sid":"6ccAiZwbvrchxZEiAAPT","upgrades":["websocket"],"pingInterval":25000,"pingTimeout":60000}) +0ms
  engine:socket sending packet "message" (0) +0ms
  engine:polling setting request +0ms
  engine:socket flushing buffer to transport +0ms
  engine:polling writing "97:0{"sid":"6ccAiZwbvrchxZEiAAPT","upgrades":["websocket"],"pingInterval":25000,"pingTimeout":60000}2:40" +0ms
  engine:socket executing batch send callback +0ms
  socket.io:server incoming connection with id 6ccAiZwbvrchxZEiAAPT +0ms
  socket.io:client connecting to namespace / +1ms
  socket.io:namespace adding socket to nsp / +0ms
  socket.io:socket socket connected - writing packet +0ms
  socket.io:socket joining room 6ccAiZwbvrchxZEiAAPT +0ms
  socket.io:socket packet already sent in initial handshake +0ms
Connected to Socket!

在大约980个连接时，我将开始看到这些断开连接的事件：

disconnected to Socket!
transport close
  engine intercepting request for path "/socket.io/" +27ms
  engine handling "GET" http request "/socket.io/?EIO=3&transport=polling&t=Ly8pg1T&b64=1" +0ms
  engine handshaking client "C-pdSXFCbwQaTeYLAAPh" +0ms
  engine:socket sending packet "open" ({"sid":"C-pdSXFCbwQaTeYLAAPh","upgrades":["websocket"],"pingInterval":25000,"pingTimeout":60000}) +0ms
  engine:socket sending packet "message" (0) +0ms
  engine:polling setting request +0ms
  engine:socket flushing buffer to transport +0ms
  engine:polling writing "97:0{"sid":"C-pdSXFCbwQaTeYLAAPh","upgrades":["websocket"],"pingInterval":25000,"pingTimeout":60000}2:40" +0ms
  engine:socket executing batch send callback +0ms
  socket.io:server incoming connection with id C-pdSXFCbwQaTeYLAAPh +0ms
  socket.io:client connecting to namespace / +0ms
  socket.io:namespace adding socket to nsp / +0ms
  socket.io:socket socket connected - writing packet +1ms
  socket.io:socket joining room C-pdSXFCbwQaTeYLAAPh +0ms
  socket.io:socket packet already sent in initial handshake +0ms
Connected to Socket!
  socket.io:client writing packet {"type":2,"data":["news",{"hello":"world"}],"nsp":"/"} +0ms
  socket.io-parser encoding packet {"type":2,"data":["news",{"hello":"world"}],"nsp":"/"} +0ms
  socket.io-parser encoded {"type":2,"data":["news",{"hello":"world"}],"nsp":"/"} as 2["news",{"hello":"world"}] +0ms
  engine:socket sending packet "message" (2["news",{"hello":"world"}]) +0ms
  socket.io:socket joined room C-pdSXFCbwQaTeYLAAPh +0ms
  engine intercepting request for path "/socket.io/" +13ms
  engine handling "POST" http request "/socket.io/?EIO=3&transport=polling&t=Ly8pg1g&b64=1&sid=C-pdSXFCbwQaTeYLAAPh" +0ms
  engine setting new request for existing client +1ms
  engine:polling received "1:1" +0ms
  engine:polling got xhr close packet +0ms
  socket.io:client client close with reason transport close +0ms
  socket.io:socket closing socket - reason transport close +1ms
disconnected to Socket!

 engine:ws writing "3" +0ms
  engine:ws received "2" +42ms
  engine:socket packet +0ms
  engine:socket got ping +0ms
  engine:socket sending packet "pong" (undefined) +1ms
  engine:socket flushing buffer to transport +0ms
  engine:ws writing "3" +0ms
  engine:ws received "2" +4ms
  engine:socket packet +0ms
  engine:socket got ping +0ms
  engine:socket sending packet "pong" (undefined) +0ms
  engine:socket flushing buffer to transport +0ms
  engine:ws writing "3" +0ms
  engine:ws received "2" +45ms
  engine:socket packet +0ms
  engine:socket got ping +0ms
  engine:socket sending packet "pong" (undefined) +0ms
  engine:socket flushing buffer to transport +0ms
  engine:ws writing "3" +0ms
  engine:ws received "2" +7ms
  engine:socket packet +0ms
  engine:socket got ping +0ms
  engine:socket sending packet "pong" (undefined) +0ms
  engine:socket flushing buffer to transport +0ms
  engine:ws writing "3" +0ms

COPY limits.conf /etc/security/
COPY sysctl.conf /etc/
COPY rc.local /etc/
COPY common-session /etc/pam.d/
COPY common-session-noninteractive /etc/pam.d/
COPY supervisord.conf /etc/supervisor/

$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
file size               (blocks, -f) unlimited
max locked memory       (kbytes, -l) unlimited
max memory size         (kbytes, -m) unlimited
open files                      (-n) 64000
pipe size            (512 bytes, -p) 1
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 2048
virtual memory          (kbytes, -v) unlimited

当我将nofiles的限制降低到容器的500时，我看到我的应用程序断开连接似乎以同样的方式失败。当我增加或减少我的内存和CPU的一半/双。我没有看到任何不同的行为，所以这似乎不是问题。

Answer 1

在本地应用程序的网络路径和在Docker for Mac中运行的应用程序之间有很大的不同。

mac上的应用程序路径是通过环回界面直接进入的：

          mac  
client -> lo -> nodejs

当将Docker用于Mac时，in的路径包括更多跳数，并包括两个用户域代理进程，Mac上的vpnkit和docker-proxy，它们接受转发端口上的TCP连接，并将数据转发到：

      mac               |                 vm                   |  container    
client -> lo -> vpnkit -> if -> docker-proxy -> NAT -> bridge -> if -> nodejs

尝试使用一个网络可以直接访问mac的VM，看看vpnkit是否有明显的不同。

  mac         |                vm                    |  container
client -> if -> if -> docker-proxy -> NAT -> bridge -> if -> nodejs

您还可以通过将容器接口直接附加到VM网络来删除docker-proxy，这样容器就不需要端口映射(-p)。这可以通过将macvlan接口映射到容器或将容器放置在连接到VM网络的桥上来实现。这是一个我用于桥接网络的流浪设置。

  mac         |  container   <- there is a little vm here, but minimal. 
client -> if -> if -> nodejs

  mac         |      vm       |  container
client -> if -> if -> bridge -> if -> nodejs

一旦您摆脱了网络差异，那么我将更详细地关注VM和容器的调优。我想你应该会看到VM减少10-20%，而不是66%。