Jenkins、MiniKube和Kubernetes插件-动态产卵奴隶

Question

问题

我已经将Csanchez的jenkins-kubernetes插件（版本0.12）部署到本地的Minikube/Kubernetes环境中。在Jenkins UI中配置pod模板和容器模板时，pod会自动生成并处理简单的作业。然而，当POD模板和容器模板在管道脚本中定义时，Jenkins Master拒绝连接，称POD已经连接到Master。

环境

minikube版本:V0.20.0

Kubernetes版本:V1.7.1

詹金斯大师：詹金斯：2.74-阿尔卑斯（码头容器）

詹金斯奴隶：JNLP-奴隶:3.10-1-Alpine（码头容器）

操作系统:4.8.0-59-Generic#64-Ubuntu SMP Thu Jun29 19:38:34 UTC 2017 x86_64 x86_64 x86_64 gnu/linux

podTemplate(name: 'my-pod-template', label: 'maven', cloud: 
'kubernetes', namespace: 'mmidevops', containers: [
  containerTemplate(
    name: 'my-container-template', 
    image: 'grantericedwards/jenkinsci-jnlp-slave:3.10-1-alpine', 
    ttyEnabled: true, 
    alwaysPullImage: true,
    workingDir: '/home/jenkins'
  )
  ],
volumes: [
    hostPathVolume(mountPath: '/var/run/docker.sock', hostPath: '/var/run/docker.sock')
]
) {

  node('maven') {
    stage('Build a Maven project') {
      container('maven') {
          sh 'sleep 10'
      }
    }
  }
}

派生豆荚kubectl get豆荚--all-namespaces-w

NAMESPACE     NAME                          READY     STATUS    RESTARTS   AGE
kube-system   kube-addon-manager-minikube   1/1       Running   9          7d
kube-system   kube-dns-1301475494-zp3tg     3/3       Running   27         7d
kube-system   kubernetes-dashboard-jlsk7    1/1       Running   9          7d
mmidevops     jenkins-83791151-msdcw        1/1       Running   30         6d
mmidevops   my-pod-template-nfdgh-9k7qh   0/2       Pending   0         0s
mmidevops   my-pod-template-nfdgh-9k7qh   0/2       Pending   0         0s
mmidevops   my-pod-template-nfdgh-9k7qh   0/2       ContainerCreating   0         0s
mmidevops   my-pod-template-nfdgh-9k7qh   2/2       Running   0         6s
mmidevops   my-pod-template-nfdgh-9k7qh   1/2       Error     0         9s

从吊舱的日志

INFO: Setting up slave: my-pod-template-nfdgh-9k7qh
Aug 28, 2017 11:35:20 AM hudson.remoting.jnlp.Main$CuiListener <init>
INFO: Jenkins agent is running in headless mode.
Aug 28, 2017 11:35:20 AM hudson.remoting.Engine startEngine
WARNING: No Working Directory. Using the legacy JAR Cache location: /home/jenkins/.jenkins/cache/jars
Aug 28, 2017 11:35:20 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Locating server among [http://172.17.0.2:8080/]
Aug 28, 2017 11:35:20 AM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve
INFO: Remoting server accepts the following protocols: [JNLP4-connect, JNLP-connect, Ping, JNLP2-connect]
Aug 28, 2017 11:35:20 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Agent discovery successful
  Agent address: 172.17.0.2
  Agent port:    50000
  Identity:      74:24:d0:2e:7b:b7:9d:13:80:47:e5:fa:45:b3:85:15
Aug 28, 2017 11:35:20 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Handshaking
Aug 28, 2017 11:35:20 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connecting to 172.17.0.2:50000
Aug 28, 2017 11:35:21 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Trying protocol: JNLP4-connect
Aug 28, 2017 11:35:21 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Remote identity confirmed: 74:24:d0:2e:7b:b7:9d:13:80:47:e5:fa:45:b3:85:15
Aug 28, 2017 11:35:21 AM org.jenkinsci.remoting.protocol.impl.ConnectionHeadersFilterLayer onRecv
INFO: [JNLP4-connect connection to 172.17.0.2/172.17.0.2:50000] Local headers refused by remote: my-pod-template-nfdgh-9k7qh is already connected to this master. Rejecting this connection.
Aug 28, 2017 11:35:21 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Protocol JNLP4-connect encountered an unexpected exception
java.util.concurrent.ExecutionException: org.jenkinsci.remoting.protocol.impl.ConnectionRefusalException: my-pod-template-nfdgh-9k7qh is already connected to this master. Rejecting this connection.
at org.jenkinsci.remoting.util.SettableFuture.get(SettableFuture.java:223)
at hudson.remoting.Engine.innerRun(Engine.java:583)
at hudson.remoting.Engine.run(Engine.java:447)
Caused by: org.jenkinsci.remoting.protocol.impl.ConnectionRefusalException: my-pod-template-nfdgh-9k7qh is already connected to this master. Rejecting this connection.
at org.jenkinsci.remoting.protocol.impl.ConnectionHeadersFilterLayer.newAbortCause(ConnectionHeadersFilterLayer.java:377)
at org.jenkinsci.remoting.protocol.impl.ConnectionHeadersFilterLayer.onRecvClosed(ConnectionHeadersFilterLayer.java:432)
at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecvClosed(ProtocolStack.java:832)
at org.jenkinsci.remoting.protocol.FilterLayer.onRecvClosed(FilterLayer.java:287)
at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.onRecvClosed(SSLEngineFilterLayer.java:172)
at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecvClosed(ProtocolStack.java:832)
at org.jenkinsci.remoting.protocol.NetworkLayer.onRecvClosed(NetworkLayer.java:154)
at org.jenkinsci.remoting.protocol.impl.BIONetworkLayer.access$1500(BIONetworkLayer.java:48)
at org.jenkinsci.remoting.protocol.impl.BIONetworkLayer$Reader.run(BIONetworkLayer.java:247)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at hudson.remoting.Engine$1$1.run(Engine.java:98)
at java.lang.Thread.run(Thread.java:748)
Suppressed: java.nio.channels.ClosedChannelException
    ... 7 more

Aug 28, 2017 11:35:21 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connecting to 172.17.0.2:50000
Aug 28, 2017 11:35:21 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Server reports protocol JNLP4-plaintext not supported, skipping
Aug 28, 2017 11:35:21 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Protocol JNLP3-connect is not enabled, skipping
Aug 28, 2017 11:35:21 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Trying protocol: JNLP2-connect
Aug 28, 2017 11:35:21 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Protocol JNLP2-connect encountered an unexpected exception
java.util.concurrent.ExecutionException: org.jenkinsci.remoting.protocol.impl.ConnectionRefusalException: Server didn't accept the handshake: 
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at hudson.remoting.Engine.innerRun(Engine.java:583)
at hudson.remoting.Engine.run(Engine.java:447)

Caused by: org.jenkinsci.remoting.protocol.impl.ConnectionRefusalException: Server didn't accept the handshake: 
at org.jenkinsci.remoting.engine.JnlpProtocol2Handler.sendHandshake(JnlpProtocol2Handler.java:134)
at org.jenkinsci.remoting.engine.LegacyJnlpProtocolHandler$2.call(LegacyJnlpProtocolHandler.java:162)
at org.jenkinsci.remoting.engine.LegacyJnlpProtocolHandler$2.call(LegacyJnlpProtocolHandler.java:158)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at hudson.remoting.Engine$1$1.run(Engine.java:98)
at java.lang.Thread.run(Thread.java:748)

Aug 28, 2017 11:35:21 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connecting to 172.17.0.2:50000
Aug 28, 2017 11:35:21 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Trying protocol: JNLP-connect
Aug 28, 2017 11:35:21 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Protocol JNLP-connect encountered an unexpected exception
java.util.concurrent.ExecutionException: org.jenkinsci.remoting.protocol.impl.ConnectionRefusalException: Server didn't accept the handshake: 
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at hudson.remoting.Engine.innerRun(Engine.java:583)
at hudson.remoting.Engine.run(Engine.java:447)
Caused by: org.jenkinsci.remoting.protocol.impl.ConnectionRefusalException: Server didn't accept the handshake: 
at org.jenkinsci.remoting.engine.JnlpProtocol1Handler.sendHandshake(JnlpProtocol1Handler.java:121)
at org.jenkinsci.remoting.engine.LegacyJnlpProtocolHandler$2.call(LegacyJnlpProtocolHandler.java:162)
at org.jenkinsci.remoting.engine.LegacyJnlpProtocolHandler$2.call(LegacyJnlpProtocolHandler.java:158)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at hudson.remoting.Engine$1$1.run(Engine.java:98)
at java.lang.Thread.run(Thread.java:748)

Aug 28, 2017 11:35:21 AM hudson.remoting.jnlp.Main$CuiListener error
SEVERE: The server rejected the connection: None of the protocols were accepted
java.lang.Exception: The server rejected the connection: None of the protocols were accepted
at hudson.remoting.Engine.onConnectionRejected(Engine.java:644)
at hudson.remoting.Engine.innerRun(Engine.java:608)
at hudson.remoting.Engine.run(Engine.java:447)

詹金斯大师的日志

Aug 28, 2017 11:35:14 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision
INFO: Template: Kubernetes Pod Template
Aug 28, 2017 11:35:14 AM hudson.slaves.NodeProvisioner$StandardStrategyImpl apply
INFO: Started provisioning Kubernetes Pod Template from kubernetes with 1 executors. Remaining excess workload: 0
Aug 28, 2017 11:35:14 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud$ProvisioningCallback call
INFO: Created Pod: my-pod-template-nfdgh-9k7qh
Aug 28, 2017 11:35:14 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud$ProvisioningCallback call
INFO: Waiting for Pod to be scheduled (0/100): my-pod-template-nfdgh-9k7qh
Aug 28, 2017 11:35:15 AM hudson.TcpSlaveAgentListener$ConnectionHandler run
INFO: Accepted JNLP4-connect connection #38 from /172.17.0.5:54934
Aug 28, 2017 11:35:21 AM hudson.TcpSlaveAgentListener$ConnectionHandler run
INFO: Accepted JNLP4-connect connection #39 from /172.17.0.5:54944
Aug 28, 2017 11:35:21 AM org.jenkinsci.remoting.protocol.impl.ConnectionHeadersFilterLayer onRecv
INFO: [JNLP4-connect connection from 172.17.0.5/172.17.0.5:54944] Refusing headers from remote: my-pod-template-nfdgh-9k7qh is already connected to this master. Rejecting this connection.
Aug 28, 2017 11:35:21 AM hudson.TcpSlaveAgentListener$ConnectionHandler run
INFO: Accepted JNLP2-connect connection #40 from /172.17.0.5:54960
Aug 28, 2017 11:35:21 AM hudson.TcpSlaveAgentListener$ConnectionHandler run
INFO: Accepted JNLP-connect connection #41 from /172.17.0.5:54962
Aug 28, 2017 11:35:23 AM hudson.node_monitors.ResponseTimeMonitor$1 monitor
WARNING: Making my-pod-template-qdgzl-29t7l offline because it’s not responding
Aug 28, 2017 11:35:24 AM hudson.slaves.NodeProvisioner$2 run
INFO: Kubernetes Pod Template provisioning successfully completed. We have now 7 computer(s)
Aug 28, 2017 11:40:20 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate
INFO: Terminating Kubernetes instance for slave my-pod-template-nfdgh-9k7qh
Aug 28, 2017 11:40:20 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate
INFO: Terminated Kubernetes instance for slave my-pod-template-nfdgh-9k7qh
Aug 28, 2017 11:40:20 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate
INFO: Disconnected computer my-pod-template-nfdgh-9k7qh
Terminated Kubernetes instance for slave my-pod-template-nfdgh-9k7qh
Aug 28, 2017 11:40:20 AM jenkins.slaves.DefaultJnlpSlaveReceiver channelClosed
WARNING: Computer.threadPoolForRemoting [#41] for my-pod-template-nfdgh-9k7qh terminated
java.nio.channels.ClosedChannelException
    at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer.onReadClosed(ChannelApplicationLayer.java:208)
    at org.jenkinsci.remoting.protocol.ApplicationLayer.onRecvClosed(ApplicationLayer.java:222)
    at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecvClosed(ProtocolStack.java:832)
    at org.jenkinsci.remoting.protocol.FilterLayer.onRecvClosed(FilterLayer.java:287)
    at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.onRecvClosed(SSLEngineFilterLayer.java:181)
    at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.switchToNoSecure(SSLEngineFilterLayer.java:283)
    at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processWrite(SSLEngineFilterLayer.java:503)
    at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processQueuedWrites(SSLEngineFilterLayer.java:248)
    at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.doSend(SSLEngineFilterLayer.java:200)
    at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.doCloseSend(SSLEngineFilterLayer.java:213)
    at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.doCloseSend(ProtocolStack.java:800)
    at org.jenkinsci.remoting.protocol.ApplicationLayer.doCloseWrite(ApplicationLayer.java:173)
    at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer$ByteBufferCommandTransport.closeWrite(ChannelApplicationLayer.java:311)
    at hudson.remoting.Channel.close(Channel.java:1304)
    at hudson.remoting.Channel.close(Channel.java:1272)
    at hudson.slaves.SlaveComputer.closeChannel(SlaveComputer.java:708)
    at hudson.slaves.SlaveComputer.access$800(SlaveComputer.java:96)
    at hudson.slaves.SlaveComputer$3.run(SlaveComputer.java:626)
    at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

Aug 28, 2017 11:40:20 AM org.jenkinsci.plugins.workflow.job.WorkflowRun finish
INFO: TestPipeLine #94 completed: FAILURE

JNLP日志

Warning: JnlpProtocol3 is disabled by default, use JNLP_PROTOCOL_OPTS to alter the behavior
Warning: SECRET is defined twice in command-line arguments and the environment variable
Warning: AGENT_NAME is defined twice in command-line arguments and the environment variable
Aug 28, 2017 11:35:15 AM hudson.remoting.jnlp.Main createEngine
INFO: Setting up slave: my-pod-template-nfdgh-9k7qh
Aug 28, 2017 11:35:15 AM hudson.remoting.jnlp.Main$CuiListener <init>
INFO: Jenkins agent is running in headless mode.
Aug 28, 2017 11:35:15 AM hudson.remoting.Engine startEngine
WARNING: No Working Directory. Using the legacy JAR Cache location: /home/jenkins/.jenkins/cache/jars
Aug 28, 2017 11:35:15 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Locating server among [http://172.17.0.2:8080/]
Aug 28, 2017 11:35:15 AM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve
INFO: Remoting server accepts the following protocols: [JNLP4-connect, JNLP-connect, Ping, JNLP2-connect]
Aug 28, 2017 11:35:15 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Agent discovery successful
  Agent address: 172.17.0.2
  Agent port:    50000
  Identity:      74:24:d0:2e:7b:b7:9d:13:80:47:e5:fa:45:b3:85:15
Aug 28, 2017 11:35:15 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Handshaking
Aug 28, 2017 11:35:15 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connecting to 172.17.0.2:50000
Aug 28, 2017 11:35:15 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Trying protocol: JNLP4-connect
Aug 28, 2017 11:35:15 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Remote identity confirmed: 74:24:d0:2e:7b:b7:9d:13:80:47:e5:fa:45:b3:85:15
Aug 28, 2017 11:35:16 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connected

更新所以在更多的谷歌搜索后，我发现了这个...“如果您在Pod定义中定义了一个JNLP容器，您需要删除它或将其重命名为JNLP，否则将创建一个名为JNLP的新容器”...也就是说...

podTemplate(name: 'default-java-slave', label: 'maven-builder', cloud: 
'kubernetes', namespace: 'mmidevops', containers: [
  containerTemplate(
    name: 'jnlp', 
    image: 'grantericedwards/jenkinsci-jnlp-slave:3.10-1-alpine', 
    ttyEnabled: true, 
    alwaysPullImage: true,
    workingDir: '/home/jenkins'
  )
  ],
volumes: [
    hostPathVolume(mountPath: '/var/run/docker.sock', hostPath: 
'/var/run/docker.sock')
]
) {

  node('maven-builder') {
    stage('Build a Maven project') {
      container('jnlp') {
          sh 'sleep 10'
      }
    }
  }
}

Answer 1

我使用kubernetes插件0.12，在Jenkins slave上也有一个关闭ChannelException的问题（我认为这不是连接到master的问题，但通信不能接受，然后连接超时并关闭），当我将Jenkins版本从2.60-Alpine升级到2.75-Apline时，我解决了这个问题。詹金斯奴隶现在工作得很完美。希望这能帮助你解决问题。

ps.Kubernetes插件在1.0上发布了新版本，我也升级到了新版本，它工作得很完美！