NodePort服务不在所有节点上可用

Question

我正在尝试运行一个3节点的Kubernetes集群。我已经启动并充分运行了集群，使得服务可以在不同的节点上运行。不幸的是，我似乎无法让基于NodePort的服务正确工作（就我所理解的正确性而言……)。我的问题是，我定义的任何NodePort服务都只能在其pod运行的节点上外部可用，我的理解是，它们应该在集群中的任何节点上外部可用。

一个例子是本地Jira服务，它应该在端口8082（内部）上运行，在外部32760上运行。以下是服务定义（仅服务部分）：

apiVersion: v1
kind: Service
metadata:
  name: jira
  namespace: wittlesouth
spec:
  ports:
  - port: 8082
  selector:
    app: jira
  type: NodePort

以下是kubectl get服务的输出--命名空间wittle south

NAME       TYPE           CLUSTER-IP      EXTERNAL-IP                       PORT(S)          AGE
jenkins    NodePort       10.100.119.22   <none>                            8081:31377/TCP   3d
jira       NodePort       10.105.148.66   <none>                            8082:32760/TCP   9h
ws-mysql   ExternalName   <none>          mysql.default.svc.cluster.local   3306/TCP         1d

此服务的pod有一个主机端口设置为8082。集群中的三个节点是nuc1、nuc2、NUC3:

Eric:~ eric$ kubectl get nodes
NAME      STATUS    ROLES     AGE       VERSION
nuc1      Ready     master    3d        v1.9.2
nuc2      Ready     <none>    2d        v1.9.2
nuc3      Ready     <none>    2d        v1.9.2

Eric:~ eric$ curl https://nuc1.wittlesouth.com:8082/
curl: (7) Failed to connect to nuc1.wittlesouth.com port 8082: Connection refused
Eric:~ eric$ curl https://nuc2.wittlesouth.com:8082/
curl: (7) Failed to connect to nuc2.wittlesouth.com port 8082: Connection refused
Eric:~ eric$ curl https://nuc3.wittlesouth.com:8082/
curl: (51) SSL: no alternative certificate subject name matches target host name 'nuc3.wittlesouth.com'
Eric:~ eric$ curl https://nuc3.wittlesouth.com:32760/
curl: (51) SSL: no alternative certificate subject name matches target host name 'nuc3.wittlesouth.com'
Eric:~ eric$ curl https://nuc2.wittlesouth.com:32760/
^C
Eric:~ eric$ curl https://nuc1.wittlesouth.com:32760/
curl: (7) Failed to connect to nuc1.wittlesouth.com port 32760: Operation timed out

Eric:~ eric$ kubectl version
Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.1", GitCommit:"3a1c9449a956b6026f075fa3134ff92f7d55f812", GitTreeState:"clean", BuildDate:"2018-01-04T11:52:23Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.2", GitCommit:"5fa2db2bd46ac79e5e00a4e6ed24191080aa463b", GitTreeState:"clean", BuildDate:"2018-01-18T09:42:01Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}

eric@nuc2:~$ ps waux | grep kube-proxy
root      1963  0.5  0.1  54992 37556 ?        Ssl  21:43   0:02 /usr/local/bin/kube-proxy --config=/var/lib/kube-proxy/config.conf
eric      3654  0.0  0.0  14224  1028 pts/0    S+   21:52   0:00 grep --color=auto kube-proxy

Eric:~ eric$ kubectl get pods --namespace=kube-system
NAME                                      READY     STATUS    RESTARTS   AGE
calico-etcd-6vspc                         1/1       Running   3          2d
calico-kube-controllers-d669cc78f-b67rc   1/1       Running   5          3d
calico-node-526md                         2/2       Running   9          3d
calico-node-5trgt                         2/2       Running   3          2d
calico-node-r9ww4                         2/2       Running   3          2d
etcd-nuc1                                 1/1       Running   6          3d
kube-apiserver-nuc1                       1/1       Running   7          3d
kube-controller-manager-nuc1              1/1       Running   6          3d
kube-dns-6f4fd4bdf-dt5fp                  3/3       Running   12         3d
kube-proxy-8xf4r                          1/1       Running   1          2d
kube-proxy-tq4wk                          1/1       Running   4          3d
kube-proxy-wcsxt                          1/1       Running   1          2d
kube-registry-proxy-cv8x9                 1/1       Running   4          3d
kube-registry-proxy-khpdx                 1/1       Running   1          2d
kube-registry-proxy-r5qcv                 1/1       Running   1          2d
kube-registry-v0-wcs5w                    1/1       Running   2          3d
kube-scheduler-nuc1                       1/1       Running   6          3d
kubernetes-dashboard-845747bdd4-dp7gg     1/1       Running   4          3d

cube-proxy似乎正在为我的服务创建iptables条目：

eric@nuc1:/var/lib$ sudo iptables-save | grep hostnames
eric@nuc1:/var/lib$ sudo iptables-save | grep jira
-A KUBE-NODEPORTS -p tcp -m comment --comment "wittlesouth/jira:" -m tcp --dport 32760 -j KUBE-MARK-MASQ
-A KUBE-NODEPORTS -p tcp -m comment --comment "wittlesouth/jira:" -m tcp --dport 32760 -j KUBE-SVC-MO7XZ6ASHGM5BOPI
-A KUBE-SEP-LP4GHTW6PY2HYMO6 -s 192.168.124.202/32 -m comment --comment "wittlesouth/jira:" -j KUBE-MARK-MASQ
-A KUBE-SEP-LP4GHTW6PY2HYMO6 -p tcp -m comment --comment "wittlesouth/jira:" -m tcp -j DNAT --to-destination 192.168.124.202:8082
-A KUBE-SERVICES ! -s 10.5.0.0/16 -d 10.105.148.66/32 -p tcp -m comment --comment "wittlesouth/jira: cluster IP" -m tcp --dport 8082 -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d 10.105.148.66/32 -p tcp -m comment --comment "wittlesouth/jira: cluster IP" -m tcp --dport 8082 -j KUBE-SVC-MO7XZ6ASHGM5BOPI
-A KUBE-SVC-MO7XZ6ASHGM5BOPI -m comment --comment "wittlesouth/jira:" -j KUBE-SEP-LP4GHTW6PY2HYMO6

不幸的是，我现在对iptables一无所知，所以我不知道这些条目看起来是否正确。我怀疑在kubeadm init期间我的非默认网络设置可能与此有关，因为我试图将Kubernetes设置为不使用我的网络的相同IP地址范围（基于192.168）。我使用的kubeadm init语句是：

kubeadm init --pod-network-cidr=10.5.0.0/16 --apiserver-cert-extra-sans ['kubemaster.wittlesouth.com','192.168.5.10'

如果您注意到我使用的calico默认为192.168.0.0的pod network pool,那么在创建calico服务时，我修改了calico的pod network pool设置（不确定是否与此相关）。

Answer 1

当您定义一个NodePort服务时，实际上有三个端口在发挥作用：

• 容器端口：这是您的pod实际上正在侦听的端口，并且只有在从集群中直接访问您的容器时才可用（pod to pod）（JIRA的默认端口是8080）。您将服务中的targetport设置为此端口。
• 服务端口：这是服务本身在集群内部公开的负载平衡端口。使用单个pod不能进行负载平衡，但它仍然是服务的入口点。服务定义中的端口定义了这一点。如果不指定targetport，则假定port和targetport相同。
• 节点端口：路由到服务的每个辅助节点上公开的端口。这是一个通常在30000-33000范围内的端口（取决于集群的配置方式）。这是您能够从群集外部访问的唯一端口。这是用nodeport.
定义的

假设您在标准端口上运行JIRA，您需要一个类似于以下内容的服务定义：

apiVersion: v1
kind: Service
metadata:
  name: jira
  namespace: wittlesouth
spec:
  ports:
  - port: 80          # this is the service port, can be anything
    targetPort: 8080  # this is the container port (must match the port your pod is listening on)
    nodePort: 32000   # if you don't specify this it randomly picks an available port in your NodePort range
  selector:
    app: jira
  type: NodePort

因此，如果使用该配置，则对NodePort服务的传入请求为:NodePort(32000)->service(80)->pod(8080)。（在内部，它实际上可能会绕过服务，我不是100%确定，但您可以从概念上这样考虑它）。

您似乎还试图用HTTPS直接攻击JIRA。您是否在JIRA吊舱中配置了证书？如果是这样，您需要确保它是nuc1.wittlesouth.com的有效证书，或者告诉curl忽略使用curl-k的证书验证错误。