kubernetes ingress 502坏网关

Question

我在裸机上安装了一个库伯内特斯集群（使用威睿虚拟机），节点如下

master-01   Ready    control-plane,master   5d3h   v1.21.3
master-02   Ready    control-plane,master   5d3h   v1.21.3
master-03   Ready    control-plane,master   5d3h   v1.21.3
worker-01   Ready    <none>                 5d2h   v1.21.3
worker-02   Ready    <none>                 5d2h   v1.21.3
worker-03   Ready    <none>                 5d2h   v1.21.3

Metallb安装为集群的负载平衡器，calico安装为CNI

我还安装了带舵柄的nginx入口控制器

$ helm repo add nginx-stable https://helm.nginx.com/stable
$ helm repo update
$ helm install ingress-controller nginx-stable/nginx-ingress

我部署了一个简单的nginx服务器进行测试

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx-app
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx-app
  template:
    metadata:
      labels:
        app: nginx-app
    spec:
      containers:
        - name: nginx
          image: nginx
          ports:
            - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: nginx-service
spec:
  selector:
    app: nginx-app
  #type: LoadBalancer
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-myapp
  annotations:
    # use the shared ingress-nginx
    kubernetes.io/ingress.class: "nginx"
spec:
  rules:
  - host: myapp.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: nginx-service
            port:
              number: 80

我使用负载均衡器类型的部署从metallb获得IP，工作正常，但当我添加入口时，尽管分配了IP，但我得到错误502坏网关，如下所示：

防火墙已启用，但所需端口已打开

6443/tcp 2379-2380/tcp 10250-10252/tcp 179/tcp 7946/tcp 7946/udp 8443/tcp on master nodes
10250/tcp 30000-32767/tcp 7946/tcp 7946/udp 8443/tcp 179/tcp on worker nodes

我的服务和pods工作正常

 kubectl get svc -o wide
NAME                               TYPE           CLUSTER-IP      EXTERNAL-IP    PORT(S)                      AGE   SELECTOR
ingress-controller-nginx-ingress   LoadBalancer   10.101.17.180   10.1.210.100   80:31509/TCP,443:30004/TCP   33m   app=ingress-controller-nginx-ingress
kubernetes                         ClusterIP      10.96.0.1       <none>         443/TCP                      6d    <none>
nginx-service                      ClusterIP      10.101.48.198   <none>         80/TCP                       31m   app=nginx-app

我的入口日志给我错误，没有到内部IP的路由

2021/07/29 07:46:24 [error] 42#42: *8 connect() failed (113: No route to host) while connecting to upstream, client: 10.1.210.5, server: myapp.com, request: "GET / HTTP/1.1", upstream: "http://192.168.171.17:80/", host: "myapp.com"
10.1.210.5 - - [29/Jul/2021:07:46:24 +0000] "GET / HTTP/1.1" 502 157 "-" "curl/7.68.0" "-"
W0729 07:50:16.416830       1 warnings.go:70] networking.k8s.io/v1beta1 Ingress is deprecated in v1.19+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
192.168.2.131 - - [29/Jul/2021:07:51:03 +0000] "GET / HTTP/1.1" 404 555 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.107 Safari/537.36" "-"
192.168.2.131 - - [29/Jul/2021:07:51:03 +0000] "GET /favicon.ico HTTP/1.1" 404 555 "http://10.1.210.100/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.107 Safari/537.36" "-"
W0729 07:56:43.420282       1 warnings.go:70] networking.k8s.io/v1beta1 Ingress is deprecated in v1.19+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
W0729 08:05:28.422594       1 warnings.go:70] networking.k8s.io/v1beta1 Ingress is deprecated in v1.19+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
W0729 08:10:45.425329       1 warnings.go:70] networking.k8s.io/v1beta1 Ingress is deprecated in v1.19+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
2021/07/29 08:13:59 [error] 42#42: *12 connect() failed (113: No route to host) while connecting to upstream, client: 10.1.210.5, server: myapp.com, request: "GET / HTTP/1.1", upstream: "http://192.168.171.17:80/", host: "myapp.com"
10.1.210.5 - - [29/Jul/2021:08:13:59 +0000] "GET / HTTP/1.1" 502 157 "-" "curl/7.68.0" "-"
2021/07/29 08:14:09 [error] 42#42: *14 connect() failed (113: No route to host) while connecting to upstream, client: 10.1.210.5, server: myapp.com, request: "GET / HTTP/1.1", upstream: "http://192.168.171.17:80/", host: "myapp.com"
10.1.210.5 - - [29/Jul/2021:08:14:09 +0000] "GET / HTTP/1.1" 502 157 "-" "curl/7.68.0" "-"

任何想法请？

编辑:根据此处的要求，描述服务和pod

$ kubectl describe pod nginx-deployment-6f7d8d4d55-sncdr
Name:         nginx-deployment-6f7d8d4d55-sncdr
Namespace:    default
Priority:     0
Node:         worker-01/10.1.210.63
Start Time:   Thu, 29 Jul 2021 08:43:59 +0100
Labels:       app=nginx-app
              pod-template-hash=6f7d8d4d55
Annotations:  cni.projectcalico.org/podIP: 192.168.171.17/32
              cni.projectcalico.org/podIPs: 192.168.171.17/32
Status:       Running
IP:           192.168.171.17
IPs:
  IP:           192.168.171.17
Controlled By:  ReplicaSet/nginx-deployment-6f7d8d4d55
Containers:
  nginx:
    Container ID:   docker://fc61b73f8a833ad13b8956d8ce151b221b75a58a9a2fbae928464f3b0a77cca2
    Image:          nginx
    Image ID:       docker-pullable://nginx@sha256:8f335768880da6baf72b70c701002b45f4932acae8d574dedfddaf967fc3ac90
    Port:           80/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Thu, 29 Jul 2021 08:44:01 +0100
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-wkc48 (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  kube-api-access-wkc48:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  16m   default-scheduler  Successfully assigned default/nginx-deployment-6f7d8d4d55-sncdr to worker-01
  Normal  Pulling    16m   kubelet            Pulling image "nginx"
  Normal  Pulled     16m   kubelet            Successfully pulled image "nginx" in 1.51808376s
  Normal  Created    16m   kubelet            Created container nginx
  Normal  Started    16m   kubelet            Started container nginx

$ kubectl describe svc ingress-controller-nginx-ingress
Name:                     ingress-controller-nginx-ingress
Namespace:                default
Labels:                   app.kubernetes.io/instance=ingress-controller
                          app.kubernetes.io/managed-by=Helm
                          app.kubernetes.io/name=ingress-controller-nginx-ingress
                          helm.sh/chart=nginx-ingress-0.10.0
Annotations:              meta.helm.sh/release-name: ingress-controller
                          meta.helm.sh/release-namespace: default
Selector:                 app=ingress-controller-nginx-ingress
Type:                     LoadBalancer
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       10.101.17.180
IPs:                      10.101.17.180
LoadBalancer Ingress:     10.1.210.100
Port:                     http  80/TCP
TargetPort:               80/TCP
NodePort:                 http  31509/TCP
Endpoints:                192.168.37.202:80
Port:                     https  443/TCP
TargetPort:               443/TCP
NodePort:                 https  30004/TCP
Endpoints:                192.168.37.202:443
Session Affinity:         None
External Traffic Policy:  Local
HealthCheck NodePort:     31108
Events:
  Type    Reason        Age                    From                Message
  ----    ------        ----                   ----                -------
  Normal  IPAllocated   18m                    metallb-controller  Assigned IP "10.1.210.100"
  Normal  nodeAssigned  3m21s (x182 over 18m)  metallb-speaker     announcing from node "worker-02"

$ kubectl describe svc nginx-service
Name:              nginx-service
Namespace:         default
Labels:            <none>
Annotations:       <none>
Selector:          app=nginx-app
Type:              ClusterIP
IP Family Policy:  SingleStack
IP Families:       IPv4
IP:                10.101.48.198
IPs:               10.101.48.198
Port:              <unset>  80/TCP
TargetPort:        80/TCP
Endpoints:         192.168.171.17:80
Session Affinity:  None
Events:            <none>

$ kubectl exec -it ingress-controller-nginx-ingress-dd5db86dc-gqdpm -- /bin/bash
nginx@ingress-controller-nginx-ingress-dd5db86dc-gqdpm:/$ curl 192.168.171.17:80
curl: (7) Failed to connect to 192.168.171.17 port 80: No route to host
nginx@ingress-controller-nginx-ingress-dd5db86dc-gqdpm:/$ curl 192.168.171.17
curl: (7) Failed to connect to 192.168.171.17 port 80: No route to host
nginx@ingress-controller-nginx-ingress-dd5db86dc-gqdpm:/$ curl 10.101.48.198
curl: (7) Failed to connect to 10.101.48.198 port 80: Connection timed out
nginx@ingress-controller-nginx-ingress-dd5db86dc-gqdpm:/$ curl nginx-deployment-6f7d8d4d55-sncdr
curl: (6) Could not resolve host: nginx-deployment-6f7d8d4d55-sncdr
nginx@ingress-controller-nginx-ingress-dd5db86dc-gqdpm:/$

老实说，我不明白为什么curl-svcip不再工作；昨天它起作用了。

Answer 1

k3s中的traefik入口也有类似的问题。我在firewalld 防火墙-cmd -永久-添加-伪装中启用了伪装

这个想法归功于这篇文章：https://github.com/k3s-io/k3s/issues/1646#issuecomment-881191877

Answer 2

问题是防火墙问题。我禁用了防火墙，它现在可以工作了。我以为必须打开端口8443，但如果有人能告诉我是哪个端口，它似乎是另一个端口

谢谢你