GKE Nginx入口控制器的入口未绑定到外部IP

Question

我正试图在GKE集群上添加一个NGINX入口控制器，使用现有的HAProxy入口控制器（它在重写规则方面有一些问题）

首先，我尝试将控制器的服务公开给LoadBalancer类型。流量可以到达入口和后端，但它不适用于托管证书。

因此，我尝试使用L7负载平衡器（URL映射）将流量转发到GKE群集IP，并为入口控制器本身创建入口对象。

问题是，这个入口对象似乎不绑定到外部IP。路由到域会产生“默认后端-404”响应。

$ kubectl -n ingress-controller get service
NAME                      TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
haproxy-ingress           NodePort    172.16.xxx.xxx  <none>        80:31579/TCP,443:31769/TCP   595d
ingress-default-backend   ClusterIP   172.16.xxx.xxx  <none>        8080/TCP                     595d
nginx-ingress-svc         NodePort    172.16.xxx.xxx  <none>        80:32416/TCP,443:31299/TCP   2d17h

$ kubectl -n ingress-controller get ing
NAME                CLASS    HOSTS   ADDRESS          PORTS   AGE
haproxy-l7-ing      <none>   *       34.xxx.xxx.aaa   80      594d
ingress-nginx-ing   nginx    *       172.xxx.xxx.xxx  80      2d16h

$ gcloud compute addresses list --global --project my-project
NAME                    ADDRESS/RANGE   TYPE      PURPOSE  NETWORK  REGION  SUBNET  STATUS
my-ext-ip               34.xxx.xxx.aaa  EXTERNAL                                    IN_USE
my-test-ext-ip          34.xxx.xxx.bbb  EXTERNAL                                    IN_USE

在这种情况下，我假设ingress-nginx-ing应该绑定到34.xxx.xxx.bbb（my-test-ext-ip），就像haproxy-l7-ing绑定到34.xxx.xxx.aaa（my-ext-ip）但它没有。

负载平衡器：

$ gcloud compute forwarding-rules list --global --project my-project
NAME                              REGION  IP_ADDRESS      IP_PROTOCOL  TARGET
haproxy-http-fwdrule                      34.xxx.xxx.aaa  TCP          haproxy-http-proxy
haproxy-https-fwdrule                     34.xxx.xxx.aaa  TCP          haproxy-https-proxy
nginx-http-fwdrule                        34.xxx.xxx.bbb  TCP          nginx-http-proxy
nginx-https-fwdrule                       34.xxx.xxx.bbb  TCP          nginx-https-proxy

$ gcloud compute target-http-proxies list --global --project my-project
NAME                URL_MAP
haproxy-http-proxy  haproxy-http-urlmap
nginx-http-proxy    nginx-https-urlmap

$ gcloud compute target-https-proxies list --global --project my-project
NAME                                  SSL_CERTIFICATES                    URL_MAP
haproxy-https-proxy                   default-cert,mcrt-xxxxxx-xxxxxx     haproxy-https-urlmap
nginx-https-proxy                     mcrt-xxxxxx-xxxxxx                  nginx-https-urlmap

$ gcloud compute url-maps list --global --project my-project
NAME                      DEFAULT_SERVICE
haproxy-https-urlmap      backendServices/k8s-be-xxxxxx--xxxxxx
haproxy-http-urlmap
nginx-https-urlmap        backendServices/nginx-lb-backendservice

$ gcloud compute backend-services list --global --project my-project
NAME                            BACKENDS                                         PROTOCOL
k8s-be-xxxxxx--xxxxxx           asia-southeast1-a/instanceGroups/k8s-ig--xxxxxx  HTTP
nginx-lb-backendservice         asia-southeast1-a/instanceGroups/k8s-ig--xxxxxx  HTTP

后端：asia-southerast1-a/instanceGroups/k8s-ig-xxxxxx指向GKE集群。

K8S YAML如下：

---
apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
  name: nginx
  namespace: ingress-controller
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
spec:
  controller: k8s.io/ingress-nginx

---
kind: Service
apiVersion: v1
metadata:
  name: nginx-ingress-svc
  namespace: ingress-controller
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
spec:
  externalTrafficPolicy: Local
  type: NodePort
  selector:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
  ports:
    - name: http
      port: 80
      targetPort: http
      protocol: TCP
      appProtocol: http
    - name: https
      port: 443
      targetPort: https
      protocol: TCP
      appProtocol: https

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-nginx-ing
  namespace: ingress-controller
  labels:
    app: ingress-nginx
    tier: ingress
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
  annotations:
    # kubernetes.io/ingress.allow-http: 'false'
    kubernetes.io/ingress.global-static-ip-name: 'my-test-ext-ip'
    ingress.kubernetes.io/url-map: nginx-https-urlmap
    networking.gke.io/managed-certificates: 'my-managed-cert'
    ingress.gcp.kubernetes.io/pre-shared-cert: 'default-cert'
spec:
  ingressClassName: nginx
  defaultBackend:
    service:
      name: nginx-ingress-svc
      port:
        number: 80

你知道我在这里遗漏了什么吗？谢谢

我为负载均衡器调整了一些配置，创建了我自己的后端和健康检查，如下所示：

$ gcloud compute backend-services describe nginx-lb-backendservice --global
affinityCookieTtlSec: 0
backends:
- balancingMode: RATE
  capacityScaler: 1.0
  group: https://www.googleapis.com/compute/v1/projects/my-project/zones/asia-southeast1-a/instanceGroups/k8s-ig--xxxxxx
  maxRatePerInstance: 1.0
cdnPolicy:
  cacheKeyPolicy:
    includeHost: true
    includeProtocol: true
    includeQueryString: false
  cacheMode: USE_ORIGIN_HEADERS
  negativeCaching: false
  requestCoalescing: true
  serveWhileStale: 0
  signedUrlCacheMaxAgeSec: '0'
connectionDraining:
  drainingTimeoutSec: 0
creationTimestamp: '2022-01-07T00:48:38.900-08:00'
description: '{"kubernetes.io/service-name":"ingress-controller/nginx-ingress-svc","kubernetes.io/service-port":"80"}'
enableCDN: true
fingerprint: ****
healthChecks:
- https://www.googleapis.com/compute/v1/projects/mtb-development-project/global/healthChecks/nginx-lb-backend-healthcheck
id: '7699213954898870409'
kind: compute#backendService
loadBalancingScheme: EXTERNAL
logConfig:
  enable: true
  sampleRate: 1.0
name: nginx-lb-backendservice
port: 31579
portName: port31579
protocol: HTTP
selfLink: https://www.googleapis.com/compute/v1/projects/my-project/global/backendServices/nginx-lb-backendservice
sessionAffinity: NONE
timeoutSec: 30

然后，我将此注释添加到Ingress nginx ing中：

ingress.kubernetes.io/url-map: nginx-https-urlmap

后端状态正常，但不知何故，ingress nginx ing仍无法绑定到保留的外部IP。

也没有这些注释附加到它：ingress.kubernetes.io/backends、ingress.kubernetes.io/https-forwarding-rule、ingress.kubernetes.io/https-target-proxy，不像HAProxy的。

正在将HTTP发送到myhost。mydomain/whatever（解析为IP:34.xxx.xxx.bbb）仍然得到“默认后端404”响应。

我在这里尝试了boredabdel的答案，从ingress nginx ing中删除了ingressClassName:nginx，这似乎奏效了。

根据新警告删除手动创建的LB对象并调整自动生成的健康检查后，流量可以按预期到达API。

（混淆的来源来自于示例中的kubernetes.io/ingress.class注释和ingressClassName。）

Answer 1

托管证书仅适用于L7（HTTP）负载平衡器，而不适用于TCP证书。

我的理解是您想在GKE上使用nginx作为入口控制器，但您想在L7 LoadBalancer后面公开它，以便您可以使用Google托管证书？

Answer 2

是的，所以我在您的YAML文件中看到的问题是，您试图使用nginx IngressClass公开NGINX入口本身，这是行不通的。

您要做的是使用GKE默认的IngressClass（称为gce）公开NGINX。如果您将其省略，它将是默认值。所以您的对象大致如下所示

HTTP LB（通过带有gce IngressClass的Inete）-

我们这里确实有一个例子

然而，你必须记住的事情很少。NGINX入口控制器的功能与GKE默认入口控制器的功能几乎相同。他们都在你的应用程序前面安装了HTTP（s）负载平衡器。在此设置中，您将尝试实现2个负载平衡器，通过Ingres配置的Google HTTP LB和另一个NGINX负载平衡器。这意味着两次tcp终止，并可能导致延迟增加。只是要记住一点