HDFS namenode在Kubernetes上不能正确显示数据阳极列表

Question

但一个奇怪的错误正在发生。当我检查Namenode GUI或dfsadmin client以获取datanodes列表时，它只随机显示一个datanode，即有时是datanode-0，有时是DataNode-1。它从不显示两个/所有数据阳极。

这里会有什么问题？我甚至在用无头服务。请帮忙。

#clusterIP service of namenode
apiVersion: v1
kind: Service
metadata:
  name: hdfs-name
  namespace: pulse
  labels:
    app.kubernetes.io/name: hdfs-name
    app.kubernetes.io/version: "1.0"
spec:
  ports:
    - port: 8020
      protocol: TCP
      name: nn-rpc
    - port: 9870
      protocol: TCP
      name: nn-web
  selector:
    app.kubernetes.io/name: hdfs-name
    app.kubernetes.io/version: "1.0"
  type: ClusterIP
---
#namenode stateful deployment 
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: hdfs-name
  namespace: pulse
  labels:
    app.kubernetes.io/name: hdfs-name
    app.kubernetes.io/version: "1.0"
spec:
  serviceName: hdfs-name
  replicas: 1       #TODO 2 namenodes (1 active, 1 standby)
  selector:
    matchLabels:
      app.kubernetes.io/name: hdfs-name
      app.kubernetes.io/version: "1.0"
  template:
    metadata:
      labels:
        app.kubernetes.io/name: hdfs-name
        app.kubernetes.io/version: "1.0"
    spec:
      initContainers:
      - name: delete-lost-found
        image: busybox
        command: ["sh", "-c", "rm -rf /hadoop/dfs/name/lost+found"]
        volumeMounts:
        - name: hdfs-name-pv-claim
          mountPath: /hadoop/dfs/name
      containers:
      - name: hdfs-name
        image: bde2020/hadoop-namenode
        env:
        - name: CLUSTER_NAME
          value: hdfs-k8s
        - name: HDFS_CONF_dfs_permissions_enabled
          value: "false"
        #- name: HDFS_CONF_dfs_replication              #not needed
        #  value: "2"  
        ports:
        - containerPort: 8020
          name: nn-rpc
        - containerPort: 9870
          name: nn-web
        resources:
          limits:
            cpu: "500m"
            memory: 1Gi
          requests:
            cpu: "500m"
            memory: 1Gi
        volumeMounts:
        - name: hdfs-name-pv-claim
          mountPath: /hadoop/dfs/name
  volumeClaimTemplates:
  - metadata:
      name: hdfs-name-pv-claim
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: ebs
      resources:
        requests:
          storage: 1Gi
---
#headless service of datanode
apiVersion: v1
kind: Service
metadata:
  name: hdfs-data
  namespace: pulse
  labels:
    app.kubernetes.io/name: hdfs-data
    app.kubernetes.io/version: "1.0"
spec:
  ports:
    ports:
    - port: 9866
      protocol: TCP
      name: dn-rpc
    - port: 9864
      protocol: TCP
      name: dn-web
  selector:
    app.kubernetes.io/name: hdfs-data
    app.kubernetes.io/version: "1.0"
  clusterIP: None
  type: ClusterIP
---
#datanode stateful deployment
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: hdfs-data
  namespace: pulse
  labels:
    app.kubernetes.io/name: hdfs-data
    app.kubernetes.io/version: "1.0"
spec:
  serviceName: hdfs-data
  replicas: 2
  selector:
    matchLabels:
      app.kubernetes.io/name: hdfs-data
      app.kubernetes.io/version: "1.0"
  template:
    metadata:
      labels:
        app.kubernetes.io/name: hdfs-data
        app.kubernetes.io/version: "1.0"
    spec:
      containers:
      - name: hdfs-data
        image: bde2020/hadoop-datanode
        env:
        - name: CORE_CONF_fs_defaultFS
          value: hdfs://hdfs-name:8020
        ports:           
        - containerPort: 9866
          name: dn-rpc
        - containerPort: 9864
          name: dn-web
        resources:
          limits:
            cpu: "500m"
            memory: 1Gi
          requests:
            cpu: "500m"
            memory: 1Gi
        volumeMounts:
        - name: hdfs-data-pv-claim
          mountPath: /hadoop/dfs/data 
  volumeClaimTemplates:
  - metadata:
      name: hdfs-data-pv-claim
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: ebs
      resources:
        requests:
          storage: 1Gi

     

运行hdfs dfsadmin-report只随机显示一个datanode，例如有时datanode-0和有时datanode-1.
Datanodes主机名不同datanode-0和datanode-1，但它们的名称相同(127.0.0.1:9866(localhost))。这会是问题所在吗？如果是，如何解决I？

而且，我没有看到任何HDFS块复制发生，即使rep因子是3。

更新
嗨，这是Istio porxy问题。我卸载了Istio，它成功了。Istio代理将名称设置为127.0.0.1，而不是实际IP。

Answer 1

我也遇到了同样的问题，我目前使用的解决方案是，通过向hadoop namenode添加以下注释，禁用特使重定向到端口9000（您的情况是8020）上的namenode的入站通信量：

traffic.sidecar.istio.io/excludeInboundPorts: "9000"

参考:https://istio.io/v1.4/docs/reference/config/annotations/

在阅读了一些Istio问题后，似乎在通过特使重定向时，源IP没有被保留。