登录
当前位置： 首页 > 知识库问答 >
问题：

与AKS的密钥库问题

东郭骁
2023-03-14 
Volumes:
  sonar-data-new:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  sonar-data-new
    ReadOnly:   false
  sonar-extensions-new2:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  sonar-extensions-new2
    ReadOnly:   false
  secrets-store-inline:
    Type:              CSI (a Container Storage Interface (CSI) volume source)
    Driver:            secrets-store.csi.k8s.io
    FSType:
    ReadOnly:          true
    VolumeAttributes:      secretProviderClass=azure-kv-provider
  default-token-zwxzg:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-zwxzg
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason       Age               From               Message
  ----     ------       ----              ----               -------
  Normal   Scheduled    12s               default-scheduler  Successfully assigned default/sonarqube-d44d498f8-46mpz to aks-agentpool-35716862-vmss000000
  Warning  FailedMount  3s (x5 over 11s)  kubelet            MountVolume.SetUp failed for volume "secrets-store-inline" : rpc error: code = Unknown desc = failed to mount secrets store objects for pod default/sonarqube-d44d498f8-46mpz, err: rpc error: code = Unknown desc = failed to mountobjects, error: failed to get objectType:secret, objectName:username, objectVersion:: azure.BearerAuthorizer#WithAuthorization: Failed to refresh the Token for request to https://SonarQubeHelm.vault.azure.net/secrets/username/?api-version=2016-10-01: StatusCode=400 -- Original Error: adal: Refresh request failed. Status Code = '400'. Response body: {"error":"invalid_request","error_description":"Identity not found"}

这是我的secret-class.yml文件（keyvault的名称是正确的）。另外，xxx-xxxx-xxx-xxx-xxx-xxx-xxx-xxx-xxx4b5ec83是AKS托管标识（SonarQubeHelm-agentpool）的对象

apiVersion: secrets-store.csi.x-k8s.io/v1alpha1
kind: SecretProviderClass
metadata:
  name: azure-kv-provider
spec:
  provider: azure
  secretObjects:
   - data:
      - key: username
        objectName: username
      - key: password
        objectName: password
     secretName: test-secret
     type: Opaque
  parameters:
    usePodIdentity: "false"
    useVMManagedIdentity: "true"
    userAssignedIdentityID: "xxx-xxxx-xxx-xxx-xxxxx4b5ec83"
    keyvaultName: "SonarQubeHelm"
    cloudName: ""
    objects:  |
      array:
        - |
          objectName: username
          objectType: secret
          objectAlias: username
          objectVersion: ""
        - |
          objectName: password
          objectType: secret
          objectAlias: password
          objectVersion: ""
    resourceGroup: "rg-LD-sandbox"
    subscriptionId: "xxxx"
    tenantId: "yyyy"

这是我的deployment.yml文件

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: sonarqube
  name: sonarqube
spec:
  selector:
    matchLabels:
      app: sonarqube
  replicas: 1
  template:
    metadata:
      labels:
        app: sonarqube
    spec:
      containers:
        - name: sonarqube
          image: sonarqube:8.9-developer
          resources:
            requests:
              cpu: 500m
              memory: 1024Mi
            limits:
              cpu: 2000m
              memory: 4096Mi
          volumeMounts:
          - mountPath: "/mnt/"
            name: secrets-store-inline
          - mountPath: "/opt/sonarqube/data/"
            name: sonar-data-new
          - mountPath: "/opt/sonarqube/extensions/plugins/"
            name: sonar-extensions-new2
          env:
          - name: "SONARQUBE_JDBC_USERNAME"
            valueFrom:
              secretKeyRef:
                name: test-secret
                key: username
          - name: "SONARQUBE_JDBC_PASSWORD"
            valueFrom:
              secretKeyRef:
                name: test-secret
                key: password
          - name: "SONARQUBE_JDBC_URL"
            valueFrom:
              configMapKeyRef:
                name: sonar-config
                key: url
          ports:
          - containerPort: 9000
            protocol: TCP
      volumes:
      - name: sonar-data-new
        persistentVolumeClaim:
          claimName: sonar-data-new
      - name: sonar-extensions-new2
        persistentVolumeClaim:
          claimName: sonar-extensions-new2
      - name: secrets-store-inline
        csi:
          driver: secrets-store.csi.k8s.io
          readOnly: true
          volumeAttributes:
           secretProviderClass: "azure-kv-provider"

我为AKS托管标识分配了正确的权限，以便访问keyvault(XXX-XXXXX-XXX-XXX-XXXX4B5EC83是AKS托管标识的对象-SonarQubeHelm-agentpool）

xxxx@Azure:~/clouddrive/kubernetes/sonarqubekeyvault$ az role assignment list --assignee xxx-xxxx-xxx-xxx-xxxxx4b5ec83 --all
[
  {
    "canDelegate": null,
    "condition": null,
    "conditionVersion": null,
    "description": null,
    "id": "/subscriptions/xxxx-xxx-xxx-xxx-xxxe22e8804e/resourceGroups/rg-LD-sandbox/providers/Microsoft.KeyVault/vaults/SonarQubeHelm/providers/Microsoft.Authorization/roleAssignments/xxxx-x-xx-xx-xx86584218f",
    "name": "xxxx-xx-x-x-xx86584218f",
    "principalId": "xxx-xxx-x-xxx-xx3a4b5ec83",
    "principalName": "xxxx-xxxx-xxx-xxx-xx79a3906b8",
    "principalType": "ServicePrincipal",
    "resourceGroup": "rg-LD-sandbox",
    "roleDefinitionId": "/subscriptions/xxxx-xxxx-xxxx-xxx-0e1e22e8804e/providers/Microsoft.Authorization/roleDefinitions/xxx-xxxx-xxx-xxxx-xxxfe8e74483",
    "roleDefinitionName": "Key Vault Administrator",
    "scope": "/subscriptions/xxxx-xxx-xxx-xxxx-xxx2e8804e/resourceGroups/rg-LD-sandbox/providers/Microsoft.KeyVault/vaults/SonarQubeHelm",
    "type": "Microsoft.Authorization/roleAssignments"
  },
  {
    "canDelegate": null,
    "condition": null,
    "conditionVersion": null,
    "description": null,
    "id": "/subscriptions/xxxx-xxxx-xxxx-xxxx-xxxe22e8804e/resourceGroups/rg-LD-sandbox/providers/Microsoft.KeyVault/vaults/SonarQubeHelm/providers/Microsoft.Authorization/roleAssignments/xxxx-xxxx-xxxx-xxxx-xxx5137f480",
    "name": "xxxx-xxxx-xxxx-xxxx-xx5137f480",
    "principalId": "xxxx-xxxx-xxxx-xxxx-xx3a4b5ec83",
    "principalName": "xxxx-xxxx-xxxx-xxxx-xx79a3906b8",
    "principalType": "ServicePrincipal",
    "resourceGroup": "rg-LD-sandbox",
    "roleDefinitionId": "/subscriptions/xxxx-xxxx-xxxx-xxxx-0e1e22e8804e/providers/Microsoft.Authorization/roleDefinitions/xxxx-xxxx-xxxx-xxxx-xx2c155cd7",
    "roleDefinitionName": "Key Vault Secrets Officer",
    "scope": "/subscriptions/xxxx/resourceGroups/rg-LD-sandbox/providers/Microsoft.KeyVault/vaults/SonarQubeHelm",
    "type": "Microsoft.Authorization/roleAssignments"
  }
]

这是关于我的钥匙库的信息。

az keyvault show--名称SonarQubeHelm

{
  "id": "/subscriptions/xxxx-xxxxx-xxxx-xxxx-xxxxxxxx804e/resourceGroups/rg-LD-sandbox/providers/Microsoft.KeyVault/vaults/SonarQubeHelm",
  "location": "xxxxx",
  "name": "SonarQubeHelm",
  "properties": {
    "accessPolicies": [
      {
        "applicationId": null,
        "objectId": "xxxx-xxx-xxxx-xxxx-xxxxa4b5ec83",
        "permissions": {
          "certificates": [
            "Get",
            "List",
            "Update",
            "Create",
            "Import",
            "Delete",
            "Recover",
            "Backup",
            "Restore",
            "ManageContacts",
            "ManageIssuers",
            "GetIssuers",
            "ListIssuers",
            "SetIssuers",
            "DeleteIssuers",
            "Purge"
          ],
          "keys": [
            "Get",
            "List",
            "Update",
            "Create",
            "Import",
            "Delete",
            "Recover",
            "Backup",
            "Restore",
            "Decrypt",
            "Encrypt",
            "UnwrapKey",
            "WrapKey",
            "Verify",
            "Sign",
            "Purge"
          ],
          "secrets": [
            "Get",
            "List",
            "Set",
            "Delete",
            "Recover",
            "Backup",
            "Restore",
            "Purge"
          ],
          "storage": null
        },
        "tenantId": "xxxx-xxxx-xxxx-xxxx-xxxxxdb8c610"
      },
      {
        "applicationId": null,
        "objectId": "xxxx-xxxx-xxxx-xxxx-xxxx531f67f8",
        "permissions": {
          "certificates": [
            "Get",
            "List",
            "Update",
            "Create",
            "Import",
            "Delete",
            "Recover",
            "Backup",
            "Restore",
            "ManageContacts",
            "ManageIssuers",
            "GetIssuers",
            "ListIssuers",
            "SetIssuers",
            "DeleteIssuers",
            "Purge"
          ],
          "keys": [
            "Get",
            "List",
            "Update",
            "Create",
            "Import",
            "Delete",
            "Recover",
            "Backup",
            "Restore",
            "Decrypt",
            "Encrypt",
            "UnwrapKey",
            "WrapKey",
            "Verify",
            "Sign",
            "Purge"
          ],
          "secrets": [
            "Get",
            "List",
            "Set",
            "Delete",
            "Recover",
            "Backup",
            "Restore",
            "Purge"
          ],
          "storage": null
        },
        "tenantId": "xxxxx-xxxxxx-xxx-xxx8db8c610"
      },
      {
        "applicationId": null,
        "objectId": "xxx-xxxx-xxxx-xxx-xxxx0df6af9",
        "permissions": {
          "certificates": [
            "Get",
            "List",
            "Update",
            "Create",
            "Import",
            "Delete",
            "Recover",
            "Backup",
            "Restore",
            "ManageContacts",
            "ManageIssuers",
            "GetIssuers",
            "ListIssuers",
            "SetIssuers",
            "DeleteIssuers"
          ],
          "keys": [
            "Get",
            "List",
            "Update",
            "Create",
            "Import",
            "Delete",
            "Recover",
            "Backup",
            "Restore"
          ],
          "secrets": [
            "Get",
            "List",
            "Set",
            "Delete",
            "Recover",
            "Backup",
            "Restore"
          ],
          "storage": null
        },
        "tenantId": "xxx-xxx-xxx-xxx-xxx8db8c610"
      }
    ],
    "createMode": null,
    "enablePurgeProtection": null,
    "enableRbacAuthorization": false,
    "enableSoftDelete": true,
    "enabledForDeployment": false,
    "enabledForDiskEncryption": false,
    "enabledForTemplateDeployment": false,
    "hsmPoolResourceId": null,
    "networkAcls": null,
    "privateEndpointConnections": null,
    "provisioningState": "Succeeded",
    "sku": {
      "family": "A",
      "name": "Standard"
    },
    "softDeleteRetentionInDays": 90,
    "tenantId": "xxx-xxx-xxx-xxx-xxx68db8c610",
    "vaultUri": "https://sonarqubehelm.vault.azure.net/"
  },
  "resourceGroup": "rg-LD-sandbox",
  "systemData": null,
  "tags": {},
  "type": "Microsoft.KeyVault/vaults"
}

这是目前正在运行的CSI吊舱：

NAME                                                READY   STATUS              RESTARTS   AGE
csi-secrets-store-provider-azure-1632148185-hggl4   1/1     Running             0          5h4m
ingress-nginx-controller-65c4f84996-99pkh           1/1     Running             0          5h49m
secrets-store-csi-driver-xsx2r                      3/3     Running             0          5h4m
sonarqube-d44d498f8-46mpz                           0/1     ContainerCreating   0          26m

export KUBE_ID=$(az aks show -g <resource group> -n <aks cluster name> --query identityProfile.kubeletidentity.objectId -o tsv)
export AKV_ID=$(az keyvault show -g <resource group> -n <akv name> --query id -o tsv)
az role assignment create --assignee $KUBE_ID --role "Key Vault Secrets Officer" --scope $AKV_ID

提前感谢！

共有1个答案

狄旻
2023-03-14

在做了一些测试后，似乎我所遵循的过程是正确的。很可能，我在为AKS托管标识分配角色时使用的是principalid而不是clientid。

对于其他面临类似问题的人来说，要点是：

>

  • 检查AKS自动创建的托管标识是什么。检查clientid；例如， 

    az vmss identity show -g MC_rg-LX-sandbox_SonarQubeHelm_southcentralus -n 
aks-agentpool-xxxxx62-vmss -o yaml

    az role assignment list --assignee xxxx-xxxx-xxx-xxx-xx79a3906b8 --all

    "roleDefinitionName": "Key Vault Administrator"

    export KUBE_ID=$(az aks show -g <resource group> -n <aks cluster name> --query identityProfile.kubeletidentity.clientId -o tsv)
export AKV_ID=$(az keyvault show -g <resource group> -n <akv name> --query id -o tsv)
az role assignment create --assignee $KUBE_ID --role "Key Vault Secrets Officer" --scope $AKV_ID

    •  类似资料：
    • 访问Azure密钥库

      是否有一种方法可以使用Delphi访问Azure密钥库？我可以使用柏林的data.cloud.azureapi单元访问数据库和blob存储，但想从密钥库中获取数据库连接字符串，但似乎找不到任何用于此操作的代码。

    • 密钥库中的密钥密码

      我用过这个命令 生成密钥库。它工作正常，但从我读到的内容来看，这个命令还应该提示您输入密钥密码（而不是存储密码）？我从来没有收到过这样的提示。我能跑 查看密钥库的内容。钥匙似乎就在那里。。。正确的别名在那里。在哪里获取/设置特定别名的密码？ 我有一个key.properties在Android目录 在build.gradle我有： 当我试图生成一个发布版本时，我得到了 我想它可能与keyPassw

    • JMeter问题加载密钥库：找不到密钥

      杰米特。我在系统中设置了jks的路径和密码。财产。我执行一个HTTP请求，在日志中看到一个错误： 和 日志：

    • Azure密钥库与存储加密

      我有一个premium storage account，在创建这个帐户后我就启用了加密，我找到了这个链接https://docs.microsoft.com/en-us/azure/storage/storage-service-encryption来检查blob是否加密。 现在，如果我使用Azure key vault加密OS盘和数据盘，它也用于保护静止数据，但加密存储帐户也会做同样的事情。有谁

    • 安全地访问密钥库机密

      我在Powershell中编写了一个程序，该程序在Azure Functions应用程序中按计划运行。为了避免硬编码的凭据，我创建了一个Azure密钥库来存储秘密。我在Azure函数中创建了一个托管标识，在Azure Key Vault中创建了秘密，然后在Azure函数中创建了应用程序设置，并使用指向Azure Key Vault中秘密的URL。程序引用应用程序机密(APPSETTING)，并按照

    • Azure密钥库：拒绝访问

      我得到的错误是“Access Denied”，这（我认为）意味着id、secret和Vault的url没有问题。但是，我不知道我可以做什么来修复这个错误，也许在Azure门户中有一个设置阻止我读取一个秘密？

    相关问答

    安全地访问密钥库机密Azure密钥库：拒绝访问Azure密钥库访问权限和密钥安全性检索密钥库机密时出现问题Android密钥库错误“无法在密钥库中生成密钥”

    相关文章

    Android中央密钥库PKCS12密钥库和PKCS11密钥库之间有什么区别？java.lang.Exception：答复和密钥库中的公共密钥不匹配Firestore在集合中生成的密钥与自定义密钥？JKS密钥库格式的规范

    相关阅读

    区块链与数据库的区别VB.Net数据库访问Lua 数据库访问JDBC访问数据库Struts2 MySQL数据库访问

    相关工具

    aks-engineaks-secure-baselineblackbelt-aks-hackfestAndroid数据库加密SDKHDCP加密/解密算法

    相关文档

    JavaScript 秘密花园Linux 内核揭密Java 密码学教程Python 密码学教程与 TensorFlow 的初次相遇
    快捷导航： 新手教程 算法原理 架构设计 Java进阶 数据库进阶 大厂专栏 面试经验 编程笔记 编程问答 所有专题 文档资料 工具软件 电子书籍 小牛导航
    在线工具： 房贷计算器 个税计算器 Linux命令查询 Json格式化 正则表达式 颜色转换 AES加解密 SHA1加密 MD5加密 毒鸡汤 字数统计 随机密码生成 进制转换 Base64编解码 励志句子
    Copyright © 2019-2024 小牛知识库@xnip.cn. All Rights Reserved.