续订服务主体后，Azure Databricks作业无法访问ADLS存储

Question

Database ricks作业用于连接到ADLS G2存储并成功处理文件。

最近，在更新了服务主体机密，并更新了Key-vault中的机密之后，现在作业正在失败。

使用databricks cli＜code＞databricks-secrets list scopes--profile mycluster＜/code＞，我能够识别正在使用的键值，还验证了相应的密钥是否正确更新。

在笔记本中，我遵循了链接并能够访问ALDS

下面我用来测试密钥保管库值，以访问ADLS。

scopename="name-of-the-scope-used-in-databricks-workspace"

appId=dbutils.secrets.get(scope=scopename,key="name-of-the-key-from-keyvault-referring-appid")
directoryId=dbutils.secrets.get(scope=scopename,key="name-of-key-from-keyvault-referring-TenantId")
secretValue=dbutils.secrets.get(scope=scopename,key="name-of-key-from-keyvaut-referring-Secretkey")
storageAccount="ADLS-Gen2-StorageAccountName"

spark.conf.set(f"fs.azure.account.auth.type.{storageAccount}.dfs.core.windows.net", "OAuth")
spark.conf.set(f"fs.azure.account.oauth.provider.type.{storageAccount}.dfs.core.windows.net", "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider")
spark.conf.set(f"fs.azure.account.oauth2.client.id.{storageAccount}.dfs.core.windows.net", appid)
spark.conf.set(f"fs.azure.account.oauth2.client.secret.{storageAccount}.dfs.core.windows.net", secretValue)
spark.conf.set(f"fs.azure.account.oauth2.client.endpoint.{storageAccount}.dfs.core.windows.net", f"https://login.microsoftonline.com/{directoryid}/oauth2/token")

dbutils.fs.ls("abfss://<container-name>@<storage-accnt-name>.dfs.core.windows.net/<folder>")

对于附加的群集，上面会成功显示 ADLS G2 存储中的文件夹/文件列表。

用于创建挂载点的代码，它使用了旧的机密信息。

scope_name="name-of-the-scope-from-workspace"
directoryId=dbutils.secrets.get(scope=scope_name, key="name-of-key-from-keyvault-which-stores-tenantid-value")
configs = {"fs.azure.account.auth.type": "OAuth",
          "fs.azure.account.oauth.provider.type": "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider",
          "fs.azure.account.oauth2.client.id": dbutils.secrets.get(scope=scope_name, key="name-of-key-from-key-vault-referring-to-clientid"),
          "fs.azure.account.oauth2.client.secret": dbutils.secrets.get(scope=scope_name, key="name-of-key-from-key-vault-referring-to-secretvalue-generated-in-sp-secrets"),
          "fs.azure.account.oauth2.client.endpoint": f"https://login.microsoftonline.com/{directoryId}/oauth2/token"}

storage_acct_name="storageaccountname"
container_name="name-of-container"

mount_point = "/mnt/appadls/content"
if not any(mount.mountPoint == mount_point for mount in dbutils.fs.mounts()):
  print(f"Mounting {mount_point} to DBFS filesystem")
  dbutils.fs.mount(
    source = f"abfss://{container_name}@{storage_acct_name}.dfs.core.windows.net/",
    mount_point = mount_point,
    extra_configs = configs)
else:
  print("Mount point {mount_point} has already been mounted.")

在我的例子中，密钥库是用clientid、租户/目录id、SP密钥更新的。

更新服务原则后，当访问/mnt/path时，我看到以下异常。

...
response '{"error":"invalid_client","error_description":"AADSTS7000215: Invalid client secret is provided.

我唯一能想到的是挂载点是用上面代码中的旧秘密创建的。续订服务主体后，我需要卸载并重新创建装载点吗？

Answer 1

所以我终于尝试卸载并挂载ADLS G2存储，现在我可以访问它了。

我没想到配置会以某种方式持久化。仅仅更新服务主体秘密就足够了。