为什么Databricks Python无法从我的Azure Datalake存储Gen1中读取数据？

Question

我正在尝试读取文件＜code＞mydir/mycsv。来自Azure Data Lake Storage Gen1的csv，来自Databricks笔记本，使用语法（受文档启发）

configs = {"dfs.adls.oauth2.access.token.provider.type": "ClientCredential",
           "dfs.adls.oauth2.client.id": "123abc-1e42-31415-9265-12345678",
           "dfs.adls.oauth2.credential": dbutils.secrets.get(scope = "adla", key = "adlamaywork"),
           "dfs.adls.oauth2.refresh.url": "https://login.microsoftonline.com/123456abc-2718-aaaa-9999-42424242abc/oauth2/token"}

dbutils.fs.mount(
  source = "adl://myadls.azuredatalakestore.net/mydir",
  mount_point = "/mnt/adls",
  extra_configs = configs)

post_processed = spark.read.csv("/mnt/adls/mycsv.csv").collect()

post_processed.head(10).to_csv("/dbfs/processed.csv")

dbutils.fs.unmount("/mnt/adls")

我的客户123abc-1e42-31415-9265-12345678有权访问Data Lake Storagemyadls，我已经创建了一些秘密

databricks secrets put --scope adla --key adlamaywork

当我在数据砖笔记本中执行上面的pyspark代码时，当使用spark.read访问csv文件时.csv，我得到

com.microsoft.azure.datalake.store.ADLException： Error get info for file /mydir/mycsv.csv

当使用dbfs-ls-dbfs:/mnt/adls导航dbfs时，父装入点似乎在那里，但我得到了

错误:b“{ " ERROR _ code ":" IO _ ERROR "，" message ":"获取访问令牌时出错\ n最后一次尝试1次后引发异常[HTTP0(null)]"} "

我做错了什么？

Answer 1

如果不一定需要将目录挂载到dbfs中，可以尝试直接从adls中读取，如下所示:

spark.conf.set("dfs.adls.oauth2.access.token.provider.type", "ClientCredential")
spark.conf.set("dfs.adls.oauth2.access.token.provider", "org.apache.hadoop.fs.adls.oauth2.ConfCredentialBasedAccessTokenProvider")
spark.conf.set("dfs.adls.oauth2.client.id", "123abc-1e42-31415-9265-12345678")
spark.conf.set("dfs.adls.oauth2.credential", dbutils.secrets.get(scope = "adla", key = "adlamaywork"))
spark.conf.set("dfs.adls.oauth2.refresh.url", "https://login.microsoftonline.com/123456abc-2718-aaaa-9999-42424242abc/oauth2/token")

csvFile = "adl://myadls.azuredatalakestore.net/mydir/mycsv.csv"

df = spark.read.format('csv').options(header='true', inferschema='true').load(csvFile)