0

我想从 Databricks 读取来自 Azure DataLake Gen 2 的数据。我有一个所有者角色和读/写/执行权限。DataLake 允许从添加 Databricks 的 Vnet 进行访问。我尝试了两种方法：Auth2 和 passthrough（首选方法）。

对于直通方法，我在 Databricks 笔记本中尝试了以下代码，并设置了所有先决条件（来自此 git）：

# MAGIC 1. Azure Databricks Premium Plan.
# MAGIC 2. Azure Data Lake Storage Gen2: Databricks Runtime 5.3 or above.
# MAGIC 3. High concurrency clusters, which support only Python and SQL. [Enabled AD Passthrough checkbox under Advanced Options](https://docs.azuredatabricks.net/spark/latest/data-sources/azure/adls-passthrough.html#enable-passthrough-for-a-cluster)
# MAGIC 4. User needs to have [Storage Data Blob Owner/Contributor/Reader role](https://docs.microsoft.com/en-us/azure/storage/common/storage-auth-aad-rbac-portal#rbac-roles-for-blobs-and-queues) OR [appropriate ACL permissions (R/W/E) on ADLA Gen2](https://docs.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-access-control#access-control-lists-on-files-and-directories) is granted

# COMMAND ----------

# Try to access file
# You'll need data.csv at root of container/filesystem
df = spark.read.csv("abfss://<STORAGE_CONTAINER>@<STORAGE_ACCOUNT>.dfs.core.windows.net/data.csv")
display(df)

并出现错误："abfss://<STORAGE_CONTAINER>@<STORAGE_ACCOUNT>.dfs.core.windows.net/data.csv" has invalid authority

对于 Auth 方法，我使用以下代码：

# Databricks notebook source
# MAGIC %md
# MAGIC ## Azure DataLake Gen2
# MAGIC 
# MAGIC Pre-requisites:
# MAGIC 1. [Create Service Principle](https://docs.microsoft.com/en-        us/azure/active-directory/develop/howto-create-service-principal-portal)
# MAGIC 1. Service Principle has [Storage Data Blob     Owner/Contributor/Reader role](https://docs.microsoft.com/en-    us/azure/storage/common/storage-auth-aad-rbac-portal#rbac-roles-for-blobs-and-    queues) OR [appropriate ACL permissions (R/W/E) on ADLA Gen2]    (https://docs.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-    access-control#access-control-lists-on-files-and-directories) is granted
# MAGIC 2. **Databricks Runtime 5.2** or above
# MAGIC 3. ADLS Gen2 storage account in the **same region** as your Azure     Databricks workspace

# COMMAND ----------

# Set spark configuration
spark.conf.set("fs.azure.account.auth.type", "OAuth")
spark.conf.set("fs.azure.account.oauth.provider.type",     "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider")
spark.conf.set("fs.azure.account.oauth2.client.id", "    <SERVICE_PRINCIPLE_CLIENT_ID>")
spark.conf.set("fs.azure.account.oauth2.client.secret", "    <SERVICE_PRINCIPLE_SECRET>")
spark.conf.set("fs.azure.account.oauth2.client.endpoint",     "https://login.microsoftonline.com/<DIRECTORY_TENANT_ID>/oauth2/token")

# COMMAND ----------

# Try to access file
# You'll need data.csv at root of container/filesystem
df =     spark.read.csv("abfss://<STORAGE_CONTAINER>@<STORAGE_ACCOUNT>.dfs.core.windows    .net/data.csv")
display(df)

错误是STORAGE_ACCOUNT.dfs.core.windows.net/STORAGE_CONTAINER//?action=getAccessControl&amp;timeout=90

这里有什么问题？我有读/写/执行权限。

我是否正确识别 STORAGE_ACCOUNT 和 STORAGE_CONTAINER？

我有以下文件系统（示例）：

在此处输入图像描述

例如，我想在 shared-read-and-write 中读取 data.csv。此代码是否对此正确：

 df =     spark.read.csv("abfss://<STORAGE_CONTAINER>@<STORAGE_ACCOUNT>.dfs.core.windows    .net/data.csv")

->

 df =     spark.read.csv("abfss://shared-read-and-write@DATALAKE GEN2_NAME.dfs.core.windows.net/shared-read-and-write/data.csv")
4

1 回答 1

1

首先在 Databricks 中创建一个安装位置，如下面提到的代码：

clientid="<clientId>"
credential="<clientSecret>"
storageAccountKey="<storageAccountKey>"
refreshurl="<url>"

configs = {"dfs.adls.oauth2.access.token.provider.type": "ClientCredential",
           "dfs.adls.oauth2.client.id": clientid,
           "dfs.adls.oauth2.credential": credential,
           "dfs.adls.oauth2.refresh.url": refreshurl}

dbutils.fs.mount(
  source = "adl://<accountname>.azuredatalakestore.net/<path of adls>",
  mount_point = "/mnt/adls",  extra_configs = configs)

默认情况下，它需要 dbfs 位置，即使您在同一个资源组中，adls 也无法工作使用 spark.read.csv("/mnt/adls/") Hpe 这会有所帮助

于 2019-07-16T09:13:39.063 回答