AzureBlobFileSystem | File system abstraction and two implementations | Storage library

I found the solution. file route must be

If you were to try version 1), you would first see warnings saying that sending large delayed objects is a bad pattern in Dask, and makes for large graphs and high memory use on the scheduler. You can send the data directly to workers using client.scatter, but it would still be essentially a serial process, bottlenecking on receiving and sending all of your data through the client process's network connection.

The best practice and canonical way to load data in Dask is for the workers to do it. All the built in loading functions work this way, and is even true when running locally (because any download or open logic should be easily parallelisable).

This is also true for the outputs of your processing. You haven't said what you plan to do next, but to grab all of those images to the client (e.g., .compute()) would be the other side of exactly the same bottleneck. You want to reduce and/or write your images directly on the workers and only handle small transfers from the client.

Note that there are examples out there of image processing with dask (e.g., https://examples.dask.org/applications/image-processing.html ) and of course a lot about arrays. Passing around whole image arrays might be fine for you, but this should be worth a read.

I summarize the solution as below.

The package com.databricks:spark-xml seems using RDD API to read xml file. When we use using the RDD API to access Azure Data Lake Storage Gen2, wecannot access Hadoop configuration options set using spark.conf.set(...). So we should update the code as spark._jsc.hadoopConfiguration().set("fs.azure.account.key.xxxxx.dfs.core.windows.net", "xxxx=="). For more details, please refer to here.

Besides, you aslo can mount Azure Data Lake Storage Gen2 as file system in Azure databricks.

From the error message, you didn't give the correct role to your service principal in the Data Lake Storage Gen2 scope.

To fix the issue, navigate to the storage account in the portal -> Access control (IAM) -> add your service principal as a role e.g. Storage Blob Data Contributor like below.

For more details, refer to this doc - Create and grant permissions to service principal.

As per my research, you will receive this error message when you have incompatible jar with the hadoop version.

I would request you to kindly go through the below issues:

http://mail-archives.apache.org/mod_mbox/spark-issues/201907.mbox/%3CJIRA.13243325.1562321895000.591499.1562323440292@Atlassian.JIRA%3E

https://issues.apache.org/jira/browse/HADOOP-16410

afraid that HADOOP_OPTIONAL_TOOLS env var isn't enough; you'll need to get hadoop-azure JAR and some others into common/lib

from share/hadoop/tools/lib copy hadoop-azure jar, azure-* and, if it's there, wildfly-openssl.jar into share/hadoop/common/lib

The cloudstore JAR is with diagnostics as it tells you which JAR is missing, e.g.