DButils | DButils for python3 | SMS library

I'm a beginner to Spark and just picked up the highly recommended 'Spark - the Definitive Edition' textbook. Running the code examples and came across the first example that needed me to upload the flight-data csv files provided with the book. I've uploaded the files at the following location as shown in the screenshot:

/FileStore/tables/spark_the_definitive_guide/data/flight-data/csv

I've in the past used Azure Databricks to upload files directly onto DBFS and access them using ls command without any issues. But now in community edition of Databricks (Runtime 9.1) I don't seem to be able to do so.

When I try to access the csv files I just uploaded into dbfs using the below command:

%sh ls /dbfs/FileStore/tables/spark_the_definitive_guide/data/flight-data/csv

I keep getting the below error:

ls: cannot access '/dbfs/FileStore/tables/spark_the_definitive_guide/data/flight-data/csv': No such file or directory

I tried finding out a solution and came across the suggested workaround of using dbutils.fs.cp() as below:

dbutils.fs.cp('C:/Users/myusername/Documents/Spark_the_definitive_guide/Spark-The-Definitive-Guide-master/data/flight-data/csv', 'dbfs:/FileStore/tables/spark_the_definitive_guide/data/flight-data/csv')

dbutils.fs.cp('dbfs:/FileStore/tables/spark_the_definitive_guide/data/flight-data/csv/', 'C:/Users/myusername/Documents/Spark_the_definitive_guide/Spark-The-Definitive-Guide-master/data/flight-data/csv/', recurse=True)

Neither of them worked. Both threw the error: java.io.IOException: No FileSystem for scheme: C

This is really blocking me from proceeding with my learning. It would be supercool if someone can help me solve this soon. Thanks in advance.

I am trying to print the selected value from a query in VB6 this is my code

I recently swicthed on the preview feature "files in repos" on Azure Databricks, so that I could move a lot of my general functions from notebooks to modules and get rid of the overhead from running a lot of notebooks for a single job.

However, several of my functions rely directly on dbutils or spark/pyspark functions (e.g. dbutils.secrets.get() and spark.conf.set()). Since these are imported in the background for the notebooks and are tied directly to the underlying session, I am at complete loss as to how I can reference these modules in my custom modules.

For my small sample module, I fixed it by making dbutils a parameter, like in the following example:

I m trying to get creationfile metadata.

File is in: Azure Storage
Accesing data throw: Databricks

right now I m using:

I am trying to mount my ADLS gen2 storage containers into DBFS, with Azure Active Directory passthrough, using the Databricks Terraform provider. I'm following the instructions here and here, but I'm getting the following error when Terraform attempts to deploy the mount resource:

Error: Could not find ADLS Gen2 Token

My Terraform code looks like the below (it's very similar to the example in the provider documentation) and I am deploying with an Azure Service Principal, which creates the Databricks workspace in the same module:

Off the back this great answer in this post: Create a notebook inside another notebook in Databricks Dynamically using Python where it was shown how to create an entirely new notebook dynamically, my question is on inserting a command into an existing notebook.

Given that

Databricks job used to connect to ADLS G2 storage and process the files successfully.

Recently after renewing the Service Principal secrets, and updating the secret in Key-vault, now the jobs are failing.

using the databricks-cli databricks secrets list-scopes --profile mycluster, i was able to identify which key valut is being used, Also verified the corresponding secrets are updated correctly.

Within the notebook, i followed link and was able to access the ALDS

Below i used to test the key vault values, to access the ADLS.

Using Microsoft's Open Datasets (here), I'd like to create (external) tables in my Databricks env available for consumption in Databricks SQL env and external (BI tools) to this parquet source.

Bit confused on the right approach. Here's what I've tried.

Approach 1: I've tried to create a mount_point (/mnt/taxiData) to the open/public azure store from which I'd use the normal CREATE TABLE dw.table USING PARQUET LOCATION '/mnt/taxi' using the following python code, however, I get an ERROR: Storage Key is not a valid base64 encoded string.

Note: This azure store is open, public. There is no key, no secret.get required.

We have recently made changes to how we connect to ADLS from Databricks which have removed mount points that were previously established within the environment. We are using databricks to find points in polygons, as laid out in the databricks blog here: https://databricks.com/blog/2019/12/05/processing-geospatial-data-at-scale-with-databricks.html

Previously, a chunk of code read in a GeoJSON file from ADLS into the notebook and then projected it to the cluster(s):