CloudFiles | Amazon S3 SDK for the needed file operations | REST library

Unfortunately, Azure HDInsight does not support Auto Loader for new file detection.

What is Auto Loader?

Autoloader – new functionality from Databricks allowing to incrementally ingest data into Delta Lake from a variety of data sources. Auto Loader is an optimized cloud file source for Apache Spark that loads data continuously and efficiently from cloud storage as new data arrives. A data ingestion network of partner integrations allow you to ingest data from hundreds of data sources directly into Delta Lake.

Under the hood (in Azure Databricks), running Auto Loader will automatically set up an Azure Event Grid and Queue Storage services. Through these services, auto loader uses the queue from Azure Storage to easily find the new files, pass them to Spark and thus load the data with low latency and at a low cost within your streaming or batch jobs. The Auto Loader logs which files were processed which guarantees an exactly once processing of the incoming data.

Auto Loader incrementally and efficiently processes new data files as they arrive in cloud storage without any additional setup. Auto Loader provides a new Structured Streaming source called cloudFiles. Given an input directory path on the cloud file storage, the cloudFiles source automatically processes new files as they arrive, with the option of also processing existing files in that directory.

For details, see Load files from Azure Blob storage or Azure Data Lake Storage Gen2 using Auto Loader.

UPDATE

In Razor view, replace

For razor pages,the page method name in razor pages is different from action method in mvc.It would be like:OnGetMethodName for get method and OnPostMethodName for post method.

Reference:

https://docs.microsoft.com/en-us/aspnet/core/razor-pages/?view=aspnetcore-3.1&tabs=visual-studio#multiple-handlers-per-page

Change your razor pages like below:

Regarding how to list all files in one Azure file share with method ListFilesAndDirectoriesSegmentedAsync, please refer to the following code

The file structure of the File Share is as such:

Thanks for all the help till now. I have solved my issue by following the below steps not sure if below is the correct one or the only one or there is another better way present but below steps solved the issue for me:

1. Updated my Azure Function template which gets displayed while creating new Azure function project from New->Project in VS
2. Then in the version selected Azure Function V3
3. Added the above code as is after installing the Storage related package

And it worked without any issues.

You can make use of CloudFileDirectory.BeginCreateIfNotExists method

More information

Your error looks similar to this issue. The user closed the issue saying it was fixed by a new release of requests, so I would try upgrading requests in your virtual environment:

What exactly does it mean by "Condition headers are not supported"?

Based on my test, there is no issue in your mentioned code. According the Azure file storage Get File API, there is no specifying conditional headers supported. So if the request with If condition header,it is not accepted by Azure file server. It sometimes happens in the browser side, as browser in the some condition append the if condition header.

If Azure blob is acceptable, please have a try to use the Azure blob. Then it will works as expected. The get blob api that supports condition header.

This operation also supports the use of conditional headers to read the blob only if a specified condition is met. For more information, see Specifying Conditional Headers for Blob Service Operations.

If I understand you correctly, you want to avoid storing the file in memory between the download and upload.

For that see:
Azure function to copy files from FTP to blob storage.