azure-storage | Azure Storage module for Nest framework ☁️ | Azure library

After looking at this issue, I figured out what could be the reason. As you might have already known that a new service principal will be created whenever you create a service connection in the Azure DevOps, I have explained this in detail here. To make the AzureFileCopy@4 task work, we will have to add a role assignment under the Role Assignment in the resource group. You can see this when you click on the Access control (IAM). You can also click on the Manage service connection roles in the service connection you had created for this purpose, which will redirect you to the IAM screen.

1. Click on the +Add and select Add role assignment
2. Select the role as either Storage Blob Data Contributor or Storage Blob Data Owner
3. Click Next; on the next screen add the service principal as a member by searching for the name of the service principal. (You can get the name of the service principal from Azure DevOps, on the page for the Service Connection, by clicking on the Manage Service Principal link. My service principal looked like "AzureDevOps.userna.[guid]".)

1. Click on Review + assign once everything is configured.
2. Wait for a few minutes and run your pipeline again. Your pipeline should run successfully now.

You can follow the same fix when you get the error "Upload to container: '' in storage account: '' with blob prefix: ''"

Firstly, feel free to submit a feedback here with the MS docs team to clarify with an official response on same.

Meanwhile, I see, as they mention "In total, it takes 292 hours to complete the migration" it would include listing from source, reading from source, writing to sink, other activities, other than the data movement itself.

If we consider approximately, for data volume of 2 PB and aggregate throughput of 2 GBps would give

2PB = 2,097,152 GB BINARY and Aggregate throughput = 2BGps --> 2,097,152/2 = 1,048,576 secs --> 1,048,576 secs / 3600 = 291.271 hours

Again, these are hypothetical. Further you can refer Plan to manage costs for Azure Data Factory and Understanding Data Factory pricing through examples.

Sol 1 : In my case the problem was due exclusively to the "Queue Storage" function.
Once deleted from Local Sources, if I had managed to delete it from the APP Service everything would have worked again.
Sol 2: Sometimes issue in VSCode, I was building with with Python 3.7 even though 3.6 was installed. Uninstalling Python 3.7 and forcing 3.6 build resolved my issue.

For further ref1, ref2.

This is possible but takes a little bit of work because in addition to being columnar Parquet also requires a schema.

The rough workflow is:

1. Open a parquet file for reading.

2. Then use iter_batches to read back chunks of rows incrementally (you can also pass specific columns you want to read from the file to save IO/CPU).

3. You can then transform each pa.RecordBatch from iter_batches further. Once you are done transforming the first batch you can get its schema and create a new ParquetWriter.

4. For each transformed batch call write_table. You have to first convert it to a pa.Table.

5. Close the files.

Parquet requires random access, so it can't be streamed easily from a URI (pyarrow should support it if you opened the file via HTTP FSSpec) but I think you might get blocked on writes.

Avoid patching, it's ugly and fragile, for each new release you will need an update due upstream's changes.

The main approach is a CMakeLists.txt as wrapper, as real example: https://github.com/conan-io/conan-center-index/blob/5f77986125ee05c4833b0946589b03b751bf634a/recipes/proposal/all/CMakeLists.txt and there many others.

I was facing the very same problem with azure dependencies last few days. Upgrading spring-boot-starter-parent to version 2.5.5 fixed it for me.

The release of setuptools 60.3.0 caused AttributeError because of a bug and now Setuptools 60.3.1 is available. You can refer to the GitHub Link here.

For more information you can refer to the stackoverflow answer as :

If you run into this pip error in a Cloud Function, you might consider updating pip in the "requirements.txt" but if you are in such an unstable Cloud Function the better workaround seems to be to create a new Cloud Function and copy everything in there.

The pip error probably just shows that the source script, in this case the requirements.txt, cannot be run since the source code is not fully embedded anymore or has lost some embedding in the Google Storage. or you give that Cloud Function a second chance and edit, go to Source tab, click on Dropdown Source code to choose Inline Editor and add main.py and requirements.txt manually (Runtime: Python).”

I had a hard time with this as well. At the end of the day, this is how I got it working:

docker-compose.yaml

Follow the below steps to install the package:

1. Add the required modules in requirements.txt

2. Open cmd and enable virtual environment for python with below commands.