azure-storage-net | Microsoft Azure Storage Libraries for .NET | Azure library

As it says in the GitHub description, the DataMovement package mostly represents all operations which you can also do with the command line tool AzCopy.

This library is based on the core data movement framework that powers AzCopy.

You can find some examples of common DataMovement use cases here: https://docs.microsoft.com/en-us/azure/storage/common/storage-use-data-movement-library

The most common use cases for me are transferring files between blob storage accounts or uploading files to an account without having to transfer them through your client. The base Azure.Storage.Blobs library does not support any of that. There are more features, but those are my highlights.

So in your case, it can definitely improve the upload speed if your data can be transferred directly to the storage account without going through your web app. For other cases, it might still be slightly faster than whatever you are currently doing, but it is hard to say with the given information.

1. Create a mock for CloudTable
   • Override the ExecuteAsync behavior via Setup
2. Create a mock for CloudTableClient
   • Override the GetTableReference behavior to return the CloudTable mock via Setup

Thanks to Mohit for the suggestion.

• Create a Task to check the stream length in the background
• If the stream hasn't increased in a set period of time, cancel the DownloadToStreamAsync

DISCLAIMER: I haven't written tests around this code or how to make it run in a performant way as you couldn't have a wait like this for every file you process. I might need to cancel the initial task if the download completes, I don't know yet, I just wanted to get it working first. I don't deem it production ready.

WindowsAzure.Storage(latest is v9.3.2) is the legacy Storage SDK we always use and Microsoft.Azure.Storage.*(latest v9.4.0) is its new release, nuget available about one year.

Check the changelog of Storage .NET SDK. List part of those important differences/changes.

1. Microsoft.Azure.Storage splits libraries to three parts, Blob, Queue and File, which means we can install separate package instead of the full edition.

2. Microsoft.Azure.Storage doesn't support Table API, it is transferred to Microsoft.Azure.Cosmos.Table.

3. Microsoft.Azure.Storage added NetStandard2.0 target support since 9.4.0-preview, which supports synchronous methods wrapped over the asynchronous APIs. WindowsAzure.Storage on NetStandard only has asynchronous APIs.

4. Microsoft.Azure.Storage v9.4 package moves back to use Microsoft.WindowsAzure.Storage namespace temporarily to ease the transition for existing libraries.

Just make choices based on our requirement.

Storage SDK for .NET Framework and .NET Core has minor differences, which are not clarified in the docs.

Begin with v9.2.0, AddMessageAsync(CloudQueueMessage message, CancellationToken cancellationToken) method has been removed in .NET Core SDK. So one workaround is to downgrade sdk to v9.1.1.

Good news is that the differences will be eliminated in next version of Azure Storage Package. See related issue comment.

This library is set to target .NET Standard 2.0 and will not contain these minor API differences.

So another workaround is to use preview version before it becomes generally available.

Note that namespace is changed in preview version.

Update 5/9:

this issue is fixed on the latest version 0.11.0

This is due to the changes in the latest version 10.0.0 of Microsoft.Azure.Storage.Blob or Microsoft.Azure.Storage.File.

You can nav to the project site of Microsoft.Azure.Storage.Blob, now the namespace is changing to Microsoft.Azure.Storage.* (for blob, it's Microsoft.Azure.Storage.Blob):

But in Data Movement library, the method TransferManager.UploadAsync(myfile, destBlob) is still refer to the type Microsoft.WindowsAzure.Storage.Blob.CloudBlob , which is now not available in v10.0.0 of Microsoft.Azure.Storage.Blob.

So in Data Movement library -> TransferManager.UploadAsync(), the parameter should be changed to Microsoft.Azure.Storage.Blob.CloudBlob. Then it can work well.

I see you have submitted an issue, and the team will fix the issue in Data Movement library soon.

As of now, you can use the lower version of Blob or File, and wait for the fix release(I also submit a bug for this issue).

Hope it helps.

Yes TableEntity.Flatten in the SDK does not support ICollection/IEnumerable type properties. But v2.0 of ObjectFlattener API here supports IEnumerable/ICollection etc. propoerties. https://www.nuget.org/packages/ObjectFlattenerRecomposer/

I put some usage example in the Nuget package description, it is very similar to using the TableEntity.Flatten/ConvertBack api but with all property types being supported.

Problem locates at this line

Internally, CloudBlockBlob.StartCopyAsync() calls the Copy Blob REST API.

The function will return once the process is started:

In version 2012-02-12 and newer, the Copy Blob operation can complete asynchronously. This operation returns a copy ID you can use to check or abort the copy operation. The Blob service copies blobs on a best-effort basis.

The source blob for a copy operation may be a block blob, an append blob, or a page blob, or a snapshot. If the destination blob already exists, it must be of the same blob type as the source blob. Any existing destination blob will be overwritten. The destination blob cannot be modified while a copy operation is in progress.

In version 2015-02-21 and newer, the source for the copy operation may also be a file in the Azure File service. If the source is a file, the destination must be a block blob.

Below is the source code to proactively check copy status after calling CloudBlockBlob.StartCopyAsync():