uploadstream | high performance file upload streaming for dotnet | Database library

Used Package

I am trying to upload a blob to the azure blob storage using

I have a function call uploadStream(compressedStream) in my code where I am passing compressedStream as a parameter to the function but before that I need to determine the length of the compressedStream. Does anyone know how can I do that in NodeJS?

I am uploading a stream to azure blob container and it uploads to the container just fine, but I don't know how to get the url that the image created so I can return it to my initial call and render it in my UI.

I can get the request ID back, would that allow me to do a request from there and get the image URL, do I need another request and if so how would I compose that?

Thanks ahead of time

I have the following

I have an API method that streams uploaded files directly to disk to be scanned with a virus checker. Some of these files can be quite large, so IFormFile is a no go:

Any single buffered file exceeding 64 KB is moved from memory to a temp file on disk. Source: https://docs.microsoft.com/en-us/aspnet/core/mvc/models/file-uploads?view=aspnetcore-3.1

I have a working example that uses multipart/form-data and a really nice NuGet package that takes out the headache when working with multipart/form-data, and it works well, however I want to add a file header signature check, to make sure that the file type defined by the client is actually what they say it is. I can't rely on the file extension to do this securely, but I can use the file header signature to make it at least a bit more secure. Since I'm am streaming directly to disk, how can I extract the first bytes as it's going through the file stream?

I'm uploading files from vuejs application to firebase storage. I can successfully write to firebase storage, but the file has zero bytes.

The files are sent to the backend via GraphQL mutation:

I have a problem uploading an image file to my server, I watched some tutorials on YouTube about multer and I do exactly the same thing that is done in the tutorial and for whatever reason I get an error: ("Cannot read property 'buffer' of undefined"), and req.file is also undefined. I googled for the error and found some people having the same issue and I tried to solve it like them, but it didn't work for me.

COMPONENT Data App

I am trying to upload a zip file to Azure file shares. The zip file is being generated using the archiver library and I tried uploading it using piping. I am always getting the error StorageError: The range specified is invalid for the current size of the resource.. How do I find out the size of my archive? I tried 'collecting' the size of the zip like this:

What would be the best way to copy a blob from one storage account to another storage account using @azure/storage-blob?

I would imagine using streams would be best instead of downloading and then uploading, but would like to know if the code below is the correct/optimal implementation for using streams.

Goal

Download and upload a file to Google Drive purely in-memory using Google Drive APIs Resumable URL.

Challenge / Problem

I want to buffer the file as its being downloaded to memory (not filesystem) and subsequently upload to Google Drive. Google Drive API requires chunks to be a minimum length of 256 * 1024, (262144 bytes).

The process should pass a chunk from the buffer to be uploaded. If the chunk errors, that buffer chunk is retried up to 3 times. If the chunk succeeds, that chunk from the buffer should be cleared, and the process should continue until complete.

Background Efforts / Research (references below)

Most of the articles, examples and packages I've researched and tested have given some insight into streaming, piping and chunking, but use the filesystem as the starting point from a readable stream.

I've tried different approaches with streams like passthrough with highWaterMark and third-party libraries such as request, gaxios, and got which have built in stream/piping support but with no avail on the upload end of the processes.

Meaning, I am not sure how to structure the piping or chunking mechanism, whether with a buffer or pipeline to properly flow to the upload process until completion, and handle the progress and finalizing events in an efficient manner.

Questions

1. With the code below, how do I appropriately buffer the file and PUT to the google provided URL with the correct Content-Length and Content-Range headers, while having enough buffer space to handle 3 retries?

2. In terms of handling back-pressure or buffering, is leveraging .cork() and .uncork() an efficient way to manage the buffer flow?

3. Is there a way to use a Transform stream with highWaterMark and pipeline to manage the buffer efficiently? e.g...