Google-Drive-Uploader | Telegram bot To upload Files | Bot library

Goal

Download and upload a file to Google Drive purely in-memory using Google Drive APIs Resumable URL.

Challenge / Problem

I want to buffer the file as its being downloaded to memory (not filesystem) and subsequently upload to Google Drive. Google Drive API requires chunks to be a minimum length of 256 * 1024, (262144 bytes).

The process should pass a chunk from the buffer to be uploaded. If the chunk errors, that buffer chunk is retried up to 3 times. If the chunk succeeds, that chunk from the buffer should be cleared, and the process should continue until complete.

Background Efforts / Research (references below)

Most of the articles, examples and packages I've researched and tested have given some insight into streaming, piping and chunking, but use the filesystem as the starting point from a readable stream.

I've tried different approaches with streams like passthrough with highWaterMark and third-party libraries such as request, gaxios, and got which have built in stream/piping support but with no avail on the upload end of the processes.

Meaning, I am not sure how to structure the piping or chunking mechanism, whether with a buffer or pipeline to properly flow to the upload process until completion, and handle the progress and finalizing events in an efficient manner.

Questions

1. With the code below, how do I appropriately buffer the file and PUT to the google provided URL with the correct Content-Length and Content-Range headers, while having enough buffer space to handle 3 retries?

2. In terms of handling back-pressure or buffering, is leveraging .cork() and .uncork() an efficient way to manage the buffer flow?

3. Is there a way to use a Transform stream with highWaterMark and pipeline to manage the buffer efficiently? e.g...