gupload | Upload files with gRPC and/or | TLS library

The issue appears to be that you've written the phrase Cache-Control in the Cache-Control configuration field, causing it to appear twice in response header, making it invalid. Presumably setting it to simply public, max-age=31536000 will make the resulting header valid.

Don't worry abut the request Cache-Control headers, you can't control them anyway. They're probably set that way because Dev Tools usually turns caching off.

The Cloud Storage HTTP Status 503 means that Cloud Storage encountered an internal error. This usually means that you can try again later (e.g. wait ten seconds) and the request will succeed provided that the original request is valid.

The recommended solution is to try again using truncated exponential backoff.

Reference: 503 - Service Unavailable

[UPDATE - see comments below]

If one instance works correctly and another identical instance consistently fails, then you have a service problem. You will need to open a support case with Google Cloud. Collect your logs to clearly specify what you commented so that Google knows what to diagnose.

I believe the problem is that both pipelines are performing a Bigquery export into the same temporary directory, and they're steppign on each other's toes. You can provide a different directory for each one, like so:

Can you try providing separate GCS locations to your ReadFromBigQuery transforms? You'd do something like this:

When you get a response from the service, it's always a good idea to first check the response code, before you try use the data you expect to have in that response.

If you have a look at the response objects, you can see that it's when it works, and when it doesn't.

403 means "Forbidden". The server doesn't return to you the data you expect, that's why there is no content-disposition header, and your regex fails.

In a streaming pipeline, Dataflow retries work items running into errors indefinitely.

The code itself does not need to have retry logic.

In your request body, I think that "parentID": "[\'16zJngsKtpLtlFe-WTo8LOCCQ2k-uqkZI\']", is required to be modified. And, there is no property of folder. So, how about the following modification?

After giving up on this I decided to reinstall one last time the whole google-cloud-sdk suite, but this time using the snap version. Installing it via snap solved the issue for me. I think this points to some issue with my environment that was bypassed thanks to the snap containerization.

So no clear answer here, but if anyone is experiencing the same problem giving a chance to snap may solve the issue as it did for me

• When you define const drive = google.drive({version: 'v3', auth:XXX}), you need to assign to auth the response of the function authorize(), as shown in the quickstart for Drive API in node.js
• Please follow the complete quickstart to obtain a valid authenticated client

• If creating authenticating with an oAuth2 client is not what you want, there are also other options to create valid crendetials, see google-api-nodejs-client library

At the moment, you can not answer the above questions just by looking at the headers on the client side.

One indications if the request was served by cache or not is by the header age, which Cloud CDN will append on the responses.

If you have enabled the cache logging on the HTTP Load Balancer level you can get all the above information from the logs.

More specifically from the fileds:

httpRequest.cacheHit which indicate if the request was served from the cache or not. jsonPayload.cacheId which is the location and cache instance that the cache response was served from.

More detailed information on the above can be found here 1.