rpc-go | RPC server written in Go which communicates with the node

It is an actual bug from Envoy and Google Cloud Run. There is a quick fix if you're using .NET6, otherwise it's a bit more hacky. I will just copy here the answer provided by Amanda Tarafa Mas from Google Cloud Platform on the github issue I opened:

Here are the potential fixes:

• When using .NET 6 you can set KestrelServerOptions.AllowAlternateSchemes to true.
• If on a lower .NET version, consider something like GRPC :scheme pseudo-header passed from proxy/loadbalancer causes ConnectionAbortedException dotnet/aspnetcore#30532 (comment). Or consider upgrading to .NET 6.

What's happening:

• Cloud Run has dependency on Envoy, which has a behavior change since 04/15/2021, see "preserve_downstream_scheme" in release notes: https://www.envoyproxy.io/docs/envoy/latest/version_history/v1.18.0 Envoy recently removed the old behaviour: https://www.envoyproxy.io/docs/envoy/latest/version_history/current#removed-config-or-runtime
• In turn, this exposes this .NET issue: GRPC :scheme pseudo-header passed from proxy/loadbalancer causes ConnectionAbortedException dotnet/aspnetcore#30532, for which the Kestrel configuration flag was added, but only for .NET 6. I'm looking into having this documented somewhere. @meteatamel can you update the tutorial so that it uses the Kestrel option?

For me setting KestrelServerOptions.AllowAlternate was enough to make my GRPC server work again.

As @Craig said, you can track the issue here and see if it gets resolved.

If you changed it over to use streams that should help. It took less than 2 seconds to transfer for me. Note this was without ssl and on localhost. This code I threw together. I did run it and it worked. Not sure what might happen if the file is not a multiple of 4 bytes for example. Also the endian order of bytes read is the default for Java.

I made my 10 meg file like this.

gRPC Reflection requires bidirectional streaming, so make sure to check the Enable HTTP/2 option (--use-http2) while deploying. That will enable bi-di streaming.

Also make sure to use the :443 port and authenticate to the server if needed by adding Authentication metadata (see Service-to-Service authentication documentation).

You are trying to set a shell variable without invoking a shell.

The proper way to do this is to set the variable in your Python script.

Interesting question. I've not tried streaming messages with (the excellent) grpcurl.

The documentation does not explain how to do this but this issue shows how to stream using stdin.

I recommend you try it that way first to ensure that works for you.

If it does, then bundling various messages into a file (out) should also work.

Your follow-on questions suggest you're doing this incorrectly.

• chunk_data is the result of having split the file into chunks; i.e. each of these base64-encoded strings should be a subset of your overall image file (i.e. a chunk).

• your first message should be { "info": "...." }, subsequent messages will be { "chunk_data": "" } until EOF.

Got it to work, thanks to protoreflect

Working sample without error handling

The gist of this error is that the version of binary used to generate the code isn't compatible with the current version of code. A quick and easy solution would be to try updating the protoc-gen-go compiler and the gRPC library to the latest version.

go get -u github.com/golang/protobuf/protoc-gen-go

then regen the proto

heres a link to a reddit thread that discusses the issue

Meaning if my window size is n bytes, the stream won't send any data until it's accumulated at least n bytes?

No. The sender can send any number of bytes less than or equal to n.

More generally, how do I maximize the performance of my stream if I maintain only one stream?

For just one stream, just use the max possible value, 2^31-1.
Furthermore, you want to configure the receiver to send WINDOW_UPDATE frames soon enough, so that the sender always has a large enough flow control window that allows it to never stop sending.

One important thing to note is that the configuration of the max flow control window is related to the memory capacity of the receiver.

Since HTTP/2 is multiplexed, the implementation must continue to read data until the flow control window is exhausted.
Using the max flow control window, 2 GiB, means that the receiver needs to be prepared to buffer at least up to 2 GiB of data, until the application decides to consume that data.

In other words: reading the data from the network by the implementation, and consuming that data by the application may happen at different speeds; if reading is faster than consuming, the implementation must read the data and accumulate it aside until the application can consume it.

When the application consumes the data, it tells the implementation how many bytes were consumed, and the implementation may send a WINDOW_UPDATE frame to the sender, to enlarge the flow control window again, so the sender can continue to send.

Note that implementations really want to apply backpressure, i.e. wait for applications to consume the data before sending WINDOW_UPDATEs back to the sender.
If the implementation (wrongly) acknowledges consumption of data before passing it to the application, then it is open to memory blow-up, as the sender will continue to send, but the receiver is forced to accumulate it aside until the host memory of the receiver is exhausted (assuming the application is slower to consume data than the implementation to read data from the network).

Given the above, a single connection, for the max flow control window, may require up to 2 GiB of memory. Imagine 1024 connections (not that many for a server), and you need 2 TiB of memory.

Also consider that for such large flow control windows, you may hit TCP congestion (head of line blocking) before the flow control window is exhausted.
If this happens, you are basically back to the TCP connection capacity, meaning that HTTP/2 flow control limits never trigger because the TCP limits trigger before (or you are otherwise limited by bandwidth, etc.).

Another consideration to make is that you want to avoid that the sender exhausts the flow control window and therefore is forced to stall and stop sending.

For a flow control window of 1 MiB, you don't want to receive 1 MiB of data, consume it and then send back a WINDOW_UPDATE of 1 MiB, because otherwise the client will send 1 MiB, stall, receive the WINDOW_UPDATE, send another 1 MiB, stall again, etc. (see also how to use Multiplexing http2 feature when uploading).

Historically, small flow control windows (as the one suggested in the specification of 64 KiB) were causing super-slow downloads in browsers, that quickly realized that they needed to tell servers that their flow control window was large enough so that the server would not stall the downloads. Currently, Firefox and Chrome set it at 16 MiB.

You want to feed the sender with WINDOW_UPDATEs so it never stalls.

This is a combination of how fast the application consumes the received data, how much you want to "accumulate" the number of consumed bytes before sending the WINDOW_UPDATE (to avoid sending WINDOW_UPDATE too frequently), and how long it takes for the WINDOW_UPDATE to go from receiver to sender.

I mean the grpcpool.ClientConn struct is just:

https://godoc.org/github.com/processout/grpc-go-pool#ClientConn