node-roles | NodeJS Role Management | Runtime Evironment library

I have a Flink job that's bulk writing/upserting a few thousands docs per second onto Elasticsearch. Every time I query it takes ~10-20 seconds to get a response.

I have second index that's exactly the same and equally as full on the same cluster but writes are now turned down to 0 on this index. When I query it it takes milliseconds to get a response.

I.e. with writes off queries take milliseconds. With writes on queries take 10-20 seconds.

CPU utilization ~10%, JVM mem pressure ~70%. ES 7.8.

It would appear then that writes to shards are somehow slowing the reads down. This is odd considering with "profile": true it's giving me query timings on the order of milliseconds yet took (total request time) is in seconds like I'm seeing.

My question is why might this be happening, and how can I optimize it?

(I did think of maybe I could have read replica nodes, but ES doesn't support a read replica node type https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-node.html#node-roles )