mmap | node mmap implementation | Runtime Evironment library

You didn't describe omitting pg_replslot during your rsync, as the docs recommend. If you didn't omit it, then now your replica has a replication slot which is a clone of the one on the master. But if nothing ever connects to that slot on the replica and advances the cutoff, then the WAL never gets released to recycling. To fix you just need to shutdown the replica, remove that directory, restart it, (and wait for the next restart point to finish).

Do they need to go to wal_archive folder on the disk just like they go to wal_archive folder on the master node?

No, that is optional not necessary. It is set by archive_mode = always if you want it to happen.

While I do not know how to deal with this problem directly, I had a somewhat similar issue(and solved). The difference is:

• I use fairseq
• I can run my code on google colab with 1 GPU
• Got RuntimeError: unable to mmap 280 bytes from file : Cannot allocate memory (12) immediately when I tried to run it on multiple GPUs.

From the other people's code, I found that he uses python -m torch.distributed.launch -- ... to run fairseq-train, and I added it to my bash script and the RuntimeError is gone and training is going.

So I guess if you can run with 21000 samples, you may use torch.distributed to make whole data into small batches and distribute them to several workers.

You've used -Xms to force the JVM to get ~30GB at JVM startup.

It has tried, and failed. It only obtained 8GB. It needs another 22-ish GB but cannot get it. That is what the error message is telling you. This is consistent with a dump that says the heap is only 8GB.

You're asking for more than the OS will provide. You'll need to figure out what's going on in the OS in general.

Your application code is probably not involved. The JVM is still initializing its heap in accordance with your command-line options.

Move this statement:

Stracing your application will reveal the issue:

What does this value represent?

It represents the amount of memory that TCMalloc has told the system that it does not need and that the system may use for other purposes.

Has my memory been freed or not?

No. The OS has decided that making it free just to have to make it used again when the program needs it is a waste of effort and has decided instead to switch it directly from one use to another use in a single step rather than going through twice the effort of making it free just to have to make it un-free to use it.

What can I do to prevent this consumption from growing?

Why would you want to? It just makes it easier if the program needs the memory later, minimizes contention on the system's free list, and has no harmful effects at all. TCMalloc has told the OS (via madvise(DONTNEED)) that the OS may recover the memory and the OS has made the decision that it's not a good idea to make it free just to have to make it used again when it's needed. Do you have some good reason to think the OS is wrong?

It's much easier to just directly transition memory from one us to another in a single step than go through the two steps of making it free just to have to make it used again. The free list can get contended under load and it's much simpler not to use it.

You can force the OS to free it by running some program that consumes a lot of memory and then terminates. That will force the OS to transition the memory to that process and then free it when that process terminates. But this would provide no benefit at all and would be a lot of effort just to eventually increase contention in the memory manager. There is no issue here.

Answering my own question

Ec2snitch uses IMDVs1 to get metadata http://169.254.169.254/latest/meta-data/placement/availability-zone to determine certain properties.

I created Ec2 instances through terraform where my code has

I don't completely understand this question as it feels incomplete. But, if you want to use view binding for MapFragment's ID as mentioned in the quoted bounty message below:

How to use View binding in SupportMapFragment' ID?

What you can do is simply write

The problem is that the referenced repository trained a model on an incredibly old version of GenSim, which makes it incompatible with current versions.

You can potentially check whether the lifecycle meta data gives you any indication on the actual version, and then try to update your model from there. The documentation also gives some tips for upgrading your older trained models, but even those are relatively weak and point mostly to re-training. Similarly, even migrating from GenSim 3.X to 4.X is not referencing direct upgrade methods, but could give you ideas on what parameters to look out for specifically.

My suggestion would be to try loading it with any of the previous 3.X versions, and see if you have more success loading it there.