repmgr | A lightweight replication manager for PostgreSQL | Key Value Database library

I found that repmgr versions are different for PostgreSQL 11 and PostgreSQL 12. Once I made it same, PostgreSQL upgrade 11 to 12 worked smoothly.

For successful switchover paswword less SSH is required. Try this. https://tecadmin.net/password-less-ssh-login-setup/

2nd is, need to set up below commands in /etc/sudoers file.

This is almost certainly a permissions error.

Make sure that whatever file is readable: chmod a+r /opt/gitlab/embedded/service/gitlab-rails/config/database.yml (according to the comments you already did this) AND

Make sure that all the directories are executable, which for directories allows cd'ing into that directory:

Updated answer based on the discussion in the comments:

These are the issues with the current solution:

1. The original command to start repmgrd:

   command = bash -c "sleep 10 && /usr/local/bin/repmgr_helper.sh"

   runs bash, which executes another bash script (that is another instance of bash), which then runs repmgrd, these are too many processes, most of them not needed

2. supervisord wants that the invoked command remains in the foreground, but repmgrd by default daemonizes itself

3. While troubleshooting there were some issues with the pid file generated by repmgrd

These can be fixed by the following changes:

1. The command to be used instead:

   command = /usr/local/bin/repmgr_helper.sh

2. /usr/local/bin/repmgr_helper.sh needs to be updated to run sleep 10 as the first step

3. /usr/local/bin/repmgr_helper.sh as the very last step should invoke repmgrd the following way:

   exec /path/to/repmgrd --daemonize=false --no-pid-file

   so, a. due to the exec it replaces the script it starts it b. it doesn't daemonize itself c. it doesn't generate a pid file.

Original answer (before the updates)

In the start command try passing --daemonize=false to repmgrd.

What do i do when the master is changing to pg-1 or pg-2?

If your python app is connected to master and master stops you should catch relevant exception and reconnect to the new master. I would suggest to add retries to re-connection attempts as new master election/manual promotion can take time. Of course, some logic should be added for detecting the new master in the app if your goal is to be connected to master only.

The way i connect from python app is using the container name - host =pg-0 What do to when db master host name container changed ?

The only case of container master hostname change i can think of is master container crash/stop. Then new master container goes up and app may reconnect to it as suggested above.

I'm still not sure what the issue is/was, but I was able to get the standby caught up with these two changes:

• set use_replication_slots=true in the repmgr config
• set wal_compression=on in the postgres config

Use replication slots didn't seem to change anything other than to cause replication_delay_gbytes to stay roughly flat. Turing on WAL compression did help, somehow, although I'm not entirely sure how. Yes, in theory it made it possible to ship WAL files to the standby faster, but reviewing network logs I see a drop in sent/received bytes that matches the effects of compression, so it seems to be shipping WAL files at the same speed just using less network.

It still seems like there is some underlying issue at play here, though, because for example when I do pg_basebackup to create the standby it generates roughly 500 MB/s of network traffic, but then when it is streaming WALs after the standby finishes recovery it drops to ~250 MB/s without WAL compression and ~100 MB/s with WAL compression, but there is no decrease in network traffic after it caught up with WAL compression, so I'm not sure what's going on there that allowed it to catch up.

The tables you show are all views, and the age of 2147483647 just reflects that relfrozenxid is 0 for them.

Views have no tuples, and they need no VACUUM, so these are false positives.

Can you determine what exactly causes the warning?

There are things that block VACUUM and survive a restart: replication slots and prepared transactions.

Your replication probably just fell behind, I don't think there is a direct connection to the transaction ID issue (but I can't be certain because you showed no log).

If you are using repmgr ver. 4 and above then you need to change in postgresql.conf
From shared_preload_libraries = 'repmgr_funcs'
To shared_preload_libraries = 'repmgr'

Below are their upgrade notes

The repmgr shared library has been renamed from repmgr_funcs to repmgr, meaning shared_preload_libraries in postgresql.conf needs to be updated to the new name: shared_preload_libraries = 'repmgr'

To give an answer to my own question, the issue lied in how the dockerfile entrypoint scripts were called. Rather, they would end up starting the instance as a master or slave according to variables that were not properly set by me.