sshauthenticator | A simple SSH authenticator for JupyterHub | SSH Utils library

OK. Zeeesus!

In JENKINS_HOME (of the MASTER server), I searched which config file was holding the OLD port# info for that/those container node(s) which were now showing as OFFLINE.

Changed directory to: nodes folder inside $JENKINS_HOME and found that there are config.xml files for each nodes.

For ex: $JENKINS_HOME/nodes/-d4745b720a02/config.xml

Resolution Steps:

1. Vim edited the file to change the OLD with NEW port.
2. Manage Jenkins > Reload configuration from Disk.
3. Manage Nodes > Selected the particular node which was OFFLINE.
4. Relaunch slave, and this time Jenkins picked the new PORT and started the container slave as expected (as SSH connection to the new port visible after the configuration change).

I think this page: https://my.company.jenkins.instance.com/projectInstance/docker-plugin/server// web page, where it shows all the containers info (in a tabular form running on a given slave machine), this page has a button (last column) to STOP a given slave's container but not to START or RESTART.

Having a START or RESTART button there should do what I just did above in some fashion.

Better solution:

What was happening is, all 4 long lived container nodes running on slave3 were competing for gaining all the available RAM (11-12GB) and over the time the JVM process (java -jar slave.jar which the Relaunch step starts on the target container's virtual machine (IP) running on the slave3 slave server) for an individual container were trying to take as much memory (RAM) as they could. That was leading to low FREE memory and thus SWAP getting used and also getting used up to a point where a monitoring tool will start screaming at us via sending notifications etc.

To fix this situation, first thing one should do is:

1) Under Jenkins Global configuration (Manage Jenkins > Configure Systems > Docker Plugin section, for that slave server's Image / Docker Template, under the Advanced Settings section, we can put JVM options to tell the container NOT to compete for all RAM. Putting the following JVM options helped. These JVM settings will try and keep the heap space of each container in a smaller box as to not starve out the rest of the system.

You can start with 3-4GB depending upon how much total RAM you have on your slave/machine where the containers based slave nodes will be running.

2) Look for any recent version of slave.jar (that may have some performance / maintenance enhancements in place which will help.

3) Integrating the monitoring solution (Incinga/etc you have) to auto launch a Jenkins job (where Jenkins job will run some piece of action - BASH one liner, Python shit or Groovy goodness, an Ansible playbook etc) to fix the issue related to any such alert.

4) Automatically have a container slave nodes relaunched (i.e. Relaunch Step) - take slave offline, online, Relaunch step as that'll bring the slave back to a rejuvenated state of freshness. All we have to do is, look for an idle slave (if it's not running any job) then, take it offline > then online > then Relaunch the slave using Jenkins REST API via a small Groovy script and put this all in a Jenkins job and let it do the above if those slave nodes were long lived.

5) OR one can spin the container based slaves on the fly - use and throw model each time Jenkins queues a job to run.