ncdu | NCurses Disk Usage

Your system has run out of disk space for the boot (root) file system.

The error message is:

Root filesystem has insufficient free space

Shutdown the VM, resize the disk larger in the Google Cloud Web GUI and then restart the VM.

Provided that there are no uncorrectable file system errors, your system will startup, resize the partition and file system, and be fine.

If you have modified the boot disk (partition restructuring, added additional partitions, etc) then you will need to repair and resize manually.

I wrote an article on resizing the Debian root file system. My article goes into more detail than you need, but I do explain the low level details of what happens.

Google Cloud – Debian 9 – Resize Root File System

I suspect it is indeed the logs.

While you are correct that nginx logs are streamed to stdout, that would means docker store the logs directly as container logs. You have several options (pick any, but only one):

1. Just delete/recreate the containers. It will create a fresh one. You'll be safe as long as you already put the persistent data in the correct volume.

2. Go to the /var/lib/docker/containers/. You will find the stdout logs there, usually in the format: -json.log depending on your docker version. Just check where the big file is. Confirm the content with tail . If this is the logs, delete or truncate it: truncate -s 0 or simply : > if you want to use redirection.

Now that we fixed the issue, we should try to prevent it from happening again.

I can see that you already running these containers for 3 years. Congratulations. Container-based approach would mean that it should be trivial if you just delete or create the new container, by the orchestrator.

If you plan on running this for a long time, I suggest you read about docker log rotation. The default one, the json log driver, doesn't have rotation by default so you should specify the limit either in your docker run command or docker compose. This is the relevant documentation about json log driver. Or you can use other log driver like syslog.

If you're looking for any solution, just grepping what you need and using printf will do what you want - the string is "beautified", it's just marking the new lines with \n:

Apparently you're about to run out of the available disk space on your node. However keep in mind that according to the documentation DiskPressure condition denotes:

Available disk space and inodes on either the node’s root filesystem or image filesystem has satisfied an eviction threshold

Try to run df -h but on your worker node, not in a Pod. What is the percentage of disk usage ? Additionally you may check kubelet logs for more details: