storageDriver | A generic Database independent storage driver | Database library

Chrome can't show nice progress of downloading because the file is downloading on the background. And after downloading, a link to the file is created and "clicked" to force Chrome to show the dialog for the already downloaded file.

It can be done more easily. You need to create a GET request and let the browser download the file, without ajax.

The container is an instantiated instance of the container image. As long as the previously created container exists and is not deleted, it does not have dependencies on the underlying container image.

For anyone who is running into a similar issue:

The issue was NOT actually network related like these error messages implied. The problem ACTUALLY ended up being that the swarm nodes (but not my host) were all using Linux containers and trying to pull a Windows-only image. They were silently throwing architecture errors -- I only found this out when I tried to pull my image directly from a node. The errors generated while deploying to the whole swarm were highly misleading.

I did not "fix" this per say. We ended up restructuring our entire solution around this fact, and the situation no longer applies.

Restarting the given container frees up the disk space but are there any known side effects?

As you stated in your question, you should not "manipulate any files or directories within /var/lib/docker/", as any side-effect may appear and no documentation trace anything about this: it's internal Docker plumbing which may highly change other Docker versions, ut's not supposed to be exposed to end-users nor be tempered with. You could look at Docker code for your Docker version and all it's dependencies to understand what happened, but it's not really practical :-)

are there any known side effects?

There maybe be side effects - I insist on the may as anything can happen depending on your Docker version and configuration. Even if it may seem to be working, some things may be broken.

Well known side effect is Docker installation corruption, which may have present itself in various fashions: random container crash, data loss, unexplained bug, etc.

• Best case scenario, you just discarded some data in your container and everything will work fine in the future.
• Not-so-good scenario: you actually broke something in your installation and corrupted it, you'll be better of re-installing Docker entirely.

Does removing that kind of files or directories from within the container (via docker exec .. rm ..) result in a proper removal or are they only marked as deleted?

Deleting a file in the container will not always remove it from the system, it depends on the drive your are using. Doc has a section about writing files for all of them:

• AUFS - it seemed implied that file is deleted, AUFS will copy the file from the image layer and work on it, it should then delete the copy

  When a file is deleted within a container, a whiteout file is created in the container layer. The version of the file in the image layer is not deleted [...] Subsequent writes to the same file operate against the copy of the file already copied up to the container.

• BTRFS - deleted and space reclaimed, doc is quite clear:

  If a container creates a file and then deletes it, this operation is performed in the Btrfs filesystem itself and the space is reclaimed.

• devicemapper - may not be deleted depending on config:

  if you are using direct-lvm, the blocks are freed. If you use loop-lvm, the blocks may not be freed

• OverlayFS - seemed implied that file is deleted, but the image file is kept

  When a file is deleted within a container, a whiteout file is created in the container (upperdir). The version of the file in the image layer (lowerdir) is not deleted

• ZFS - deleted:

  If you create and then delete a file or directory within the container’s writable layer, the blocks are reclaimed by the zpool.

• VFS is using a copy of the previous layer and work directly in a directory representing that layer, a deletion in the container should probably delete it from the related directory on host machine

The documentation currently doesn't describe this special case.

Yes, and it probably won't ;)

In the end I solved this by

• downgrading to https://docs.docker.com/docker-for-mac/release-notes/#docker-community-edition-18060-ce-mac70-2018-07-25 (warning: wipes all images)

• in Preferences/Daemon/Advanced, typing

  { "storage-driver" : "aufs" }

Short story:

A thin pool is a storage source that provides on-demand allocation for storage space. It is more or less similar to virtual memory, which provides full address space to every process.

Long story:

The traditional storage allocation method is called "fat" or "thick" provisioning.

For example, a user claims to use 10G storage space. Fat provisioning then reserves 10G physical storage space for this user even though he/she only uses 1% of it. No one else can use this reserved space.

Thin provisioning provides a mechanism of on-demand storage allocation, which allows a user to claim more storage space than has been physically reserved for that user.

In other words, it enables over-allocation for storage space. Think about RAM's over-commit feature.

Thin pool is a conceptional term which stands for the backing storage source used by thin provisioning. Thin provisioning allocates virtual chunks of storage from thin pool, while fat provisioning allocates physical blocks of storage from the traditional storage pool.

The Docker Engine can be configured to use Device Mapper as its storage driver. This is where you deal with thin provisioning. According to Docker's documentation:

Production hosts using the devicemapper storage driver must use direct-lvm mode. This mode uses block devices to create the thin pool.

Two different spaces of thin pool need to be taken care of: the Metadata space (which stores pointers) and the Data space (which stores the real data). At the very beginning, all the pointers in Metadata space point to no real chunks in the pool. No chunk in data space is really allocated until a write request arrives. This is nothing new if you are familiar with the virtual memory mechanism.

Let's take a look at the output of docker info:

Docker images are now multi platform: https://blog.docker.com/2017/09/docker-official-images-now-multi-platform/

There is a Dockerfile for running the registry on Windows. I would recommend avoiding using third party images though. https://hub.docker.com/r/stefanscherer/registry-windows/

This is all thanks to UnionFS and Union mounts!

Straight from wikipedia:

It allows files and directories of separate file systems, known as branches, to be transparently overlaid, forming a single coherent file system.

And from an interesting article:

In the kernel, the filesystems are stacked in order of their mount sequence, the first mounted filesystem is at the bottom of the mount stack, and the latest mount is at the top of the stack. Only the files and directories of the top of the mount stack are visible. With union mounts, directory entries from the lower filesystems are merged with the directory entries of upper filesystem, thus making a logical combination of all mounted filesystems. Files with the same name in a lower filesystem are masked, as the upper one takes precedence.

So it doesn't "go through layers" in the conventional sense (e.g one at a time) but rather it knows (at any given time) which file resides on which disk.

Doing this in the filesystem layer also means none of the software has to worry about where the file resides, it knows to ask for /bin/sh and the filesystem knows where to get it.

More info can be found in this webinar.

Where am I wrong?

You are thinking that it has to look through the layers one at a time while it doesn't have to do that. (UnionFS is awesome!)

To list the files that are changed/stored since the original image use