ubuntu-image | Report bugs at Launchpad

The hostfwd option is for forwarding connections from the outside world to a server which is running on the guest. "hostfwd=tcp::HOSTPORT-:GUESTPORT" says "QEMU should listen on the host on port HOSTPORT; whenever a connection arrives there, it should forward it to the guest's port GUESTPORT (which hopefully has a server listening there)".

You seem to be running a server on the host. You can't have more than one thing listening on a particular port on one machine, so either the python3 server program can listen on port 28009 and respond to connections there, or QEMU can listen on port 28009 to respond to connections there (forwarding them to the guest), but not both at once. Whichever is started second will complain that something's already using the port.

If you want to run a server on the host and connect to it from the guest, you don't need any QEMU options at all. QEMU's 'usermode' networking will allow guest programs to make connections outwards to any IP address (including the wider internet but also directly to the host), so if you are trying to run a client on the guest and a server on the host that should just work. You can tell the guest client to connect either to the host's real IP address or you can use the special 'gateway' IP address 10.0.2.2 which is how the host machine appears on the fake network that the guest sees.

Try

You only need to set the GOOGLE_APPLICATION_CREDENTIALS env variable when using the client libraries.

As you are using the gcloud CLI change this line:

RUN export GOOGLE_APPLICATION_CREDENTIALS=/workdir/ml6-sandbox-cdc8cb4bcae2.json

to

GitLab folks have a reference on their docs about using docker-build inside docker-based jobs: https://docs.gitlab.com/ce/ci/docker/using_docker_build.html#use-docker-in-docker-executor. Since you seem to have everything in place (i.e. the right image for the job and the additional docker:dind service), it's most likely a runner-config issue.

If you look at the second step in the docs:

1. Register GitLab Runner from the command line to use docker and privileged mode:

   [...]

   Notice that it's using the privileged mode to start the build and service containers. If you want to use docker-in-docker mode, you always have to use privileged = true in your Docker containers.

Probably you're using a runner that was not configured in privileged mode and hence can't properly run the docker daemon inside. You can directly edit the /etc/gitlab-runner/config.toml on your registered runner to add that option.

(Also, read on the section on the docs for some more info about the performance related to the storage driver you choose/your runner supports when using dind)