docker-flink | Apache Flink docker image | Continuous Deployment library

I wanted to use Google Cloud Storage to write (sink) elements of DataStream from my streaming job using StreamingFileSink.

For doing that, I used Google Cloud Storage connector for Hadoop as an implementation of org.apache.hadoop.fs.FileSystem, and used HadoopFileSystem as an implementation of org.apache.flink.core.fs.FileSystem that wraps the hadoop FileSystem class for Flink.

I included the following dependencies in my gradle file:

• compile("com.google.cloud.bigdataoss:gcs-connector:1.9.4-hadoop2")
• compile("org.apache.flink:flink-connector-filesystem_2.11:1.6.0")
• provided("org.apache.flink:flink-shaded-hadoop2:1.6.0")

Now, from what I understand looking at the sources ^{[1] [2] [3]}, Flink dynamically loads the implementations of FileSystemFactory at runtime (via java.util.ServiceLoader) and also loads the HadoopFsFactory at runtime (via reflection, if it finds Hadoop in classpath) which it then uses to create instances of FileSystem.

The issue I faced was that the default RecoverableWriter for Hadoop compatibility package only supports hdfs file scheme (I use gs) and hence throws an error at runtime.

So, I extended the HadoopFileSystem (I called GCSFileSystem) and @overrided the FileSystem#createRecoverableWriter() to return a custom implementation of RecoverableWriter which then handle the details of recovery, etc. and also created a corresponding FileSystemFactory class (the class is decorated with @AutoService and thus should be discoverable by ServiceLoader).

The setup works well locally and on a local docker cluster (actually the GCS connector throws an error due to lack of authorization, but that's fine since it means that the FileSystem is loaded and running) but it fails when I deploy it to a docker cluster running on Google Compute Engine.

On GCE, the default HadoopFileSystem gets loaded and throws the exception as the scheme is gs and not hdfs, but my assumption is that it should have loaded my implementation of the factory and thus this error shouldn't have arised.

I am on Flink v1.6.0 and running as long running session cluster on Docker using docker-flink

I'm trying to run a simple Apache Beam pipeline on a Flink 1.5.2 docker image. When I run the main class to deploy the pipeline, I get a weird 404 error. The pipeline runs fine on Google Cloud Dataflow.

I run the main with parameters --runner=FlinkRunner, --flinkMaster=localhost:8081, and I can see the Flink dashboard on http://localhost:8081. The deploy fails with:

Unrecognised token 'failure': was expecting ('true', 'false' or 'null)

and it appears to have been trying to access localhost:8081/blobserver/port based on the debug output - I can confirm this path returns a 404 when I do a get request to it.

I get the a similar problem when I try to deploy the job as a fat jar from the web UI. RestException in JarPlanHandler.

I've tried versions 1.6.0 and 1.5.x - specifically I'm using https://github.com/apache/flink/blob/master/flink-contrib/docker-flink/docker-compose.yml like this:

FLINK_DOCKER_IMAGE_NAME=flink:1.5.0 docker-compose up

What am I doing wrong?