mailing-list | Mailing List service powered by AWS Lambda | Function As A Service library

You can create a alias for multiple collection.

Below is the way you create an alias for multiple collection.

With Table API you are just constructing an expression that will be executed later. So all expressions just return another expression and thus construct a tree of expressions. Use org.apache.flink.table.api.Expressions#ifThenElse or CASE WHEN END in SQL.

Table.execute().collect() returns the result of the view to your client side for interactive purpose. In your case, you can use the filesystem connector and use INSERT INTO for writing the view to the file. For example:

To summarize the series of updates I did to fix my problem:

1. Change the connection port within createRemoteEnvironment() to 8081
2. Check the Scala version on the Apache Flink server, then downgrade your Scala version in the IDE to match

Copying the answer from the mailing list.

If you reuse the cluster for several jobs, they need to share the JVM_ARGS since it's the same process. [1] On Spark, new processes are spawned for each stage afaik.

However, the current recommendation is to use only one ad-hoc cluster per job/application (which is closer to how Spark works). So if you use YARN, every job/application spawns a new cluster that just has the right size for it. Then you can supply new parameters for new YARN submission with

Out of the box, Flink requires a ZooKeeper cluster to recover from JobManager crashes. However, I think you can have a lightweight implementation of the HighAvailabilityServices, CompletedCheckpointStore, CheckpointIDCounter and SubmittedJobGraphStore which can bring you quite far.

Given that you have only one JobManager running at all times (not entirely sure whether K8s can guarantee this) and that you have a persistent storage location, you could implement a CompletedCheckpointStore which retrieves the completed checkpoints from the persistent storage system (e.g. reading all stored checkpoint files). Additionally, you would have a file which contains the current checkpoint id counter for CheckpointIDCounter and all the submitted job graphs for the SubmittedJobGraphStore. So the basic idea is to store everything on a persistent volume which is accessible by the single JobManager.

As Satish said in email, your machine likely ran out of memory. gfortran can take more the 2GB of RAM when compiling. An alternative, if you do not need Fortran, is to configure using --with-fc=0.

Currently we do think that we will try go with Job Cluster approach although we would like to know what is the community trend with this? We would rather not deploy more than one job per Flink cluster.

This question is probably better suited on the user mailing list.

How can I change the number of task slots per task manager for Job and Session Cluster?

You can control this via the config option taskmanager.numberOfTaskSlots

How I can resubmit job using Session Cluster?

This is described here. The bottom line is that you create a savepoint and resume your job from it. It is also possible to resume a job from retained checkpoints.

How can I resubmit job if I'm using a Job Cluster.

Conceptually, this is no different from resuming a job from a savepoint in a session cluster. You can specify the path to the savepoint as a command line argument to the cluster entrypoint. The details are described here.

How I can provide log config for Job/Session cluster?

If you are using the scripts in the bin/ directory of the Flink binary distribution to start your cluster (e.g., bin/start-cluster.sh, bin/jobmanager.sh, bin/taskmanager.sh, etc.), you can change the log4j configuration by adapting conf/log4j.properties. The logging configuration is passed to the JobManager and TaskManager JVMs as a system variable (see bin/flink-daemon.sh). See also the Chapter "How to use logging" in the Flink documentation.

I was able to get past the above error by using a fresh download of Zeppelin. I did run into another issue however when trying to use the python interpreter: Apache Zeppelin 8.1 Win10 Python Interpreter Error

The stack trace in the cmd window used to launch Zeppelin revealed the follow error: Apache Zeppelin 8.1 Win10 Python Interpreter Error Stack Trace

Basically by referencing the code I was able to see the python interpreter is attempting to create a temporary file and put into a folder named tmp: https://github.com/apache/zeppelin/blob/branch-0.8/python/src/main/java/org/apache/zeppelin/python/PythonInterpreter.java, line 102

From this point I wasn't sure where it was trying use this folder named tmp. On my PC the location ended up being in root of C, so C:\tmp