spring-cloud-dataflow | based Streaming and Batch data processing | Stream Processing library

What is the format for passing in additional arguments or environment variables to the Data Flow Server in SCDF running on Kubernetes? When running locally in Docker Compose, I can do something like below, but not sure what the equivalent is when deploying to Kubernetes using the helm chart.

I have about 16 tasks configured in parallel like .

My intention is to only have 3 tasks running at one time. I don't mind which tasks run first as long as the order of the sequential tasks are maintained (BBB is always run after AAA, DDD after CCC etc.)

As per the documentation here - https://docs.spring.io/spring-cloud-dataflow/docs/current/reference/htmlsingle/#_configuration_options, I tried setting --split-thread-core-pool-size=3, but it gave me this error -

Split thread core pool size 3 should be equal or greater than the depth of split flows 17. Try setting the composed task property splitThreadCorePoolSize

What do I do here ?

I am trying to run/configure a Spring Data Cloud Data Flow (SCDF) to schedule a task for a Spring Batch Job.

I am running in a minikube that connects to a local postgresql(localhost:5432). The minikube runs in a virtualbox where I assigned a vnet thru the --cidr, so minikube can connect to the local postgres.

Here is the postgresql service yaml: https://github.com/msuzuki23/SpringCloudDataFlowDemo/blob/main/postgres-service.yaml

Here is the SCDF config yaml: https://github.com/msuzuki23/SpringCloudDataFlowDemo/blob/main/server-config.yaml

Here is the SCDF deployment yaml: https://github.com/msuzuki23/SpringCloudDataFlowDemo/blob/main/server-deployment.yaml

Here is the SCDF server-svc.yaml: https://github.com/msuzuki23/SpringCloudDataFlowDemo/blob/main/server-svc.yaml

To launch the SCDF server in minikube I do the following kubectl commands:

TL;DR

Spring Cloud Data Flow does not allow multiple executions of the same Task even though the documentation says that this is the default behavior. How can we allow SCDF to run multiple instances of the same task at the same time using the Java DSL to launch tasks? To make things more interesting, launching of the same task multiple times works fine when directly hitting the rest enpoints using curl for example.

Background :

I have a Spring Cloud Data Flow Task that I have pre-registered in the Spring Cloud Data Flow UI Dashboard

Background

I have Spring Cloud Data Flow Server running in Kubernetes as a Pod. I am able to launch tasks from the SCDF server UI dashboard. I am looking to develop a more complicated, real-world task- pipeline use-case.

Instead of using the SCDF UI dashboard, I want to launch a sequential list of tasks from a standard Java application. Consider the following task pipeline :

Task 1 : Reads data from the database for the unique id received as task argument input and performs enrichments. The enriched record is written back to the database. Execution of one task instance is responsible for processing one unique id.

Task 2 : Reads the enriched data written by step 1 for the unique id received as task argument input and generates reports. Execution of one task instance is responsible for generating reports for one unique id.

It should be clear from the above explanation that Task 1 and Task 2 are sequential steps. Assume that the input database contains 50k unique ids. I want to develop an orchestrator Java program that would launch task 1 with a limit of 40. (i.e only 40 pods can be running at any given time for task 1. Any requests to launch more pods for task 1 should be put on wait). Once all 50k unique ids have been processed through Task 1 instances, only then can Task 2 pods should be launched.

What I found so far

Going through the documentation, I found something known as the CompositeTaskRunner. However, the examples show commands triggered on a shell/cmd window. I want to do something similar but instead of opening up a data-flow shell program, I want to pass arguments to a Java program that can internally launch tasks. This allows me to easily integrate my application with legacy code that knows how to integrate with Java code (Either by launching a Java program on-demand that should launch a set of tasks and wait for them to complete or by calling a Rest API).

Question

1. How to programmatically launch tasks on-demand with Spring Cloud Data Flow using Java instead of a data-flow shell? (Is there a Rest-API to do this or a simple Java program that will be run on a stand alone server should be fine too)
2. How to programmatically build a sequential pipeline with an upper limit on the number of pods that can be launched per task and with dependencies such that a task can only start once the previous task completed processing all the inputs.

The only way I am running into is using curl command as per the docs: https://docs.spring.io/spring-cloud-dataflow/docs/2.7.1/reference/htmlsingle/#resources-app-registry-post

This uses a curl command to hit the api. Which I can develop a script for, but I would like to set this up within the helm charts so that these tasks and applications are created when the helm chart is deployed. Any ideas?

Recently I updated my JDK from 8 (1.8_275) to 11 (openjdk version "11.0.9.1" 2020-11-04)

While I am trying to launch SCDF local server using

I am currently trying to integrate keycloak with spring cloud dataflow 2.3.0 but the configurations are showing in the documentation is not working for this version. I tried the same with version spring cloud dataflow 2.2.2 and the integrations worked okay. This the config I am added in application.yaml for both the versions,

We are trying to turn on the security for Spring Cloud Data Flow following the documentation (https://docs.spring.io/spring-cloud-dataflow/docs/current-SNAPSHOT/reference/htmlsingle/#configuration-security) but we have some knowledge gaps that we are not capable to fill.

According to the point 9.2, it is possible to configure the authentication with OAuth 2.0 and integrate it with SSO. We use RedHat SSO, so we are trying to integrate both of them, but we are not capable to make it works, is it possible or there is a limitation about the SSO used?

Following the documentation, we have set these properties:

• spring.security.oauth2.client.registration.uaa.client-id=xxxxxxx
• spring.security.oauth2.client.registration.uaa.client-secret=xxxxxx
• spring.security.oauth2.client.registration.uaa.redirect-uri='{baseUrl}/login/oauth2/code/{registrationId}'
• spring.security.oauth2.client.registration.uaa.authorization-grant-type=authorization_code
• spring.security.oauth2.client.registration.uaa.scope[0]=openid
• spring.security.oauth2.client.provider.uaa.jwk-set-uri=../openid-connect/certs
• spring.security.oauth2.client.provider.uaa.token-uri=../openid-connect/token
• spring.security.oauth2.client.provider.uaa.user-info-uri=../openid-connect/userinfo
• spring.security.oauth2.client.provider.uaa.user-name-attribute=user_name
• spring.security.oauth2.client.provider.uaa.authorization-uri=../openid-connect/auth
• spring.security.oauth2.resourceserver.opaquetoken.introspection-uri=../openid-connect/token/introspect
• spring.security.oauth2.resourceserver.opaquetoken.client-id=xxxxxxx
• spring.security.oauth2.resourceserver.opaquetoken.client-secret=xxxxxxx

So we have some considerations:

• The properties resourceserver.opaquetoken are needed for the introspection of the token, so we are pretty sure that they are necessary for when we receive a REST request and it must have the Authorization header
• If we are not using UAA, should the properties be named uaa?
• When we try to access to de UI, it redirects to the authorization-uri because the authorization-grant-type=authorization_code, so it will login in the SSO, is that right?
• If we use the grant-type Password it would request directly a username/password for login, where does it is validated?
• The user-info URI is mandatory but, is it really used?
• What are the other URIs (jwk and token) used for?
• Why the redirect URI has that format? where does that variables point to?

Finally, we have test the configuration in a SCDF running in a Docker container, but it does "nothing":