FlinkSQL | 仿照阿里blink使用sql开发flink的实时程序
kandi X-RAY | FlinkSQL Summary
kandi X-RAY | FlinkSQL Summary
FlinkSQL
Support
Quality
Security
License
Reuse
Top functions reviewed by kandi - BETA
- Verify sql
- Checks whether the SQL contains the given regular expression
- Check if the given character is Chinese
- Split a semicolon
- Get a JobGraph
- Get watermark environment info
- Gets the job name
- Parse insert
- Parse sql
- Get SQL parser from input stream
- Returns stream - local environment info
- Gets the SQLS
- Sets up the yarn client
- Replies the YARN configuration
- Splits a string into two strings ignoring quotes
- Checks if application is running
- Creates a ClusterDescriptor with log configuration
- Parses the given view SqlCreateView
- Get the table source from a SQL CreateTable
- Creates new cluster cluster descriptor
- Parse SQL create functions
- Splits the given string using the given delimiter
- Parse the SQL table
- Shuts down the yarn client
FlinkSQL Key Features
FlinkSQL Examples and Code Snippets
Community Discussions
Trending Discussions on FlinkSQL
QUESTION
I have some batch processed data in a relational database that I want to push to a message bus using Flink. As Flink supports both Batch and Streaming paradigms it appears to be a good fit. Having said that, I cannot tell if this task would belong to the StreamingJob or the BatchJob and how to connect the two. Is this task better suited to a FlinkSQL environment?
Is this possible? what do I need to be aware of?
...ANSWER
Answered 2021-Apr-28 at 12:38It really depends on what You actually want to do, the size of the data and so on. If You just want to read data from a database and write it to say Kafka, You may want to take a look at flink here, since it has the sources for JDBC already implemented.
You could technically use DataStream API, but it doesn't have JDBC source AFAIK, so it would be harder, since You would need to implement that Yourself.
Generally, Flink has two main APIs for handling data, DataSet
and DataStream
and You can't convert between them. DataSet
is more or less what what DataSet
in Spark is, so basically Flink knows that this is bounded, partitioned data, so You can for example sort that or do other transformations that exploit the fact that the data has some defined size.
DataStream
is a Streaming API, which in general allows You to handle technically infinite streams of data, You may use DataStream
to create finite stream, by for example reading from file. But, in general when handling data with DataStream
You will not be able to directly sort it or do some things You might be able to do when dealing with DataSet
, even if the data held in stream is finite.
So, as for the moving data between paradigms, You can write the application that processes DataStream
and You will be able to work both with infinite stream of events from Kafka and with 10 records from CSV file, Flink will be able to do some optimizations based on whether the data is finite or not (introduced in 1.12 IIRC). But You won't be able to read a file into DataSet
sort partitions and then map it to DataStream
, without doing some additional work, like for example storing the data in file and reading it again as DataStream.
Community Discussions, Code Snippets contain sources that include Stack Exchange Network
Vulnerabilities
No vulnerabilities reported
Install FlinkSQL
You can use FlinkSQL like any standard Java library. Please include the the jar files in your classpath. You can also use any IDE and you can run and debug the FlinkSQL component as you would do with any other Java program. Best practice is to use a build tool that supports dependency management such as Maven or Gradle. For Maven installation, please refer maven.apache.org. For Gradle installation, please refer gradle.org .
Support
Reuse Trending Solutions
Find, review, and download reusable Libraries, Code Snippets, Cloud APIs from over 650 million Knowledge Items
Find more librariesStay Updated
Subscribe to our newsletter for trending solutions and developer bootcamps
Share this Page