flink-htm | streaming anomaly detection and prediction with HTM
kandi X-RAY | flink-htm Summary
kandi X-RAY | flink-htm Summary
Distributed, streaming anomaly detection and prediction with HTM in Apache Flink
Support
Quality
Security
License
Reuse
Top functions reviewed by kandi - BETA
- Process a single input element
- Process an input record
- Converts an Inference to a NetworkInference object
- Converts a record into an input map
- Initialize the network state
- Generates an input function for the given encoder
- Initialize the input function
- The init method
- Resets the input sequence
- Gets the execution environment
- Invokes the given function on the execution environment
- Converts a DataStream into an HTMStream
- Register the HttpSerializer with the Kryo Serializer
- Returns the latest classification
- Implementation of copy
- Reads an object from the given input stream
- Get input network
- Create an accessor function for the given field
- Writes the given object to the output
- Emit water mark
flink-htm Key Features
flink-htm Examples and Code Snippets
Community Discussions
Trending Discussions on flink-htm
QUESTION
EDIT: I just discovered that there exists a class in Java for this: NullByteKeySelector, allowing to use keyed operators for non-keyed use cases
In one of the examples of the library Flink-HTM I can read:
.keyBy { _ => None } .mapWithState { ... }
I'm trying to do the equivalent in Java. Since the mapWithState is a syntactic sugar just for Scala, I attemped to register the usual state handling a in Flink Java MappingFunction, however, I don't know how to go about the keyBy.
In this case the stream is just time series tuples of a timestamp and a numeric. I tried keying by the timestamp, the scalar, and both of them in order to enforce turning the stream into a keyedStream, so I could use keyed state as in the original example. The result was that in the next step, the registered state would be always NULL, if I understood correctly, that would make sense since the keys are constantly unique, and no partitioning is happening at all (unsure about this point).
Next thing I tried is to preserve state by switching to operator state rather than keyed state, but in this context, the state is preserved via checkpointing, checkpointing happening as defined in the checkpointing policy, which leaves me with undesired side effects at times.
I would really want to pull this together using keyed state, could somebody illustrate me on:
- what keying by "None" in the Scala version is doing, and if it possible to achieve an equivalent in Java?
ANSWER
Answered 2020-Jun-14 at 13:31When you key by something that is constantly unique, then you are, in fact, partitioning. The partitions are just very small -- each has one event. That makes the keyed state not very useful, unless you also use timers.
Keying by None is the same as keying by a constant. There's no partitioning, so you can't have a parallelism greater than one, but you can use keyed state and timers.
Community Discussions, Code Snippets contain sources that include Stack Exchange Network
Vulnerabilities
No vulnerabilities reported
Install flink-htm
You can use flink-htm like any standard Java library. Please include the the jar files in your classpath. You can also use any IDE and you can run and debug the flink-htm component as you would do with any other Java program. Best practice is to use a build tool that supports dependency management such as Maven or Gradle. For Maven installation, please refer maven.apache.org. For Gradle installation, please refer gradle.org .
Support
Reuse Trending Solutions
Find, review, and download reusable Libraries, Code Snippets, Cloud APIs from over 650 million Knowledge Items
Find more librariesStay Updated
Subscribe to our newsletter for trending solutions and developer bootcamps
Share this Page