Streaming Anomaly Detection

share link

by kandikits dot icon Updated: Aug 21, 2023

technology logo
technology logo

1-Click Kit 1-Click Kit  


Streaming is the continuous transmission of data from a server to a client. These streaming data can be processed, stored, analyzed, and acted upon in real-time using stream processing technology. The common applications of streaming data process include IoT sensors, server logs etc. to find bugs/anomalies in real-time.


Anomaly Detection is the process of identification of suspicious/rare events by monitoring their significant difference from the standard behavioral pattern of data. In this solution, we identify anomaly in real-time streaming data by Machine Learning techniques.


The Streaming architecture and processing is handled by Kafka and Zookeeper.


A representative output of the execution of the Producer and consumer parts of files is provided below. The left pane is a command prompt which runs the producer file (which sends the data stream) and the right pane is a command prompt which runs the consumer file (which receives the data stream).



Deployment Information

The instructions for running a Streaming Anomaly detection application created using this kit are added in this section. The entire solution is available as a package to download from the source code repository.


For Windows OS,

  1. Download, extract and double-click the kit installer file to install the kit. Note: Do ensure to extract the zip file before running it. The installation may take from 10 to 20 minutes based on network bandwidth.
  2. When you're prompted during the installation of the kit, press Y to launch the app automatically.
  3. To run the app manually, press N when you're prompted and locate the folder 'streaming-anomaly-detection' in the "C://kandikits/streaming-anomaly-detection" location
  4. Navigate into the directory 'streaming-anomaly-detection'
  5. Download and setup Kafka by following the steps
  6. Once the kafka is setup and started, run the kafkaproducer.py and kafkaconsumer.py files in separate cmd prompts


For other Operating System,

  1. Download python
  2. Download the repository
  3. Extract the zip file and navigate to the directory 'streaming-anomaly-detection'
  4. Run the following commands to install Python
tar -xf python*.tar.gz
cd python3.*
./configure
sudo make install


  • Open the terminal in the extracted directory 'streaming-anomaly-detection'
  • Create and activate a virtual environment by these commands:
python3.9 -m venv example
source example/bin/activate


  • Install dependencies by executing the command
pip install -r requirements.txt
  • Download and setup Kafka by following the steps.
  • Once the kafka is setup and started, run the kafkaproducer.py and kafkaconsumer.py files in separate cmd prompts


Click on the button below to download the solution and follow the deployment instructions to begin set-up. This 1-click kit has all the required dependencies and resources you may need to build your Streaming Anomaly Detection App.

Libraries used in this solution


Streaming


Streaming libraries are essential for transmission of data in real-time.

Python doticonstar image 5211 doticonVersion:2.0.2doticon
License: Permissive (Apache-2.0)

Python client for Apache Kafka

Support
    Quality
      Security
        License
          Reuse

            kafka-pythonby dpkp

            Python doticon star image 5211 doticonVersion:2.0.2doticon License: Permissive (Apache-2.0)

            Python client for Apache Kafka
            Support
              Quality
                Security
                  License
                    Reuse

                      kafkaby apache

                      Java doticonstar image 25123 doticonVersion:Currentdoticon
                      License: Permissive (Apache-2.0)

                      Mirror of Apache Kafka

                      Support
                        Quality
                          Security
                            License
                              Reuse

                                kafkaby apache

                                Java doticon star image 25123 doticonVersion:Currentdoticon License: Permissive (Apache-2.0)

                                Mirror of Apache Kafka
                                Support
                                  Quality
                                    Security
                                      License
                                        Reuse

                                          zookeeperby apache

                                          Java doticonstar image 11294 doticonVersion:Currentdoticon
                                          License: Permissive (Apache-2.0)

                                          Apache ZooKeeper

                                          Support
                                            Quality
                                              Security
                                                License
                                                  Reuse

                                                    zookeeperby apache

                                                    Java doticon star image 11294 doticonVersion:Currentdoticon License: Permissive (Apache-2.0)

                                                    Apache ZooKeeper
                                                    Support
                                                      Quality
                                                        Security
                                                          License
                                                            Reuse

                                                              Machine Learning


                                                              Machine learning libraries and frameworks here are helpful in providing state-of-the-art solutions using Machine learning.

                                                              scikit-learnby scikit-learn

                                                              Python doticonstar image 54584 doticonVersion:1.2.2doticon
                                                              License: Permissive (BSD-3-Clause)

                                                              scikit-learn: machine learning in Python

                                                              Support
                                                                Quality
                                                                  Security
                                                                    License
                                                                      Reuse

                                                                        scikit-learnby scikit-learn

                                                                        Python doticon star image 54584 doticonVersion:1.2.2doticon License: Permissive (BSD-3-Clause)

                                                                        scikit-learn: machine learning in Python
                                                                        Support
                                                                          Quality
                                                                            Security
                                                                              License
                                                                                Reuse

                                                                                  Support


                                                                                  For any support, you can reach us at OpenWeaver Community Support

                                                                                  See similar Kits and Libraries