go-mysql-elasticsearch | Sync MySQL data into elasticsearch | Data Processing library

 by   go-mysql-org Go Version: Current License: MIT

kandi X-RAY | go-mysql-elasticsearch Summary

kandi X-RAY | go-mysql-elasticsearch Summary

go-mysql-elasticsearch is a Go library typically used in Data Processing applications. go-mysql-elasticsearch has no bugs, it has no vulnerabilities, it has a Permissive License and it has medium support. You can download it from GitHub.

go-mysql-elasticsearch is a service syncing your MySQL data into Elasticsearch automatically. It uses mysqldump to fetch the origin data at first, then syncs data incrementally with binlog.

            kandi-support Support

              go-mysql-elasticsearch has a medium active ecosystem.
              It has 3968 star(s) with 788 fork(s). There are 179 watchers for this library.
              It had no major release in the last 6 months.
              There are 200 open issues and 113 have been closed. On average issues are closed in 46 days. There are 9 open pull requests and 0 closed requests.
              It has a neutral sentiment in the developer community.
              The latest version of go-mysql-elasticsearch is current.

            kandi-Quality Quality

              go-mysql-elasticsearch has 0 bugs and 0 code smells.

            kandi-Security Security

              go-mysql-elasticsearch has no vulnerabilities reported, and its dependent libraries have no vulnerabilities reported.
              go-mysql-elasticsearch code analysis shows 0 unresolved vulnerabilities.
              There are 0 security hotspots that need review.

            kandi-License License

              go-mysql-elasticsearch is licensed under the MIT License. This license is Permissive.
              Permissive licenses have the least restrictions, and you can use them in most projects.

            kandi-Reuse Reuse

              go-mysql-elasticsearch releases are not available. You will need to build from source code and install.
              Installation instructions, examples and code snippets are available.
              It has 1869 lines of code, 84 functions and 11 files.
              It has high code complexity. Code complexity directly impacts maintainability of the code.

            Top functions reviewed by kandi - BETA

            kandi's functional review helps you automatically verify the functionalities of the libraries and avoid rework.
            Currently covering the most popular Java, JavaScript and Python libraries. See a Sample of go-mysql-elasticsearch
            Get all kandi verified functions for this library.

            go-mysql-elasticsearch Key Features

            No Key Features are available at this moment for go-mysql-elasticsearch.

            go-mysql-elasticsearch Examples and Code Snippets

            No Code Snippets are available at this moment for go-mysql-elasticsearch.

            Community Discussions


            Using ElasticSeach as primary source for part of my DB
            Asked 2019-Oct-18 at 21:13

            I have seen many similar question to this topic (including this one, which talks about how ElasticSearch version 6 has overcome many of its limitations as the primary data store), but I am still not clear on the following:

            I am creating an online shopping website and I am using MySQL as my DB.

            This is a simplified version of my DB (Users can post Product on the website for sale)

            I am learning about ElasticSearch and I want to use it to search the products on my website. I don't need User and ProductReview to be searched - only Product table.

            I can think of 2 solutions to achieve this:

            1. Periodically copy Product table from MySQL to ES
            2. Keep User and ProductReview in MySQL and Product in ES

            As far as I know, if I use option 1, then I can use go-mysql-elasticsearch to sync ES with MySQL: Is this a good solution?

            I am more tempted to use option 2, as it is easier and I don't need to worry about data synchronization. What concerns me about this option is:

            • Is ES reliable to be the primary source of data?
            • At some point in time, if I have to modify the Product table structure, would I be able to do so without deleting and recreating the Product Index?
            • In case of MySQL, I normally take a backup of Prod DB and Restore it on Test DB... Is it still possible to do a Backup and Restore from Prod to Test using ES?

            I have no experience with ES/NoSQL and would appreciate any advice.



            Answered 2018-Apr-03 at 12:43

            Let me start by stating that Elasticsearch is NOT a database, in the strict sense of the term, and should ideally not be used as such. However, nothing prevents you from doing it (and many people are doing it) and according to the good folks at Elastic, they won't ever strive to try and make ES a real database. The main goal of ES is to be a fast and reliable search and analytics engine, period.

            If you can, you should always keep another primary source of truth from which you can easily (re-)build your ES indices anytime if something goes south.

            In your case, option 1 seems to be the way to go since all you want to do is to allow users to search your products, so there's no point in synching the other tables in ES.

            Option 2 sounds appealing, but only if you decide to go only with ES, which you really shouldn't if you want to rely on transactions (ES doesn't have transactional support). Another thing you need to know is that if you only have your data in ES and your index gets corrupted for some reason (during an upgrade, a bug in ES, a bug in your code, etc), your data is gone and your business will suffer.

            So to answer your questions more precisely:

            1. ES can be reliable as a primary source of truth provided you throw enough efforts and money into the game. However, you probably don't have millions of products and users (yet), so having a HA cluster with minimum three nodes to search a few thousands products with a few fields doesn't seem like a good spend.

            2. When your products table changes, it is easy to reindex the table into ES (or even in real time) and if you have a few thousand products, it can go fast enough that you don't really have to worry about it. If the synch fails for some reason, you can run the process again without wasting too much time. With the zero-downtime alias technique, you can do it without impacting your users.

            3. ES also provides snapshot/restore capabilities so that you can take a snapshot of PROD and install it in your TEST cluster with a single REST call.

            Source https://stackoverflow.com/questions/49629830


            RDS MySQL, Kinesis Firehose, and Elasticsearch, initial data load?
            Asked 2018-May-25 at 07:07

            I have a need to index data from RDS (MySQL) and S3 (documents) into Elasticsearch in order to perform fulltext searches.

            I've noted that AWS Kinesis seems ideal for this, and can listen to both S3 and MySQL, streaming the formatted results into Elasticsearch.

            What I don't understand, however, is how I could bulk-onboard existing data using Kinesis.

            For RDS-to-Elasticsearch I've seen the alternative of go-mysql-elasticsearch that would handle this for me, but this still leaves me stuck with gigabytes of S3 data to ingest.

            Has anybody solved this problem? I'd rather have as simple a setup as possible.




            Answered 2018-May-25 at 07:07
            Data Enrichment / Metadata

            As far as adding metadata to entries in ElasticSearch, you're probably thinking of what's sometimes called data "enrichment." There's a very detailed blog post over here, that talks about how to ingest and enrich data using both static and dynamic reference data. By using AWS Lambda to enrich your data, you can run dynamic queries against data sources and modify your records before they're ingested into ElasticSearch via Kinesis Firehose.

            Bulk Import

            The Kinesis Data Streams API supports a batch ingestion API called PutRecords. You can ingest up to 500 records into your Kinesis Data Stream with a single API call. The announcement about this is over here.

            Once you've set up your ingestion and enrichment pipeline for new records, you could write an application that retrieves records, older than the date that you established the pipeline, and writes them into the Kinesis Data Stream.

            Amazon Kinesis Data Streams | Service API Reference | PutRecords

            Source https://stackoverflow.com/questions/50514004

            Community Discussions, Code Snippets contain sources that include Stack Exchange Network


            No vulnerabilities reported

            Install go-mysql-elasticsearch

            Install Go (1.9+) and set your GOPATH
            go get github.com/siddontang/go-mysql-elasticsearch, it will print some messages in console, skip it. :-)
            cd $GOPATH/src/github.com/siddontang/go-mysql-elasticsearch


            go-mysql-elasticsearch is still in development, and we will try to use it in production later. Any feedback is very welcome.
            Find more information at:

            Find, review, and download reusable Libraries, Code Snippets, Cloud APIs from over 650 million Knowledge Items

            Find more libraries
          • HTTPS


          • CLI

            gh repo clone go-mysql-org/go-mysql-elasticsearch

          • sshUrl


          • Stay Updated

            Subscribe to our newsletter for trending solutions and developer bootcamps

            Agree to Sign up and Terms & Conditions

            Share this Page

            share link

            Explore Related Topics

            Consider Popular Data Processing Libraries

            Try Top Libraries by go-mysql-org


            by go-mysql-orgGo