mongo-oplog | Watch mongodb oplog in a simple way | Runtime Evironment library
kandi X-RAY | mongo-oplog Summary
kandi X-RAY | mongo-oplog Summary
Watch mongodb oplog in a simple way
Support
Quality
Security
License
Reuse
Top functions reviewed by kandi - BETA
Currently covering the most popular Java, JavaScript and Python libraries. See a Sample of mongo-oplog
mongo-oplog Key Features
mongo-oplog Examples and Code Snippets
Community Discussions
Trending Discussions on mongo-oplog
QUESTION
we are planning to continuously sync data a collection from MongoDB to another database (in this case Cassandra).
I'm thinking of listening to the mongo-oplog then push those changes to Cassandra. It's risky since the data from MongoDB might be invalid for Cassandra or the Cassandra cluster my down any moment. In the event of Cassandra failure, we gotta call some sort of alert, route all read request to MongoDB then re-sync data to Cassandra from the point of failure. That's a lot of work and any more work may add another point of failure in there.
So is there any best practice for this case, or any sort of libraries or services out there that done this seamlessly? Thanks.
...ANSWER
Answered 2020-Sep-24 at 12:14If you can publish the MongoDB updates to a Kafka topic, DataStax has an open-source Kafka connector for Cassandra. It would be a more resilient and highly-available solution.
For more info, see the Kafka connector for Cassandra docs and kafka-sink repository on GitHub.
There's also a 15-minute Katakoda tutorial here if you're interested -- https://www.datastax.com/dev/scenario/datastax-kafka-connector. Cheers!
QUESTION
I have 2 approaches:
Approach #1
Kafka --> Spark Stream (processing data) --> Kafka -(Kafka Consumer)-> Nodejs (Socket.io)
Approach #2
Kafka --> Kafka Connect (processing data) --> MongoDB -(mongo-oplog-watch)-> Nodejs (Socket.io)
Note: in Approach #2, I use mongo-oplog-watch
to check when inserting data.
What is the advantage and disadvantage when using Kafka as a storage vs using another storage like MongoDB in real-time application context?
...ANSWER
Answered 2019-Jul-03 at 10:50Kafka topics typically have a retention period (default to 7 days) after which they will be deleted. Though, there is no hard rule that we must not persist in Kafka.
You can set the topic retention period to -1
(reference)
The only problem, I know of persisting data in Kafka, is security. Kafka, out of the box (atleast as of now) doesn't provide Data-at-rest encryption. You need to go with a custom solution (or a home-grown one) to have that.
Protecting data-at-rest in Kafka with Vormetric
A KIP is also there, but it is Under discussion
Add end to end encryption in Kafka (KIP)
MongoDB on the other hand seems to provide Data-at-rest encryption.
Security data at rest in MongoDB
And most importantly, it also depends on the type of the data that you are going to store and what you want to do with it.
If you are dealing with data that is quite complex (not easy as Key-Value i.e., give the key and get the value model), for example, like querying by indexed fields etc (as you do typically with logs), then MongoDB could probably make sense.
In simple words, if you are querying by more than one field (other than the key), then storing it in MongoDB could make sense, if you intend to use Kafka for such a purpose, you would probably end up with creating a topic for every field that should be queried... which is too much.
Community Discussions, Code Snippets contain sources that include Stack Exchange Network
Vulnerabilities
No vulnerabilities reported
Install mongo-oplog
Support
Reuse Trending Solutions
Find, review, and download reusable Libraries, Code Snippets, Cloud APIs from over 650 million Knowledge Items
Find more librariesStay Updated
Subscribe to our newsletter for trending solutions and developer bootcamps
Share this Page