oplog | A generic oplog/replication system for microservices | Microservice library

I'll get straight to the point :

Is it possible to register the MongoDB Kafka sink and source connectors as applications in Spring Cloud Data Flow ? or other types of Kafka connectors in the matter of fact ?

MongoDB Kafka Source connector requires MongoDB to be configured as a replicaset cluster in order to read the change streams in the opLog (a standalone MongoDB instance cannot produce a change stream). Does the SCDF MongoDB source starter use MongoDb cluster's change streams to detect change events ? or does it read changes directly from the MongoDB database ?

Thanks

Since in my country the time was changeg to one hour "ahead". My replica set is doing something I can't understand. This is a 4.2 version and P-S-S replica set.

The Primary oplog is registering actions with the wrong time, it would register 8:00 for an action done at 9:00.

The OS time is properly set ande when I checked the time in mongo shell i got the following:

I am writing a Splittable DoFn to read a MongoDB change stream. It allows me to observe events describing changes to a collection, and I can start reading at an arbitrary cluster timestamp I want, provided oplog has enough history. Cluster timestamps are seconds since epoch combined with the serial number of operation in a given second.

I have looked at other examples of an SDF but all I have seen so far assume a "seekable" data source (Kafka topic-partition, Parquet/Avro file, etc.)

The interface exposed by MongoDB is a simple Iterable, so I cannot really seek to a precise offset (aside from getting a new Iterable starting after a timestamp), and events produced by it have only cluster timestamps - again, no precise offset associated with an output element.

To configure the SDF I use the following class as my input element type:

How to create collection in mongodb with specific uuid or update the uuid ?

I have spent this week to make Mongo 4.4 PSA replica created by a freelancer to work. I gave up, deleted the complete mongod from all three servers and install from scratch following the Mongo doc. The only change was to create new db and import data before the replica initialization.

It failed for the first time (connection timeout) and I revisited my firewalls rules. Then it connected immediately and mongo shell refreshed on all nodes:

I've noticed a difference in the Oplog between v4.4.6 and v5.0.2 when performing updates.

When updating a field in the document, it seems to add the character 's' as a prefix. For example when myField (which is an array) is updated:

I cannot find the way to configure a higher MongoDB oplog size in the Bitnami Helm chart for MongoDB, here.

My understanding is that the oplog will keep all the recent data up to a certain size, or age, and then discard it. That oplog allows replicas that went offline to catch up with the primary oplog once they come back online.

However I cannot see the way to configure it, nor I can see the default value that it takes. According to MongoDB documentation, for linux will be 5% of free space, minium of 990MB and maximum of 50GB, but for the Bitnami helm chart this might be different.

I will be saving pictures in the database and they can be around 1MB each after compression. That will fill up an oplog faster than databases using text only.

As a bonus question, is it required for a Hidden Mongo Node to have an oplog the same size as other nodes that could become primary? I could not find the answer to that either.

I'm working on backup solutions for mongodb instances using percona backup manager.

When I do pbm list if the PITR option is enabled, I get the output for snapshot and oplog slice ranges.

Is there a way to determine which oplog slice range belongs to which backup from the output programmatically so that I can associate an oplog slice range to a snapshot.

I have a PSA replica cluster, and after doing an insert I can see entries in collections but unable to find entries in oplog with this command when insertions are done in a transaction and transaction.