pgoutput | Postgres logical replication in Go | SQL Database library

I have a Postgres DB with CDC setup.

I deployed the Kafka Debezium connector 1.8.0.Final for a Postgres DB by

POST http://localhost:8083/connectors

with body:

I succeed generating CDC in a Postgres DB. Today, when I use same step to try to set up Kafka Debezium connector for another Postgres DB.

First I ran

POST http://localhost:8083/connectors

with body:

I am trying to add debezium-connector-postgres to my Kafka Connect.

First I validated my config by

PUT http://localhost:8083/connector-plugins/io.debezium.connector.postgresql.PostgresConnector/config/validate

I'm trying to solve the problem of data denormalization before indexing to the Elasticsearch. Right now, my Postgres 11 database is configured with pgoutput plugin and Debezium with Postgresql Connector is streaming the log changes to RabbitMq which are then aggregated by doing a reverse lookup on the db and feeding to the Elasticsearch.

Although, this works okay, the lookup at the App layer to aggregate the data is expensive and taking a lot of execution time (the query is already refined but it has about 10 joins making it sloppy).

The other alternative I explored was to use KStreams for data aggregation. My knowledge on Apache Kafka is minimal and thus I'm here. My question here is it a requirement to have Apache Kafka as the broker to be able to utilize the Java KStreams API or can it be leveraged with any broker such as RabbitMq? I'm unsure about this because all the articles talk about Kafka Topics and Key Value pairs which are specific to Apache Kafka.

If there is a better way to solve the data denormalization problem, I'm open to it too.

Thanks

I'm implementing an outbox pattern using the debezium postgres connector, building up upon the official documentation: https://debezium.io/documentation/reference/stable/transformations/outbox-event-router.html.

Everything is working quite fine - except that the property "transforms.outbox.table.expand.json.payload: true" is not working.

Using the following database record (SQL instert):

I've set up a CDC pipeline in docker network using following scripts

1. zookeper

   ...

Let's say we have two microservices: service_A and service_B.
Each one has its own database (db_a and db_b respectively) in a single Postgres server instance (This is just a staging environment, so we don't have a cluster).

There is also another service, service_debezium (with an Embedded Debezium v1.6.1Final) that should be listening for changes in db_a and db_b. So basically there are two Debezium engines configured in this service.

But somehow service_debezium cannot listen for db_a and db_b at the same time. It only listens for one of them for some reason and there are no error logs.

Additionally, if I configure service_debezium (i.e. its Debezium engine) to listen for either db_a or db_b, it works just as expected so I'm certain their configuration properties are correct, and (when there is only one engine) everything is working.

1. So why can't we use multiple Debezium engines to listen for multiple databases in a single Postgres server? What am I missing here?
2. Another alternative I thought is to use just one Debezium engine that listens for all databases in that Postgres server instance but apparently it requires database.dbname in its configuration so I guess the preferred way is to define a new Debezium engine for each database. Is that correct?

Here are the Debezium configurations in service_debezium:

• db_a config bean:

When working with Debezium and Postgres, we're seeing an issue where the heartbeat doesn't seem to be working. We have created a dummy table in the target database for performing the heartbeat actions on, but we don't ever see any change to the data in that table.

We've enabled the heartbeat, as we're seeing the same behavior that it was designed to address, namely https://issues.redhat.com/browse/DBZ-1815.

We're using Postgres 12, and Debezium 1.3 (or 1.5, have experimented with both)

The configuration is

I'm trying to run a Pulsar DebeziumPostgresSource connector.

This is the command I'm running: