pglogical | Logical Replication extension for PostgreSQL | Change Data Capture library

 by   2ndQuadrant C Version: REL2_4_2 License: Non-SPDX

kandi X-RAY | pglogical Summary

kandi X-RAY | pglogical Summary

pglogical is a C library typically used in Utilities, Change Data Capture, MariaDB, Amazon S3 applications. pglogical has no bugs, it has no vulnerabilities and it has medium support. However pglogical has a Non-SPDX License. You can download it from GitHub.

Logical Replication extension for PostgreSQL 15, 14, 13, 12, 11, 10, 9.6, 9.5, 9.4 (Postgres), providing much faster replication than Slony, Bucardo or Londiste, as well as cross-version upgrades.
Support
    Quality
      Security
        License
          Reuse

            kandi-support Support

              pglogical has a medium active ecosystem.
              It has 725 star(s) with 133 fork(s). There are 86 watchers for this library.
              OutlinedDot
              It had no major release in the last 12 months.
              There are 129 open issues and 235 have been closed. On average issues are closed in 144 days. There are 6 open pull requests and 0 closed requests.
              It has a neutral sentiment in the developer community.
              The latest version of pglogical is REL2_4_2

            kandi-Quality Quality

              pglogical has no bugs reported.

            kandi-Security Security

              pglogical has no vulnerabilities reported, and its dependent libraries have no vulnerabilities reported.

            kandi-License License

              pglogical has a Non-SPDX License.
              Non-SPDX licenses can be open source with a non SPDX compliant license, or non open source licenses, and you need to review them closely before use.

            kandi-Reuse Reuse

              pglogical releases are available to install and integrate.

            Top functions reviewed by kandi - BETA

            kandi's functional review helps you automatically verify the functionalities of the libraries and avoid rework.
            Currently covering the most popular Java, JavaScript and Python libraries. See a Sample of pglogical
            Get all kandi verified functions for this library.

            pglogical Key Features

            No Key Features are available at this moment for pglogical.

            pglogical Examples and Code Snippets

            No Code Snippets are available at this moment for pglogical.

            Community Discussions

            QUESTION

            AWS DMS task failing after some time in CDC mode
            Asked 2021-Jun-01 at 05:03

            I'm having trouble in setting up a task migrating the data in a RDS Database (PostgreSQL, engine 10.15) into an S3 bucket in the initial migration + CDC mode. Both endpoints are configured and tested successfully. I have created the task twice, both times it ran a couple of hours at most, the first time the initial dump went fine and some of the incremental dumps took place as well, the second time only the initial dump finished and no incremental dump was performed before the task failed.

            The error message is now:

            ...

            ANSWER

            Answered 2021-Jun-01 at 05:03

            Should anyone get the same error in the future, here is what we were told by the AWS tech specialist:

            There is a known (to AWS) issue with the pglogical plugin. The solution requires using the test_decoding plugin instead.

            1. Enforce using the test_decoding plugin on the DMS Endpoint by specifying pluginName=test_decoding in Extra Connection Attributes
            2. Create a new DMS task using this endpoint (using the old task may cause it to fail due to dissynchronization between the task and the logs)

            It sure did resolve the issue, but we still don't know what the problem really was with the plugin that is strongly suggested everywhere in the DMS documentation (at the moment).

            Source https://stackoverflow.com/questions/67634311

            QUESTION

            Upgrade Amazon Aurora PostgreSQL from 9.6 to 10
            Asked 2021-Feb-15 at 11:29

            I am trying to upgrade Amazon Aurora PostgreSQL from verion 9.6 to 10.14 via CloudFormation template command aws cloudformation update-stack --stack-name base --template-body file://cloudformation/base.yml but every time update is getting rollback with one of the below messages:

            1. Unmodifiable DB Parameter: wal_consistency_checking

            2. Unmodifiable DB Parameter: ssl_dh_params_file

            Here's the part of base.yml looks like:

            ...

            ANSWER

            Answered 2021-Feb-15 at 11:29

            Those who are finding it difficult to upgrade, here's what I did:

            • Updated the base template and added AWS::RDS::DBParameterGroup settings for Postgres 10 version
            • After that I executed aws cloudformation update-stack --stack-name base --template-body file://cloudformation/base.yml --capabilities CAPABILITY_IAM command to update the base stack
            • Then I executed: aws rds modify-db-cluster command from console

            Source https://stackoverflow.com/questions/66132886

            QUESTION

            Is it possible to replicate just one schema in postgresql?
            Asked 2020-Dec-05 at 02:16

            NOTE: a similar Q. was asked but was not answered here: How do I replicate a single schema in postgresql?

            I am trying to replicate just one schema in a postgresql (13) database using pglogical to another server.

            In my test setup on the provider node I have 2 schemas "shared" and "not_shared".

            On the subscriber node I have one existing schema called "not_shared".

            I want to replicate "shared" from the provider to the subscriber, I have already set up the nodes on both servers.

            On the provider I create a replication set for just the "shared" schema (in theory) called "shared_repl":

            ...

            ANSWER

            Answered 2020-Dec-05 at 02:16

            synchronize_structure just doesn't do what you want. You can copy the structure of just one schema from provider to subscriber yourself, using pg_dump -n shared -s for example, then don't specify synchronize_structure := TRUE when you create the subscription.

            Source https://stackoverflow.com/questions/65145005

            QUESTION

            Could not access file "pglogical" while trying to install pglogical
            Asked 2020-Oct-05 at 04:51

            I'm following instructions from https://github.com/2ndQuadrant/pglogical to install pglogical on postgres 12 on Centos 8. The install seems be successful:

            ...

            ANSWER

            Answered 2020-Oct-05 at 04:51

            Your session log tells the the server was installed as a prerequisite, but the "link" messages insinuate that there was already an incompatible client version in place. Probably you had installed PostgreSQL from the CentOS packages, but the pglogical RPMs pulled in the PGDG packages.

            The error message probably means that shared_preload_libraries contains pglogical, but pglogical.so could not be found in the lib directory.

            Presumably the installation process edited the configuration in your old server installation, but installed the shared object in the new one.

            Upshot: you cannot use those pglogical binaries with your installation. Either switch to the PGDG RPMs or build pglogical from source.

            You see that there is a certain amount of conjecture in my deductions, but that should help you solve the problem.

            Source https://stackoverflow.com/questions/64148866

            QUESTION

            How to fix `column "xlog_position" does not exist` error when using AWS DMS for Postgres to Postgres data migration
            Asked 2020-Jan-31 at 06:28

            I'm trying to migrate and synchronize a PostgreSQL database using AWS DMS and I'm getting the following error.

            ...

            ANSWER

            Answered 2020-Jan-31 at 06:28

            To enable logical decoding for an Amazon RDS for PostgreSQL DB instance

            1. The user account requires the rds_superuser role to enable logical replication. The user account also requires the rds_replication role to grant permissions to manage logical slots and to stream data using logical slots.

            2. Set the rds.logical_replication static parameter to 1. As part of applying this parameter, we also set the parameters wal_level, max_wal_senders, max_replication_slots, and max_connections. These parameter changes can increase WAL generation, so you should only set the rds.logical_replication parameter when you are using logical slots.

            3. Reboot the DB instance for the static rds.logical_replication parameter to take effect.

            4. Create a logical replication slot as explained in the next section. This process requires that you specify a decoding plugin. Currently we support the test_decoding output plugin that ships with PostgreSQL.

            The last item can be done with the following command:

            Source https://stackoverflow.com/questions/59323593

            QUESTION

            pglogical: list replication sets?
            Asked 2019-Sep-23 at 20:47

            I looked at the docs of pglogical.

            But I could not find a way to list all replication sets, which I created with create_replication_set().

            How to list all replication sets?

            ...

            ANSWER

            Answered 2019-Sep-23 at 20:47

            According to the source, that should be as simple as

            Source https://stackoverflow.com/questions/58007483

            QUESTION

            Postgres Notify not working with logical replication
            Asked 2019-Sep-18 at 14:42

            I am replicating data from Postgres 10.4 to another Postgres 10.4 instance using logical replication.

            The subscriber has several triggers that log events to a single table. This table has a trigger that executes another function (that returns a trigger) to call NOTIFY for a downstream listener.

            Trigger on the audit table looks like this:

            ...

            ANSWER

            Answered 2019-Sep-18 at 14:42

            Update: It looks like this is a bug with PostgreSQL 10.4, and at least up to at least 11.4. There's an experimental patch available here.

            According to this post on the PostgreSQL mailing list it looks like by default logical replication won't cause triggers to fire on replicas because tables generally have the "local" replication role and on logical replicas the data gets inserted with the "replica" role.

            It looks like you can alter your table to always fire triggers, including on replication by doing the following (see the documentation here):

            Source https://stackoverflow.com/questions/51457034

            QUESTION

            How can I transform a UUID to a UUID5 during select (or insert) queries in postgres?
            Asked 2019-May-01 at 16:55

            Is there a way to convert a UUID to a UUID5 in an existing postgres data set?

            I have a master/slave setup with pglogical replicating specific tables & columns. I need to give access to this replicated data to a third party, but I want to anonymise it as much as possible.

            Every row in a table uses a UUID (generated in code) as primary key, but in order to ensure anonymity I need to ensure that the primary keys of the master & replicated tables are one-way transformed to make back-tracing harder. Some of the replicated tables refer to each other as well, meaning I can't just exclude those columns - values need to match between tables. Hopefully this makes sense so far...

            Is there a way of using uuid-ossp and uuid_generate_v5() to essentially use the existing UUIDs as the namespace for a v5 UUID, returning those in the view to the third party (or even to use to transform the specific columns during replication)?

            I'm a little stumped. If I can get this sorted entirely in postgres I won't have to muck about with additional tools to transform the data before allowing the third party access, I can just give them real-time data instead of batched. But if I need to involve some additional code then I guess that's not the end of the world...

            Thanks in advance!

            ...

            ANSWER

            Answered 2019-May-01 at 16:55

            Simply provide read-only views to a third-party (in a separate schema) and limit data access to them to those views (or schema).

            A view can then be defined with a query such as:

            Source https://stackoverflow.com/questions/55938941

            QUESTION

            pglogical: REPLICA TRIGGER before delete causes database corruption
            Asked 2019-Apr-04 at 15:21

            I'm experimenting with a 2-node bidirectional replication using pglogical-9.6. I'm able to consistently cause a seg fault when I set pglogical.conflict_resolution = 'last_update_wins'.

            I'm create a BEFORE DELETE TRIGGER to prevent insertion some values into replicated table.

            ...

            ANSWER

            Answered 2019-Apr-04 at 15:21

            A segmentation fault is always a bug unless it is caused by hardware problems or data corruption.

            You should file a bug with 2ndQuadrant; it is their software.

            Make sure to include the reproducer with the bug report.

            Source https://stackoverflow.com/questions/55514283

            QUESTION

            Different select results when using multimaster via pglogical in PostgreSQL
            Asked 2019-Mar-29 at 07:35

            There are two PostgreSQL 9.6 nodes subscribed to each other via pglogical. If node A inserts a row into the replicated table then node B sees it and vice versa.

            However, when I update a row on one node, then subsequent SELECT queries on both nodes will keep returning different results - the current one and some of the previous ones.

            Moreover, there are log entries about replication conflicts in the logs of both nodes.

            Why does that happen and how do I fix that?

            upd: setting pglogical.conflict_resolution to last_update_wins helps. Might consider other options of conflict resolution too

            ...

            ANSWER

            Answered 2019-Mar-29 at 07:20

            Multi-master replication is difficult.

            There are conflicts that are bound to occur unless your application is aware of and specifically tailored to multi-master replication:

            • Rows inserted on different nodes with the same (automatically generated primary key must conflict.

            • If you modify the primary key of a row on one node while updating or deleting it on another, the databases will “drift apart”, leading to future conflicts.

            You will have to fix your application so that it avoids problems like the above, and you will have to manually find and resolve all the conflicts that occurred so far.

            Here is an example of the second case:

            Source https://stackoverflow.com/questions/55411936

            Community Discussions, Code Snippets contain sources that include Stack Exchange Network

            Vulnerabilities

            No vulnerabilities reported

            Install pglogical

            You can download it from GitHub.

            Support

            For any new features, suggestions and bugs create an issue on GitHub. If you have any questions check and ask questions on community page Stack Overflow .
            Find more information at:

            Find, review, and download reusable Libraries, Code Snippets, Cloud APIs from over 650 million Knowledge Items

            Find more libraries

            Stay Updated

            Subscribe to our newsletter for trending solutions and developer bootcamps

            Agree to Sign up and Terms & Conditions

            Share this Page

            share link

            Consider Popular Change Data Capture Libraries

            debezium

            by debezium

            libusb

            by libusb

            tinyusb

            by hathach

            bottledwater-pg

            by confluentinc

            WHID

            by whid-injector

            Try Top Libraries by 2ndQuadrant

            bdr

            by 2ndQuadrantC

            pg_stat_plans

            by 2ndQuadrantC

            ansible-ssh

            by 2ndQuadrantPython

            bdr-postgres

            by 2ndQuadrantC

            array_textsort

            by 2ndQuadrantC