pgsync | Postgres to Elasticsearch/OpenSearch sync

by toluaina Python Version: 3.1.0 License: LGPL-3.0

X-Ray Key Features Code Snippets Community Discussions(5)Vulnerabilities Install Support

kandi X-RAY | pgsync Summary

pgsync is a Python library typically used in Big Data, Spark applications. pgsync has no bugs, it has no vulnerabilities, it has build file available, it has a Weak Copyleft License and it has medium support. You can install using 'pip install pgsync' or download it from GitHub, PyPI.

Postgres to elasticsearch sync

Support

Quality

Security

License

Reuse

Support

pgsync has a medium active ecosystem.

It has 860 star(s) with 136 fork(s). There are 20 watchers for this library.

There were 1 major release(s) in the last 6 months.

There are 123 open issues and 234 have been closed. On average issues are closed in 29 days. There are 5 open pull requests and 0 closed requests.

It has a neutral sentiment in the developer community.

The latest version of pgsync is 3.1.0

Quality

pgsync has 0 bugs and 0 code smells.

Security

pgsync has no vulnerabilities reported, and its dependent libraries have no vulnerabilities reported.

pgsync code analysis shows 0 unresolved vulnerabilities.

There are 0 security hotspots that need review.

License

pgsync is licensed under the LGPL-3.0 License. This license is Weak Copyleft.

Weak Copyleft licenses have some restrictions, but you can use them in commercial projects.

Reuse

pgsync releases are available to install and integrate.

Deployable package is available in PyPI.

Build file is available. You can build the component from source.

Installation instructions are not available. Examples and code snippets are available.

Top functions reviewed by kandi - BETA

kandi has reviewed pgsync and discovered the below as its top functions. This is intended to give you an instant insight into pgsync implemented functionality, and help decide if they suit your requirements.

Validate the schema
Check if a replication slot can create a replication slot
Fetch a single row from the database
Create a replication slot
Create a pg engine
Return auth string
Construct a Postgres URL
Create a pg engine instance
Tear down the database
Load config files
Get schema configuration
Build subquery
Create a WSGI application
Load template from config file
Get authentication key
Get current transaction id
Enable triggers for a schema
Disable triggers
Get schema config
Delete an index
Creates a controlled sub - transaction
Poll the database for notifications
Show the settings
Delete nsize rows
Initialize the database
Get logging configuration
Analyze the table
Poll the database

Get all kandi verified functions for this library.

pgsync Key Features

No Key Features are available at this moment for pgsync.

pgsync Examples and Code Snippets

No Code Snippets are available at this moment for pgsync.

Community Discussions

Trending Discussions on pgsync

Django "surrogates not allowed" error on model.save() call when text includes emoji character

How i can change _type param in during import data into elasticsearch?

How to add (text)fields to all field in pgsync / elasticsearch?

How to get version number and then compare it to minimum

Looking for a solution to synchronize data between Postgres database running in AWS Aurora DB (pgsync maybe?)

QUESTION

Django "surrogates not allowed" error on model.save() call when text includes emoji character

Asked 2022-Mar-18 at 20:40

We are currently in the process of building a system that stores text in a PostgreSQL DB via Django. The data gets then extracted via PGSync to ElasticSearch.

At the moment we have encountered the following issue in a testcase

Error Message:

UnicodeEncodeError: 'utf-8' codec can't encode characters in position 159-160: surrogates not allowed

We identified the character that causes that issue. It is an emoji.

The text itself is a mixture of Greek Characters, "English Characters" and as it seems emojis. The greek is not shown as greek, but instead in the \u form.

Relevant Text that causes the issue:

...

ANSWER

Answered 2022-Mar-18 at 20:22

Reproduce the issue:

Source https://stackoverflow.com/questions/71532212

QUESTION

How i can change _type param in during import data into elasticsearch?

Asked 2021-Nov-28 at 14:11

I getting this

while import with schema.json description from postgres via pgsync

I want to override the value from _type: _doc to something else. Can you help me?

...

ANSWER

Answered 2021-Nov-28 at 14:11

This is not possible, _type is a metadata field that was used until version 6.X to have different document types in the same index, this is not possible anymore.

In version 6.X you could have indices with a custom value for _type, but just one per index, on version 7.X the value for the _type field is _doc, and on the upcoming version 8.X this field and its related API won't exist anymore.

You can however create a new field named type and use any value you want, how you will do this depends on how you are ingesting your data.

For more information about the removal of types in elasticsearch you can check this official blog post and this part of the documentation.

Source https://stackoverflow.com/questions/70143986

QUESTION

How to add (text)fields to all field in pgsync / elasticsearch?

Asked 2021-Jul-14 at 20:05

I am exploring pgsync to add elasticsearch support for some tables, however, I would like to be able to copy all the textfields to one "all" field. Elastic has support for this in the form of a mapping to a group field, with copy-to, see here

Like this:

...

ANSWER

Answered 2021-Jul-14 at 20:05

You can do this in pgsync by creating a transform node with a mapping type. Here is an example of how to achieve this for the book example defined here

Source https://stackoverflow.com/questions/68364932

QUESTION

How to get version number and then compare it to minimum

Asked 2021-Jul-06 at 20:24

I am using this answer to compare the min version number that is required. But before i go to comparison, I am actually stuck on how to extract the version number.

My current script looks like this

...

ANSWER

Answered 2021-Jul-06 at 20:24

If you're trying to capture a command's output in a variable and it's instead getting printed to the terminal, that's a sign the command isn't writing to its standard output, but to another stream - usually standard error. So just redirect it:

Source https://stackoverflow.com/questions/68274971

QUESTION

Looking for a solution to synchronize data between Postgres database running in AWS Aurora DB (pgsync maybe?)

Asked 2020-Jan-02 at 16:20

We are trying to find a solution to synchronize the data from one database (let's call it Master) database to another database (let's call it Analytics) to perform ETL operations in Analytics.

The setup would be as follows:

Tables from Master would be replicated in Analytics
Replicated tables in Analytics would be read-only in analytics but used in certain ETL scripts (some of them purely SQL) to perform some insertions in other tables in Analytics

From our initial research it seems PGSync would be a good solution to perform this. We have been trying AWS DMS as well but it created too much of a drag on resources on Master.

Has anyone attempted a similar setup in the past? Would love to hear your thoughts!

...

ANSWER

Answered 2020-Jan-02 at 16:15

You can create read_replicas in AWS RDS here is some additional info. I think this is what you are searching for - note that for aurora Read replicas can also become master in case of a failure.

Source https://stackoverflow.com/questions/59566327

Community Discussions, Code Snippets contain sources that include Stack Exchange Network

Vulnerabilities

No vulnerabilities reported

Install pgsync

You can install using 'pip install pgsync' or download it from GitHub, PyPI.
You can use pgsync like any standard Python library. You will need to make sure that you have a development environment consisting of a Python distribution including header files, a compiler, pip, and git installed. Make sure that your pip, setuptools, and wheel are up to date. When using pip it is generally recommended to install packages in a virtual environment to avoid changes to the system.

Support

For any new features, suggestions and bugs create an issue on GitHub. If you have any questions check and ask questions on community page Stack Overflow .

Find more information at: