sharding | PostgreSQL sharding for go-pg and Golang | SQL Database library

by go-pg Go Version: v8.0.0 License: BSD-2-Clause

X-Ray Key Features Code Snippets(3)Community Discussions(10)Vulnerabilities Install Support

kandi X-RAY | sharding Summary

sharding is a Go library typically used in Database, SQL Database, PostgresSQL applications. sharding has no bugs, it has no vulnerabilities, it has a Permissive License and it has low support. You can download it from GitHub.

:heart: Uptrace.dev - distributed traces, logs, and errors in one place. This package uses a go-pg PostgreSQL client to help sharding your data across a set of PostgreSQL servers as described in Sharding & IDs at Instagram. In 2 words it maps many (2048-8192) logical shards implemented using PostgreSQL schemas to far fewer physical PostgreSQL servers. API docs: Examples:

Support

Quality

Security

License

Reuse

Support

sharding has a low active ecosystem.

It has 246 star(s) with 23 fork(s). There are 10 watchers for this library.

It had no major release in the last 6 months.

There are 1 open issues and 1 have been closed. On average issues are closed in 3 days. There are 4 open pull requests and 0 closed requests.

It has a neutral sentiment in the developer community.

The latest version of sharding is v8.0.0

Quality

sharding has no bugs reported.

Security

sharding has no vulnerabilities reported, and its dependent libraries have no vulnerabilities reported.

License

sharding is licensed under the BSD-2-Clause License. This license is Permissive.

Permissive licenses have the least restrictions, and you can use them in most projects.

Reuse

sharding releases are not available. You will need to build from source code and install.

Installation instructions, examples and code snippets are available.

Top functions reviewed by kandi - BETA

kandi's functional review helps you automatically verify the functionalities of the libraries and avoid rework.
Currently covering the most popular Java, JavaScript and Python libraries. See a Sample of sharding

Get all kandi verified functions for this library.

sharding Key Features

No Key Features are available at this moment for sharding.

sharding Examples and Code Snippets

Distribute datasets from a function .

python

Lines of Code : 78

License : Non-SPDX (Apache License 2.0)

Copy

def distribute_datasets_from_function(self, dataset_fn, options=None):
    # pylint: disable=line-too-long
    """Distributes `tf.data.Dataset` instances created by calls to `dataset_fn`.

    The argument `dataset_fn` that users pass in is an input

Applies sharding operations to a tensor .

python

Lines of Code : 43

License : Non-SPDX (Apache License 2.0)

Copy

def apply_to_tensor(self,
                      tensor,
                      assign_tuple_sharding=False,
                      use_sharding_op=False,
                      unspecified_dims=None):
    """Applies this Sharding attribute to `tensor`.

Creates a Sharding Sharding .

python

Lines of Code : 34

License : Non-SPDX (Apache License 2.0)

Copy

def split(cls, tensor, split_dimension, num_devices, input_shape=None):
    """Returns a Sharding that splits a tensor across a dimension.

    This creates a Tiled attribute, similar to tile(), but easier to use for the
    common case of tiling a t

Community Discussions

Trending Discussions on sharding

mongodb this db does not have sharding enabled even though i did connect to mongos

MongoDB Sharding and Replication

Connect to MongoDB query router for Sharding on a docker container running on windows10

Expect the distributed table to return the results of each shard, not the aggregated value

Improving MySQL performance on RDS by partitioning

Connect to MongoDB config server for Sharding on a docker container running on windows10

Understanding bits and time in milliseconds

Antlr no viable alternative at input when the keyword is POINT

Akka Cluster Sharding - Entity to Actor communication

Portainer Docker Swarm import secrets to compose

QUESTION

mongodb this db does not have sharding enabled even though i did connect to mongos

Asked 2021-Jun-03 at 19:56

Im trying to addshard via router to 2 replication set on windows. I already searched a lot of similar questions and tried the same steps. But unfornately ... Below is my steps: for config node, config file:

...

ANSWER

Answered 2021-Jun-03 at 19:56

Have a look at your service manager services.msc, there you should be able to stop it.

or use

Source https://stackoverflow.com/questions/67798868

QUESTION

MongoDB Sharding and Replication

Asked 2021-Jun-02 at 22:04

I've already setup MongoDB sharding and now I need to setup replication for availability. How do I do this? I've currently got this:

2 mongos instances running in different datacenters
2 mongod config servers running in different datacenters
2 mongod shard servers running in different datacenters
all communication is over a private network setup by my provider that is available cross-datacenter

Do I just setup replication on each server (by assigning each a secondary)?

...

ANSWER

Answered 2021-Jun-02 at 05:30

You need 3 servers in each replica set for redundancy. Either put the third one in one of the data centers or get a third data center.

The config replica set needs 3 servers.
Each of the shard replica sets needs 3 servers.
You can keep the 2 mongoses.

Source https://stackoverflow.com/questions/67795965

QUESTION

Connect to MongoDB query router for Sharding on a docker container running on windows10

Asked 2021-May-31 at 14:01

This is a follow up of my previous question. Alex Blex's solution for connecting to the config servers works great. But I am facing the same issue while connecting to the MongoDB Query router.

Below is the command I am using to create the mongos server

...

ANSWER

Answered 2021-May-31 at 14:01

So I figured this one out. Apparently config servers are light weight and do not store any data. Hence, we do not require to bind it to a volume. I first bound all the config servers to a fixed IP (so that docker doesn't assign them a new IP every time I stop and start a container). But for the sake of this answer, I will be using the IPs mentioned in the question itself. I used the below command to create a query router.

Source https://stackoverflow.com/questions/67670978

QUESTION

Expect the distributed table to return the results of each shard, not the aggregated value

Asked 2021-May-28 at 20:30

There is a user tag table table_tag, the corresponding distributed table is table_tag_all, there are 6 shards in the cluster, sharding_key is intHash64(user_id).

through setting the parameters distributed_product_mode='local' and distributed_group_by_no_merge=1 so that the returned result is the value of 6 separate shards instead of an aggregated value.

The following are two tests. Test 1 gets the correct result (6 records of count_1's number), but test 2 is aggregated(just 2 records). How can I make test 2 return the results of 6 shards?

...

ANSWER

Answered 2021-May-28 at 20:30

Try this query:

Source https://stackoverflow.com/questions/67718009

QUESTION

Improving MySQL performance on RDS by partitioning

Asked 2021-May-25 at 17:43

I am trying to improve a performance of some large tables (can be millions of records) in a MySQL 8.0.20 DB on RDS.

Scaling up DB instance and IOPS is not the way to go, as it is very expensive (the DB is live 24/7). Proper indexes (including composite ones) do already exist to improve the query performance. The DB is mostly read-heavy, with occasional massive writes - when these writes happen, reads can be just as massive at the same time.

I thought about doing partitioning. Since MySQL doesn't support vertical partitioning, I considered doing horizontal partitioning - which should work very well for these large tables, as they contain activity records from dozens/hundreds of accounts, and storing each account's records in a separate partition makes a lot of sense to me. But these tables do contain some constraints with foreign keys, which rules out using MySQL's horizontal partitioning : Restrictions and Limitations on Partitioning

Foreign keys not supported for partitioned InnoDB tables. Partitioned tables using the InnoDB storage engine do not support foreign keys. More specifically, this means that the following two statements are true:

No definition of an InnoDB table employing user-defined partitioning may contain foreign key references; no InnoDB table whose definition contains foreign key references may be partitioned.

No InnoDB table definition may contain a foreign key reference to a user-partitioned table; no InnoDB table with user-defined partitioning may contain columns referenced by foreign keys.

What are my options, other than doing "sharding" by using separate tables to store activity records on a per account basis? That would require a big code change to accommodate such tables. Hopefully there is a better way, that would only require changes in MySQL, and not the application code. If the code needs to be changed - the less the better :)

...

ANSWER

Answered 2021-May-24 at 18:27

Before sharding or partitioning, first analyze your queries to make sure they are as optimized as you can make them. This usually means designing indexes specifically to support the queries you run. You might like my presentation How to Design Indexes, Really (video).

Partitioning isn't as much a solution as people think. It has many restrictions, including the foreign key issue you found. Besides that, it only improves queries that can take advantage of partition pruning.

Also, I've done a lot of benchmarking of Amazon RDS for my current job and also a previous job. RDS is slow. It's really slow. It uses remote EBS storage, so it's bound to incur overhead for every read from storage or write to storage. RDS is just not suitable for any application that needs high performance.

Amazon Aurora is significantly better on latency and throughput. But it's also very expensive. The more you use it, the more you use I/O requests, and they charge extra for that. For a busy app, you end up spending as much as you did for RDS with high provisioned IOPS.

The only way I found to get high performance in the cloud is to forget about managed databases like RDS and Aurora, and instead install and run your own instance of MySQL on an ec2 instance with locally-attached NVMe storage. This means the i3 family of ec2 instances. But local storage is ephemeral instance storage, so if the instance restarts, you lose your data. So you must add one or more replicas and have a failover plan.

If you need an OLTP database in the cloud, and you also need top-tier performance, you either have to spend $$$ for a managed database, or else you need to hire full-time DevOps and DBA staff to run it.

Sorry to give you the bad news, but the TANSTAAFL adage remains true.

Source https://stackoverflow.com/questions/67676942

QUESTION

Connect to MongoDB config server for Sharding on a docker container running on windows10

Asked 2021-May-20 at 13:38

I am using Windows 10 Operating system.
I have Docker for windows installed on my machine.
I have mongo shell for Windows installed on my machine.
I am creating the config servers using the latest mongo image from docker.

I am trying to create config servers (in a replica set; one primary and two secondaries) in order to set up Sharding for MongoDB. I am able to connect to the mongod servers if I create them as replica sets, without specifying the --configsvr parameter. But when I specify the --configsvr parameter, it fails with below error -

connecting to: mongodb://127.0.0.1:27017/?compressors=disabled&gssapiServiceName=mongodb Error: couldn't connect to server 127.0.0.1:27017, connection attempt failed: SocketException: Error connecting to 127.0.0.1:27017 :: caused by :: No connection could be ma de because the target machine actively refused it. : connect@src/mongo/shell/mongo.js:374:17 @(connect):2:6 exception: connect failed exiting with code 1

Case 1 - Creating 3 mongod servers as a replica set

Step 1:- Creating 3 mongod containers asia, america and europe.

...

ANSWER

Answered 2021-May-20 at 13:38

It's 27019 for config servers.

When you add --configsvr you need to change port mapping too:

C:/> docker run -d -p 30001:27019 -v C:/mongodata/data/db --name asiaCS mongo mongod --configsvr --bind_ip=0.0.0.0 --replSet "rs1"

Source https://stackoverflow.com/questions/67620803

QUESTION

Understanding bits and time in milliseconds

Asked 2021-May-15 at 20:38

I was reading this page, where it says that 41 bits are used to represent 41 years using a custom epoch.

I am unable to understand the relationship between time in milliseconds, bits and years. Can any one help?

Eg. In Java, System.currentTimeMillis() returns a long, which is 64 bits. Does that mean it could represent 64 years worth of unique values if I had to generate 1 per millisecond?

In the above case, what happens after 41 years? Will they have to increase the bits used to designate if they keep the same approach?

...

ANSWER

Answered 2021-May-12 at 15:39

Eg. In Java, System.timeinmillis() returns a long, which is 64 bits. Does that mean it could represent 64 years worth of unique values if I had to generate 1 per millisecond?

No, far more than that. Don't forget that for every bit you add in your storage, you get to store twice as many values.

2^64 is 18,446,744,073,709,551,616. That's how many distinct values can be held in a 64-bit integer data type.

So at millisecond precision, that's:

18,446,744,073,709,551,616 milliseconds
18,446,744,073,709,551 seconds
307,445,734,561,825 minutes
5,124,095,576,030 hours
213,503,982,334 days
584,542,046 years

Also known as "probably more range than you'll ever need" :)

Source https://stackoverflow.com/questions/67505496

QUESTION

Antlr no viable alternative at input when the keyword is POINT

Asked 2021-May-13 at 10:42

import org.antlr.v4.runtime.tree.ParseTree;
import org.apache.shardingsphere.sql.parser.core.parser.SQLParserExecutor;
import org.junit.Test;

import javax.xml.bind.SchemaOutputResolver;

public class T1 {
    @Test
    public void t1() {
        ParseTree parseTree = new SQLParserExecutor("MySQL", "insert into T_NAME (POINT) values (?)").execute().getRootNode();
    }
}

...

ANSWER

Answered 2021-May-07 at 13:45

I suspect that all (MySQL), or more, keywords would trigger this error (POLYGON probably also produces this error). The grammar probably is trying to match an identifier, but since the input POINTS is already matched as a keyword, it fails to match it properly.

Something like this:

Source https://stackoverflow.com/questions/67415281

QUESTION

Akka Cluster Sharding - Entity to Actor communication

Asked 2021-May-10 at 15:39

I have a problem with rewriting my app to Akka Cluster Sharding. I have a Sharded Entity let's call it A, and a bunch of local actors on each nodes, let's call one of them B. Now I send a message B -> A containing (String, ActorRef[B]) and I want to respond A -> B by using the ref provided in the previous message.

On one hand, documentation suggest it should work https://doc.akka.io/docs/akka/current/typed/cluster-sharding.html#basic-example

But as far as I understand, it should not be possible for A to locate actor B in the cluster because it's not the entityID.

How does it work? Do I have to make B an Entity as well?

...

ANSWER

Answered 2021-May-10 at 15:39

An ActorRef is location transparent: it includes the information needed to route the message to an actor in a different ActorSystem (which typically maps 1:1 to a cluster node). If you have an ActorRef for actor B, you can send it a message regardless of where in the cluster you are.

So then why have cluster sharding, when you can always send messages across the cluster?

Cluster sharding allows entities to be addressable independently of any actor's lifecycle: an incarnation of an entity runs as an actor and sharding manages spawning an actor to serve as an incarnation on demand, limits an entity to at most one incarnation at a given time, and reserves the right to move an entity to an incarnation on a different node (typically in response to cluster membership changes). If you don't need those aspects for a particular type of actor, there's no need to make it a sharded entity.

Source https://stackoverflow.com/questions/67457214

QUESTION

Portainer Docker Swarm import secrets to compose

Asked 2021-May-06 at 05:32

add Secrets in Portainer swarm and trying to import as variable any one could give example how i can import it to compose

...

ANSWER

Answered 2021-May-05 at 22:53

Docker secrets will be mounted as files in the container under /run/secrets/secret-name (if no explicit mount point was specified). To use it, the application must be able to read the data from these files. That's not always supported. If anything, only a small part of the available variables can usually specified as file.

The official Docker mongodb Image states support only for MONGO_INITDB_ROOT_USERNAME_FILE and MONGO_INITDB_ROOT_PASSWORD_FILE.

The readme from the bitnami/mongodb-sharded image doesn't provide any info, wether there is support for docker secrets.

The compose file with pre definied secrets for the official image would look something like that:

Source https://stackoverflow.com/questions/67408934

Community Discussions, Code Snippets contain sources that include Stack Exchange Network

Vulnerabilities

No vulnerabilities reported

Install sharding

This package requires Go modules support:.

Support

For any new features, suggestions and bugs create an issue on GitHub. If you have any questions check and ask questions on community page Stack Overflow .

Find more information at: