sharding | Sharding manager contract, and related software and tests | Cryptocurrency library
kandi X-RAY | sharding Summary
kandi X-RAY | sharding Summary
Sharding manager contract, and related software and tests
Support
Quality
Security
License
Reuse
Top functions reviewed by kandi - BETA
- Submits vote for the given vote
- Get the member of a committee
- Check if the shard has been voted on the shard
- Get the vote count for a shard
- Add a header to the log
- Builds a signed transaction
- Make a transaction context
- Calculate the next period of the block
- Registers a new nonary message
- Submit a vote on a shard
- Add a block header
- Deregisters the given sender
- Generate a JSON file
- Update the vote for a shard
- Register a new notification
- Release a NOTary transaction
- Deregisters a notification
- Set the attributes of the log
- Parse the value of a given value
- Set the value of each topic
- Extract an eventabi
- Returns the SMC JSON
- Create a basic call context
- Make a call context
- Returns the SMC json file
sharding Key Features
sharding Examples and Code Snippets
Lines of Code : 78License : Non-SPDX (Apache License 2.0)
Copydef distribute_datasets_from_function(self, dataset_fn, options=None):
# pylint: disable=line-too-long
"""Distributes `tf.data.Dataset` instances created by calls to `dataset_fn`.
The argument `dataset_fn` that users pass in is an input def apply_to_tensor(self,
tensor,
assign_tuple_sharding=False,
use_sharding_op=False,
unspecified_dims=None):
"""Applies this Sharding attribute to `tensor`.
def split(cls, tensor, split_dimension, num_devices, input_shape=None):
"""Returns a Sharding that splits a tensor across a dimension.
This creates a Tiled attribute, similar to tile(), but easier to use for the
common case of tiling a t path = 'path_to_parent_dir'
in_paths = [path + '/1/' + f for f in ['0.png', '1.png']] + [path + '/2/' + f for f in ['0.png', '1.png']]
out_paths = [path + '/2/' + f for f in ['0.png', '1.png']] + [path + '/3/' + f for f in ['0.png', '1.pngdef map_dir(s, n=2):
import hashlib
m = hashlib.sha256(s.encode('utf-8'))
return int(m.hexdigest(), 16)%n+1
>>> map_dir('example.txt')
1
>>> map_dir('file.csv')
2
<
Lines of Code : 19License : Strong Copyleft (CC BY-SA 4.0)Copyimport multiprocessing
import os
import subprocess
def convert(objectfile):
elfdumpExePath = "C:\.....\elfdump.exe"
output_dir = "C:\.....\out"
cmd = "{elfdump} -T {obj} -o {lst}".format(
elfdump=elfdumpExePath,
zadd instrument 1577883600 672.2,432,1577883600
zadd instrument 1577883610 672.2,412,1577883610
zrangebyscore instrument 1577883600 1577883610
db.mycollection.create_index([('key1', pymongo.ASCENDING),
('key2', pymongo.ASCENDING)], unique=True)
operations = []
for doc in document_list[0:]:
key = dict((k, doc[k]) for k in ('key1', 'key2'))
odataset = dataset.apply(tf.contrib.data.unbatch())
dataset = dataset.batch(batch_size)
def gen():
for i in range(1, 5):
yield [i] * i
# Create dataset from generator
# The output shape is variable: (NoSEED = 123
os.environ['PYTHONHASHSEED']=str(SEED)
random.seed(SEED)
np.random.seed(SEED)
tf.set_random_seed(SEED)
session_config.intra_op_parallelism_threads = 1
session_config.inter_op_parallelism_threads = 1
Community Discussions
Trending Discussions on sharding
QUESTION
Im trying to addshard via router to 2 replication set on windows. I already searched a lot of similar questions and tried the same steps. But unfornately ... Below is my steps: for config node, config file:
...ANSWER
Answered 2021-Jun-03 at 19:56Have a look at your service manager services.msc, there you should be able to stop it.
or use
QUESTION
I've already setup MongoDB sharding and now I need to setup replication for availability. How do I do this? I've currently got this:
- 2 mongos instances running in different datacenters
- 2 mongod config servers running in different datacenters
- 2 mongod shard servers running in different datacenters
- all communication is over a private network setup by my provider that is available cross-datacenter
Do I just setup replication on each server (by assigning each a secondary)?
...ANSWER
Answered 2021-Jun-02 at 05:30You need 3 servers in each replica set for redundancy. Either put the third one in one of the data centers or get a third data center.
- The config replica set needs 3 servers.
- Each of the shard replica sets needs 3 servers.
- You can keep the 2 mongoses.
QUESTION
This is a follow up of my previous question. Alex Blex's solution for connecting to the config servers works great. But I am facing the same issue while connecting to the MongoDB Query router.
Below is the command I am using to create the mongos server
...ANSWER
Answered 2021-May-31 at 14:01So I figured this one out. Apparently config servers are light weight and do not store any data. Hence, we do not require to bind it to a volume. I first bound all the config servers to a fixed IP (so that docker doesn't assign them a new IP every time I stop and start a container). But for the sake of this answer, I will be using the IPs mentioned in the question itself. I used the below command to create a query router.
QUESTION
There is a user tag table table_tag, the corresponding distributed table is table_tag_all, there are 6 shards in the cluster, sharding_key is intHash64(user_id).
through setting the parameters distributed_product_mode='local' and distributed_group_by_no_merge=1 so that the returned result is the value of 6 separate shards instead of an aggregated value.
The following are two tests. Test 1 gets the correct result (6 records of count_1's number), but test 2 is aggregated(just 2 records). How can I make test 2 return the results of 6 shards?
...ANSWER
Answered 2021-May-28 at 20:30Try this query:
QUESTION
I am trying to improve a performance of some large tables (can be millions of records) in a MySQL 8.0.20 DB on RDS.
Scaling up DB instance and IOPS is not the way to go, as it is very expensive (the DB is live 24/7). Proper indexes (including composite ones) do already exist to improve the query performance. The DB is mostly read-heavy, with occasional massive writes - when these writes happen, reads can be just as massive at the same time.
I thought about doing partitioning. Since MySQL doesn't support vertical partitioning, I considered doing horizontal partitioning - which should work very well for these large tables, as they contain activity records from dozens/hundreds of accounts, and storing each account's records in a separate partition makes a lot of sense to me. But these tables do contain some constraints with foreign keys, which rules out using MySQL's horizontal partitioning : Restrictions and Limitations on Partitioning
Foreign keys not supported for partitioned InnoDB tables. Partitioned tables using the InnoDB storage engine do not support foreign keys. More specifically, this means that the following two statements are true:
No definition of an InnoDB table employing user-defined partitioning may contain foreign key references; no InnoDB table whose definition contains foreign key references may be partitioned.
No InnoDB table definition may contain a foreign key reference to a user-partitioned table; no InnoDB table with user-defined partitioning may contain columns referenced by foreign keys.
What are my options, other than doing "sharding" by using separate tables to store activity records on a per account basis? That would require a big code change to accommodate such tables. Hopefully there is a better way, that would only require changes in MySQL, and not the application code. If the code needs to be changed - the less the better :)
...ANSWER
Answered 2021-May-24 at 18:27Before sharding or partitioning, first analyze your queries to make sure they are as optimized as you can make them. This usually means designing indexes specifically to support the queries you run. You might like my presentation How to Design Indexes, Really (video).
Partitioning isn't as much a solution as people think. It has many restrictions, including the foreign key issue you found. Besides that, it only improves queries that can take advantage of partition pruning.
Also, I've done a lot of benchmarking of Amazon RDS for my current job and also a previous job. RDS is slow. It's really slow. It uses remote EBS storage, so it's bound to incur overhead for every read from storage or write to storage. RDS is just not suitable for any application that needs high performance.
Amazon Aurora is significantly better on latency and throughput. But it's also very expensive. The more you use it, the more you use I/O requests, and they charge extra for that. For a busy app, you end up spending as much as you did for RDS with high provisioned IOPS.
The only way I found to get high performance in the cloud is to forget about managed databases like RDS and Aurora, and instead install and run your own instance of MySQL on an ec2 instance with locally-attached NVMe storage. This means the i3 family of ec2 instances. But local storage is ephemeral instance storage, so if the instance restarts, you lose your data. So you must add one or more replicas and have a failover plan.
If you need an OLTP database in the cloud, and you also need top-tier performance, you either have to spend $$$ for a managed database, or else you need to hire full-time DevOps and DBA staff to run it.
Sorry to give you the bad news, but the TANSTAAFL adage remains true.
QUESTION
- I am using
Windows 10Operating system. - I have
Docker for windowsinstalled on my machine. - I have
mongo shellfor Windows installed on my machine. - I am creating the config servers using the latest
mongoimage from docker.
I am trying to create config servers (in a replica set; one primary and two secondaries) in order to set up Sharding for MongoDB. I am able to connect to the mongod servers if I create them as replica sets, without specifying the --configsvr parameter. But when I specify the --configsvr parameter, it fails with below error -
connecting to: mongodb://127.0.0.1:27017/?compressors=disabled&gssapiServiceName=mongodb Error: couldn't connect to server 127.0.0.1:27017, connection attempt failed: SocketException: Error connecting to 127.0.0.1:27017 :: caused by :: No connection could be ma de because the target machine actively refused it. : connect@src/mongo/shell/mongo.js:374:17 @(connect):2:6 exception: connect failed exiting with code 1
Case 1 - Creating 3 mongod servers as a replica set
Step 1:- Creating 3 mongod containers asia, america and europe.
...ANSWER
Answered 2021-May-20 at 13:38It's 27019 for config servers.
When you add --configsvr you need to change port mapping too:
C:/> docker run -d -p 30001:27019 -v C:/mongodata/data/db --name asiaCS mongo mongod --configsvr --bind_ip=0.0.0.0 --replSet "rs1"
QUESTION
I was reading this page, where it says that 41 bits are used to represent 41 years using a custom epoch.
I am unable to understand the relationship between time in milliseconds, bits and years. Can any one help?
Eg. In Java, System.currentTimeMillis() returns a long, which is 64 bits. Does that mean it could represent 64 years worth of unique values if I had to generate 1 per millisecond?
In the above case, what happens after 41 years? Will they have to increase the bits used to designate if they keep the same approach?
...ANSWER
Answered 2021-May-12 at 15:39Eg. In Java, System.timeinmillis() returns a long, which is 64 bits. Does that mean it could represent 64 years worth of unique values if I had to generate 1 per millisecond?
No, far more than that. Don't forget that for every bit you add in your storage, you get to store twice as many values.
2^64 is 18,446,744,073,709,551,616. That's how many distinct values can be held in a 64-bit integer data type.
So at millisecond precision, that's:
- 18,446,744,073,709,551,616 milliseconds
- 18,446,744,073,709,551 seconds
- 307,445,734,561,825 minutes
- 5,124,095,576,030 hours
- 213,503,982,334 days
- 584,542,046 years
Also known as "probably more range than you'll ever need" :)
QUESTION
import org.antlr.v4.runtime.tree.ParseTree;
import org.apache.shardingsphere.sql.parser.core.parser.SQLParserExecutor;
import org.junit.Test;
import javax.xml.bind.SchemaOutputResolver;
public class T1 {
@Test
public void t1() {
ParseTree parseTree = new SQLParserExecutor("MySQL", "insert into T_NAME (POINT) values (?)").execute().getRootNode();
}
}
ANSWER
Answered 2021-May-07 at 13:45I suspect that all (MySQL), or more, keywords would trigger this error (POLYGON probably also produces this error). The grammar probably is trying to match an identifier, but since the input POINTS is already matched as a keyword, it fails to match it properly.
Something like this:
QUESTION
I have a problem with rewriting my app to Akka Cluster Sharding. I have a Sharded Entity let's call it A, and a bunch of local actors on each nodes, let's call one of them B. Now I send a message B -> A containing (String, ActorRef[B]) and I want to respond A -> B by using the ref provided in the previous message.
On one hand, documentation suggest it should work https://doc.akka.io/docs/akka/current/typed/cluster-sharding.html#basic-example
But as far as I understand, it should not be possible for A to locate actor B in the cluster because it's not the entityID.
How does it work? Do I have to make B an Entity as well?
ANSWER
Answered 2021-May-10 at 15:39An ActorRef is location transparent: it includes the information needed to route the message to an actor in a different ActorSystem (which typically maps 1:1 to a cluster node). If you have an ActorRef for actor B, you can send it a message regardless of where in the cluster you are.
So then why have cluster sharding, when you can always send messages across the cluster?
Cluster sharding allows entities to be addressable independently of any actor's lifecycle: an incarnation of an entity runs as an actor and sharding manages spawning an actor to serve as an incarnation on demand, limits an entity to at most one incarnation at a given time, and reserves the right to move an entity to an incarnation on a different node (typically in response to cluster membership changes). If you don't need those aspects for a particular type of actor, there's no need to make it a sharded entity.
QUESTION
add Secrets in Portainer swarm and trying to import as variable any one could give example how i can import it to compose
...ANSWER
Answered 2021-May-05 at 22:53Docker secrets will be mounted as files in the container under /run/secrets/secret-name (if no explicit mount point was specified). To use it, the application must be able to read the data from these files. That's not always supported. If anything, only a small part of the available variables can usually specified as file.
The official Docker mongodb Image states support only for
MONGO_INITDB_ROOT_USERNAME_FILE and MONGO_INITDB_ROOT_PASSWORD_FILE.
The readme from the bitnami/mongodb-sharded image doesn't provide any info, wether there is support for docker secrets.
The compose file with pre definied secrets for the official image would look something like that:
Community Discussions, Code Snippets contain sources that include Stack Exchange Network
Vulnerabilities
No vulnerabilities reported
Install sharding
You can use sharding like any standard Python library. You will need to make sure that you have a development environment consisting of a Python distribution including header files, a compiler, pip, and git installed. Make sure that your pip, setuptools, and wheel are up to date. When using pip it is generally recommended to install packages in a virtual environment to avoid changes to the system.
Support
Reuse Trending Solutions
Find, review, and download reusable Libraries, Code Snippets, Cloud APIs from over 650 million Knowledge Items
Find more librariesStay Updated
Subscribe to our newsletter for trending solutions and developer bootcamps
Share this Page
