ec2instances.info | Amazon EC2 instance comparison site | AWS library
kandi X-RAY | ec2instances.info Summary
kandi X-RAY | ec2instances.info Summary
I was sick of comparing EC2 instance metrics and pricing on Amazon's site so I made EC2Instances.info. EC2Instances.info was originally created by Garret Heaton, is now hosted by Vantage and developed by the community of contributors.
Support
Quality
Security
License
Reuse
Top functions reviewed by kandi - BETA
Currently covering the most popular Java, JavaScript and Python libraries. See a Sample of ec2instances.info
ec2instances.info Key Features
ec2instances.info Examples and Code Snippets
Community Discussions
Trending Discussions on ec2instances.info
QUESTION
I have some containers with GPU Tensorflow jobs, and, if 2+ of them are executed simultaneously on a single host, only 1 will succeed (2018-05-11 13:02:19.147869: E tensorflow/core/common_runtime/direct_session.cc:171] Internal: failed initializing StreamExecutor for CUDA device ordinal 0: Internal: failed call to cuDevicePrimaryCtxRetain: CUDA_ERROR_ECC_UNCORRECTABLE
, i.e. they cannot share GPUs properly).
Perfect scenario would be like following: I have 10 GPU jobs and max 5 containers. First 5 are executed, other 5 wait (at the moment, they don't wait but try to execute and fail), when one finished, 6th immediately starts on the same host, then 7th, 8th, 9th, 10th.
I use p2.xlarge, and set up 4 vCPU and 42000 memory for gpu job. According to ec2instances.info, this machine has 61.0 GiB memory and 4 vCPUs. But, anyway, batch seems to schedule several containers simultaneously, leading to described failure.
So far I tried to play with vCPU
and memory
parameters, but Batch's scheduler seems to ignore those.
Interesting that relevant ECS task definition has 1/--
as value for Hard/Soft memory limits (MiB)
, so looks like values from Batch 'job definition' are not propagated to ECS 'task definition'.
Another alternative is to setup a very big number of attempts, but
- it's ugly
- for long-running jobs even big number might get exhausted
- I lose defense from forever-running jobs (e.g. mis-configured)
- not sure how that kind of interruption would affect already running Tensorflow jobs
ANSWER
Answered 2018-May-11 at 20:31What is the vCPU and Memory requirement of your Jobs, what are the instance types in your compute environment ?
If you update the vCpu and Memory of your jobs so that only one job can fit in an instance, Batch will schedule your jobs one after the other and not try to run two jobs at the same time.
For example if your Compute environment has p3.16xlarge (64vCpus,488Gib) instances and want to ensure that only one jobs runs in the instance at a time, make sure that the job specifies vCPU > 32 and Memory > 244GB
QUESTION
I am struggling to load most of the Drug Ontology OWL files and most of the ChEBI OWL files into GraphDB free v8.3 repository with Optimized OWL Horst reasoning on.
is this possible? Should I do something other than "be patient?"
Details:
I'm using the loadrdf offline bulk loader to populate an AWS r4.16xlarge instance with 488.0 GiB and 64 vCPUs
Over the weekend, I played around with different pool buffer sizes and found that most of these files individually load fastest with a pool buffer of 2,000 or 20,000 statements instead of the suggested 200,000. I also added -Xmx470g
to the loadrdf script. Most of the OWL files would load individually in less than one hour.
Around 10 pm EDT last night, I started to load all of the files listed below simultaneously. Now it's 11 hours later, and there are still millions of statements to go. The load rate is around 70/second now. It appears that only 30% of my RAM is being used, but the CPU load is consistently around 60.
- are there websites that document other people doing something of this scale?
- should I be using a different reasoning configuration? I chose this configuration as it was the fastest loading OWL configuration, based on my experiments over the weekend. I think I will need to look for relationships that go beyond rdfs:subClassOf.
Files I'm trying to load:
...ANSWER
Answered 2017-Dec-18 at 16:45@MarkMiller you can take a look at the Preload tool, which is part of GraphDB 8.4.0 release. It's specially designed to handle large amount of data with constant speed. Note that it works without inference, so you'll need to load your data and then change the ruleset and reinfer the statements.
http://graphdb.ontotext.com/documentation/free/loading-data-using-preload.html
Community Discussions, Code Snippets contain sources that include Stack Exchange Network
Vulnerabilities
No vulnerabilities reported
Install ec2instances.info
Support
Reuse Trending Solutions
Find, review, and download reusable Libraries, Code Snippets, Cloud APIs from over 650 million Knowledge Items
Find more librariesStay Updated
Subscribe to our newsletter for trending solutions and developer bootcamps
Share this Page