icecat | Icecat API - Find product description | REST library
kandi X-RAY | icecat Summary
kandi X-RAY | icecat Summary
Icecat API - Find product description with EAN, UPC or GTIN-13
Support
Quality
Security
License
Reuse
Top functions reviewed by kandi - BETA
Currently covering the most popular Java, JavaScript and Python libraries. See a Sample of icecat
icecat Key Features
icecat Examples and Code Snippets
Community Discussions
Trending Discussions on icecat
QUESTION
Can I do threads instead of tasks to make this run faster? I'm trying to get 114000 products into the database. As my code is right now I get about 100 products into the database a minute.
My Tasks (producers) each scrape an XML
File which contains a product data, packages it in the Product class, then queue's it for the consumer.
my Consumer takes each product from the queue and puts it into the database 1 at a time. I use Entity Framework so it's not safe for threading.
...ANSWER
Answered 2020-Feb-07 at 14:02First, read the speed rant to make sure this is even worth investigating.
Can I do threads instead of tasks to make this run faster?
Extremely unlikely. Multithreading has been used as a cheap way to implement Multitasking for a while, but it is technically only usefull if the task is CPU bound. You are doing a DB operation. That one will be Network bound. More likely DB bound (they apply additional bottlenecks as part of their reliabiltiy and concurrency issue prevention).
Im trying to get 114000 products into the db.
Then your best bet is not trying to do that in code. Every DBMS worth it's memory footprint has bulk insert options. Doing that in C# code? That will just make it slower and less reliable.
At best you add the Network load to send the data to the DB to the whole operation. At worst, you make it even slower then that. It is one of the most common mistakes with DB's, thinking you can beat the DBMS performance with code. It will not work.
QUESTION
Im trying to run a producer consumer pattern on my code to make it faster. My process is that I want to use multiple tasks running at the same time, for getting data, wrapping it in a custom class Product and then adding the product to a queue. Then the consumer saves it in the DB through Entity Framework through a single task. I tested the code I have now by inserting around 1000 products in the database, and then running sql query to check for duplicate rows. SQL Query result
As you can see on the picture around 30 products appear more then once in the database.
This is my code:
...ANSWER
Answered 2020-Feb-06 at 13:15To avoid duplication try adding new field to Product table and lets name it Code
which contains the hash of the product`s xml.
And make sure to add unique attribute to it.
this way any attempt to add a duplication will fail due to the unique constraint/index.
Community Discussions, Code Snippets contain sources that include Stack Exchange Network
Vulnerabilities
No vulnerabilities reported
Install icecat
Support
Reuse Trending Solutions
Find, review, and download reusable Libraries, Code Snippets, Cloud APIs from over 650 million Knowledge Items
Find more librariesStay Updated
Subscribe to our newsletter for trending solutions and developer bootcamps
Share this Page