curl_multi | PHP多线程数据采集 | Command Line Interface library

I have a function which uses curl_multi to make several GET requests. However, I can't figure out how to turn the response to JSON so I could access it on the front end. Usually I use json_decode(), however, it is not working this time. Maybe it's because I'm trying to decode a whole array of strings instead of a singular string?. Currently the response looks like this on the front-end:

And here's my function:

I'm getting "Could not resolve host" when I call curl_error($ch) in curl_multi context, the errno is 0 though, and the request is done successfully also(I'm getting the expected response from the remote server).

I'm getting this error for all the requests in the curl_multi group, except for only the first request of them which is done without any error messages!

(for example if the curl_multi has 10 sub requests attached to it, only the first request will not have the error and the other 9 will have the error)

My development organization has its own wrapper implementation for threads and select(). The application needs to be enhanced to perform HTTPS requests, and I've decided to use libcurl. After some research, I see that the curl_easy_perform is a blocking call, so I've decided to use the curl_multi_perform approach for non-blocking calls to allow other work in the threads.

The HTTPS request will need to be performed periodically to the same URL. I know I can keep the same curl_easy handle and give that to the curl_multi handle. I would perform the curl_multi_perform to get the results, but I would later need to use curl_multi_perform to resend the request, say in 5 minutes. So, this would be a consecutive request using the same easy handle. However, I'm not sure how the curl_easy interface tells the multi interface as to when to resend the request after I have received results from the first request. How do I accomplish this?

(Maybe drop the easy handle from the multi handle, and re-add it to the multi handle when a request is needed again?)

I presume that whatever technique is used, that the outgoing request will be using the same outgoing port.

Hi everyone. So I need to check huge amount of rows in database via API calls. I managed to do that but it is very slow.

Here are steps my code is doing:

1.) I select unique identifiers from my local database.

2.) I check them on remote database via curl_multi (100 at the time).

3.) Depending on status code I update rows in my database.

4.) Repeat first 3 steps until all rows are checked.

How can I speedup this process ? At this time checking 600 rows takes more than one minute.

I have a script that make a few calls. The 1st call get a array of keys and hash code. Then goes into a foreach and calls out to a 2nd api that will return a list of items that is then pass into a 2nd foreach loop that work with that data and post to a mysql data base. it take 11 hours to run the script i need ideas on how i can speed this up have look in to multi processing but everything i see says it a dad idea to make web request. Any ideas or links are welcome. Here is the code

I use curl_multi to scrap several site, and I hope it will run asynchronously or parallel so it will not other to finish first.

I use jquery to trigger xmlhttprequest to php.

But what I get it still run synchronous, I dont know what is wrong whether my code or my PC specification (I run local server use core-i5 and 4GB memory)

This is my jquery code

References:

http://php.net/manual/en/language.variables.variable.php

http://php.net/manual/en/language.variables.scope.php

Why does the following (example) code block:

Is curl_multi interface spawning new threads internally to handle multiple requests concurrently? Is it equal to spawning threads manually and just using curl_easy handles? What is more performant. I need to make up to 1000 concurrent requests.

https://curl.haxx.se/libcurl/c/multithread.html

Is using curl_multi equal to the example in the link above?

From: https://curl.haxx.se/libcurl/c/libcurl-multi.html

Enable multiple simultaneous transfers in the same thread without making it complicated for the application.

What does it mean? How does it handle multiple transfers in the same thread? I could as well create 100 threads with 100 curl_easy handles and make requests there.

Maybe the question should be: When to use multiple threads and when to use curl_multi.

I have a PHP script which runs many http requests via curl - I use a variation on Rolling Curl (curl_multi) so that the requests can be run simultaneously. The script runs every few minutes with cron.

It is on a VPS and I received some 'Excessive resource usage' warnings from lfd (ConfigServer Security & Firewall) because the resource usage of the script went over the threshold of 512MB.

An example notification is:

Resource: Virtual Memory Size

Exceeded: 582 > 512 (MB)

Executable: /path/to/php-cgi

Command Line: /path/to/myscript.php

So I upped the threshold to 800MB and recorded the memory usage of the script using memory_get_peak_usage(true) every time the script runs.

However, the results from memory_get_peak_usage(true) are consistently 2MB... which is nowhere near the Virtual Memory usage as seen in the warning.

Note - only one instance of the script runs as multiple instances are prevented using flock.

So what am I missing?

Update - virtual memory usage also greater than php.ini memory_limit

After upping the threshold to 800MB I still get occasional notifications from lfd. I also checked the php.ini settings and memory_limit is set to 256MB - in theory the script wouldn't run if it was using more than this. From this I am guessing that either:

a) It is not PHP that is using the memory (could it be MySQL or CURL - is the memory used by these included in the value frommemory_get_peak_usage(true)?)

b) I'm not getting an accurate figure from lfd

Second Update - memory used by MySQL is not included in memory_get_peak_usage(true)

I suspect this is the issue - however I'm not sure what exactly CSF includes in resource figure. I will look into making the MySQL requests more efficient and see how that effects things.