Simple-Web-Crawler | 基于C # .NET的简单网页爬虫，支持异步并发、切换代理、操作Cookie、Gzip加速。

There's no way of stopping a forEach. You can simulate a stop by checking a flag inside the forEach, but that will still loop through all the elements. By the way, using a loop for an io operation is not optimal.

As you have stated, the best way to process a set of increasing data to process is to do it one-by-one, but I'll add a twist: Threaded-one-by-one.

NOTE: With thread I don't mean actual threads. Take it more of a definition of "multiple lines of work". As IO operations don't lock the main thread, while one or more requests are waiting for the data, other "line of work" can run the JavaScript to process the data received, as JavaScript is single threaded (Not talking about WebWorkers).

Is as easy as having an array of pages, which receives pages to be crawled on the fly, and one function that reads one page of that array, process the result and then returns to the starting point (loading the next page of the array and processing the result).

Now you just call that function the amount of threads that you want to run, and done. Pseudo-code: