proxy_list | Gather list of proxies from various sources | Proxy library

use key argument of sorted function. convert each element to a tuple of priority. here (len(s), s), this means len(s) have more priority to s.

Use itertools.cycle to create an iterator that repeats your proxies indefinitely

1. don't do asynchronous code in a .forEach, that never works like you expect - use a regular for loop
2. make tetProxies async
3. use await proxytestLogic
4. get rid of that new Promise you never resolve anyway

so - you end up with:

By creating the pool as a class attribute, it gets executed when NameToLinkedInScraper is defined during import (the "main" file is imported by children so they have access to all the same classes and functions). If it were allowed to do this it would recursively keep creating more children who would then import the same file and create more children themselves. This is why spawning child processes is disabled on __main__ import. You should instead only call Pool in __init__ so new child processes are only created if you create an instance of your class. In general using class attributes rather than instance attributes should be avoided unless it is static data, or needs to be shared between all instances of the class.

When doing multiprocessing, each process runs in its own memory space. You would need to refactor your code so that internal_linkedin_job returns the dataframe.

You could do:

(Copying my comment to answer)

In the scrape_page, the results are send back using yield. This converts the function to an iterator. To process the results, you need to iterate them.