aiodns | Simple DNS resolver for asyncio | DNS library

Any ideas why, how and what is this ThreadPoolExecutor?

ThreadPoolExecutor is the thread pool implementation provided by the concurrent.futures module. It is used for asynchronous execution of synchronous code by handing it to a separate thread. The pool's purpose is to avoid the latency of creating and joining a thread for each separate task; instead, a pool creates the worker thread only once, and keeps it in the pool for later usage. The maximum number of threads in the pool can be configured and defaults to the number of cores multiplied by 5.

The threads you see in your code belongs to a ThreadPoolExecutor instantiated by one of the libraries you are using. Specifically, asyncio creates an executor for use by the run_in_executor method. This executor is used by asyncio itself to provide async interface to calls that natively do not have one, such as OS-provided DNS resolution.

In general, when using non-trivial third-party libraries, you cannot assume that your code will be the only one to create threads. When iterating over live threads, you simply ignore those that you didn't create, which can be accomplished for example by marking the threads you create with a custom attribute on the Thread object.

After some further investigation, this issue does not appear to be directly caused by aiohttp/asyncio but rather limitations/limits stemming from both:

• The capacity/rate-limiting of your DNS Servers
• The max number of open files at the system level.

Firstly, for those looking to get some beefed-up DNS servers (I will probably not go that route), the big-name options seem to be:

• 1.1.1.1 (Cloudflare)
• 8.8.8.8 (Google Public DNS)
• Amazon Route 53

(Good intro to DNS for those like me for whom network concepts are lacking.)

The first thing that I did was to run the above on a beefed-up AWS EC2 instance - h1.16xlarge running Ubuntu which is IO optimized. I can't say this in itself helped, but it certainly cannot hurt. I'm not too familiar with the default DNS server used by an EC2 instance, but the OSError with errno == 8 from above went away when replicating the above script.

However, that presented a new exception in its place, OSError with code 24, "Too many open files." My hotfix solution (not arguing this is the most sustainable or safest) was to increase the max file limits. I did this via:

But if results have to go back to one file, is it even possible to do lookups asynchronously?

If you don't care about the order of the results, it's straightforward to implement asynchronous lookups. For example, you can use asyncio.as_completed to schedule all coroutines to run in parallel and get notified as each completes:

I know this sounds ridiculous, given that it works sometimes, but can you switch 'localhost' to '127.0.0.1' and see if that fixes? I'm seeing that certain python modules may be confused by localhost... even though it works intermittently.

They seems to not have updated for last 1 year and their requirements.txt is below

Question: ...but others like a timeout I want to re-queue the item.

Requeue the Task could lead to a Deadlock.
Instead of requeue the task, hold the Task, for instance: