ngram | Fast n-Gram Tokenization | Natural Language Processing library

There's quite a bit I don't understand in your code. So instead I'll just give you code that works ;-)

• I'm baffled by how your code can run at all. A .gz file is compressed binary data (gzip compression). You should need to open it with Python's gzip.open(). As is, I expect it to die with an encoding exception, as it does when I try it.

• temp[2] is not an integer. It's a string. You're not adding integers here, you're catenating strings with +. int() needs to be applied first.

• I don't believe I've ever seen asyncio mixed with concurrent.futures before. There's no need for it. asyncio is aimed at fine-grained pseudo-concurrency in a single thread; concurrent.futures is aimed at coarse-grained genuine concurrency across processes. You want the latter here. The code is easier, simpler, and faster without asyncio.

• While concurrent.futures is fine, I'm old enough that I invested a whole lot into learning the older multiprocessing first, and so I'm using that here.

• These ngram files are big enough that I'm "chunking" the reads regardless of whether running the serial or parallel version.

• collections.Counter is much better suited to your task than a plain dict.

• While I'm on a faster machine than you, some of the changes alluded to above have a lot do with my faster times.

• I do get a speedup using 3 worker processes, but, really, all 3 were hardly ever being utilized. There's very little computation being done per line of input, and I expect that it's more memory-bound than CPU-bound. All the processes are fighting for cache space too, and cache misses are expensive. An "ideal" candidate for coarse-grained parallelism does a whole lot of computation per byte that needs to be transferred between processes, and not need much inter-process communication at all. Neither are true of this problem.