starmap | Starmap is a program for viewing the positions of the stars

Looking at the three instances of apiString and rewriting them to use the more succinct F-strings, they all appear to be of the form:

I do not think Pool or the way you do this is the best approach. If you have a Pool, it will have N workers that can run in parallel. By default N is the number of your cores. In your case the first thing a worker does is that it goes to sleep. It blocks the worker in the pool but it is not doing anything. You can try increasing the number of pool workers but you will most likely hit an OS limit of something if you try to launch 1500 workers.

My strong advice is to redesign your app so that you will do the sleeping and waiting somewhere else and only dispatch tasks to a Pool worker when the nap time is over. Not sure if it is suitable in your case, but you could combine threads and Pool. Make the napping happen in a thread and only dispatch a worker when there is work to be done, and remove all sleeps from your pool worker.

This is just a dummy snippet to demonstrate the idea but it runs.

You are entering the max_iteration value as an array and then comparing an int count with it.You might want to do something like this, I'm not sure which dimension you want, but since both are same so:

Python has the multiprocessing module which can run multiple tasks in parallel and inside each process you can have multiple threads or async io code

Here is a working example which uses 3 Processes and Multithreading

Let's look at your original approach where power was undefined. First, a couple of comments. You are calling worker function multiply_by, but I do not see where that is defined either. I do see, however, function raise_to_power. The error you should have gotten is that multiply_by is not defined. So this is a bit puzzling. Second, I see that you are computing a chunksize rather than using the default function for doing this which would compute a value roughly 1/4 the size you compute. Larger chunksizes mean fewer memory transfers (good) but could result in processors ending up being idle if they do not process their tasks at the same rate (bad), which granted is not that likely in your case. I can see having your own function for calculating the chunksize since the function used by the map method must convert its iterable argument to a list if necessary in order to get its length and if the iterable is very large, this could be very memory-inefficient. But you have already converted a range to a list so you haven't taken advantage of the opportunity to save memory by doing your own calculation.

As Mark Satchell indicated, the simple solution for your case would be just to make power a global. But let's consider the general case. If your platform is Windows or any platform that uses spawn to create new processes (I am guessing this well might be the case based on your use of if __name__ == '__main__': governing the code that creates new processes), then any code that is at the global scope will be executed for every new process created. This is not an issue for a statement like power = 10. But if power required far more complicated code to initialize its value, it would be inefficient to re-execute this code over and over again for each process in the pool of processes. Or consider the case where power was a very large array. It would be perhaps too costly to create instances of this array in each sub-process's memory space. Then what is required is a single instance of the array in shared memory.

There is a mechanism for initializing global variables for each sub-process in a pool by using the initializer and initargs arguments when creating the Pool instance. I have also made an additional change to save memory by taking advantage of the fact that you are using your own chunksize calculation:

First optimization is to avoid concatenate your dataframe at each iteration.
You can try something like that:

It is possible for another delayed function to unpack the tuple, in the example below, the delayed value of return_tuple(1) was not computed, but passed as a delayed object:

600 000 ( number of cases or columns)

Judging by the names of the columns you provide in the sample, your workflow might benefit from a better re-organization of the data, which might also simplify the calculations.

How can I use append at each step or assign to dask array without loading memory or is there a better approach ?

You are probably interested in delayed API, but the question/problem is not sufficiently clear to provide further advice.

This question is maybe better suited for code review see this post

Issues that you could encounter:

• you have to deal with a lot of files.
• your process might be limited by the amount of open files you can have at one time.
• you might be limited by the speed you can open and read the files.
• you might be limited by the speed that you can process those files.
• maybe you can't keep the lists that you are constructing in memory
• the process can last a long time and might fail after having already done a lot of work.

Ways you might mitigate these issues:

• you should close file handlers when your done processing a file
• you might use multithreading or multiprocessing or another way to ask the operating system to open and read multiple files at once if you are limited by the speed you can open or read files in parallel
• you might use multiprocessing to parallelize processing of read files. (don't use multithreading for this because of the GIL)
• you might want to write a partial result to disk so you can limit your memory footprint
• you might want to write a partial result to disk so you pause and resume your process. for example if you encounter an error.
• you might want to store the end result to disk so you only have to parse all these files once and can later run more queries on the dataframe

One format to store a partial result that I recommend is CSV. It is easy to read and append and panda's already has support to read and write from CSV.