Find-String | PowerShell script to provide functionality | Command Line Interface library

It seems that you missed a ) on the fourth line:

Regarding the first question (configuring the pipeline), I can't really offer any guidance. Optimizing the parameters of a dataflow pipeline seems like a black art to me!

Regarding the second question (how to batch a work load consisting of work items having unknown size at compile time), you could use the custom BatchBlock below. It uses the DataflowBlock.Encapsulate method in order to combine two dataflow blocks to one. The first block in an ActionBlock that receives the input and puts it into a buffer, and the second is a BufferBlock that holds the batched items and propagates them downstream. The weightSelector is a lambda that returns the weight of each received item. When the accumulated weight surpasses the batchWeight threshold, a batch is emitted.

This architecture is not optimal, because each of the worker blocks, the produceCSFiles and the produceMatchingLines, is doing mixed I/O-bound and CPU-bound work. Ideally you would like to have a block dedicated at doing exclusively I/O-bound, and another one doing exclusively CPU-bound work. This way you would be able to configure optimally the degree of parallelism of each block, according to the capabilities of the associated hardware component. With your current configuration it is entirely possible that at a given moment both blocks are doing I/O work, competing with each other for the SSD's attention, while the CPU is idle. And at another moment the exact opposite could be happening. The result is a chaotic and uncoordinated hubbub. Which is similar with what you would get if you used a monolithic Parallel.ForEach loop, which would probably yield comparable (mediocre) performance improvements over a single-thread approach.

Something else that you should have in mind is that the TPL Dataflow performs well when the messages passed from block to block are chunky. As the introductory document says: "provides in-process message passing for coarse-grained dataflow and pipelining tasks" (emphasis added). If the processing of each individual message is too lightweight, you'll end up with significant overhead. If you need to, you can chunkify your workload by batching the messages, using BatchBlocks, the Chunk LINQ operator, or other means.

Having said all that, my assumption is that your work is disproportionately I/O bound, rendering less relevant the capabilities of your CPU. Honestly I wouldn't expect massive performance improvements, even with the most sophisticated implementation.

You were almost there with your attempt. The UnboundLocalError happens because the variable found_coordinate isn't defined in your function if the line is a resistance line. You should define that as a global variable too, because you need to keep track of it over multiple function calls. I'm intrigued that the first set of coordinate/resistance worked. So do

Consider this option using re.findall:

You can do:

I won't give you the full answer but here's what you should be doing.