murex | smarter shell and scripting environment | Command Line Interface library

I think the problem is obvious - you're running out of memory because you're trying to load so much data into memory at once, and then process it.

You need to either:

• get a machine with more memory.
• re-architect the solution to use a pipelined approach using either a generator or coroutine pipeline to do the processing stepwise over your data.

The problem with the first approach is it won't scale indefinitely and is expensive. The second way is the right way to do it, but needs more coding.

As a good reference on generator/coroutine type pipeline approaches check out any of the pycon talks by David Beazley.

Take the trouble to read the spec. The third argument of substring() is the length of the substring required, not the end position.

A common mistake: if you use lots of languages, you'll know that they all have a substring() function and it's always different from the last language you used.

To me, merge looks like a better/simpler choice.

Try it this way: