parseline | gem parseline is to help the developers to load external CSV | CSV Processing library

This code reads your file and stores the data in the std::vector Info using stringstreams and other things to parse.

You just forgot to declare your function inline, do that and it will work just fine:

A common error:

That was a general failure of memory management in my code.

mapValues does not alter partitioning already in effect, it's a narrow transformation. You have two of them.

reduceByKey is associative. Spark aggregates locally and sends those results to driver or to the relevant partitions - in your case. If you do not use the parameter on reduceByKey for number of partitions, you retain the same number of partitions for the new RDD, albeit with different distribution.

What you observe is simply how memory allocation works.

glibc makes use of two different kinds of allocation, described in details here. In a nutshell, small allocations are served as pieces cut out from larger memory allocations called arenas. When you free such a piece, it is returned back to its arena and the RSS of the process doesn't decrease. Larger allocations are taken directly from the OS using anonymous memory mappings. Freeing large allocations unmaps the memory region, which restores the value of RSS to its earlier value, provided that no new pages are mapped into other memory regions.

The reason for having two different algorithms is that asking the kernel to perform memory mapping is a slow operation as it involves modification to the process page tables. Cutting out pieces of a larger block happens entirely in user space, but it is suboptimal for large allocations.

When you push_back() in a loop to an std::vector without resizing it first, the vector will continuously reallocate on demand doubling its storage size. Initially, the allocations will be small enough and will come from the arena until it reaches a point where it starts memory mapping from the OS. When you do points.resize(2e7), you force the vector to allocate all the storage at once, which is directly served by an anonymous memory mapping operation.

glibc uses per-thread arenas and creates a new arena for each thread. The arena of the main thread can shrink back if you free blocks at its very end. The arenas of the other threads do not shrink that readily. This is due to the different mechanisms used to allocate memory for the arenas. The main one is (de-)allocated by moving the program break, also known as the end of the data segment. The other arenas employ anonymous memory mappings.

When OpenMP spawns a new thread, it starts with a small arena. When the vector starts asking for increasingly larger memory blocks, the arena grows in order to accommodate the demand. At one point, the vector grows so large it starts receiving anonymous memory mappings instead. When those mappings aren't too big, instead of unmapping them upon destruction of the vector they get added to the arena as cache for future memory allocations (unmapping is also quite expensive). Not only this, but also the threshold for the switch to anonymous memory mappings gets updated upwards. This is why each thread ends up having 32 MiB of cached memory in each arena, which multiplied by 7 threads gives 224 MiB. Add a couple of MiB from the main arena and you get 227 MiB.

If having 32 MiB of memory cached in each thread makes you uneasy, simply tell glibc to not dynamically increase the memory mapping threshold by setting it to a fixed value via the MALLOC_MMAP_THRESHOLD_ environment variable:

The std::vector holds the element by value so just using std::vector::push_back will copy your local object into the vector.

To mitigate this you could either implement move semantics and move the the Customer object like this:

The problem is that ...([^"]+)""" means that the input should end with the capture group, even before any closing quote mark.

To fix it you can A) add .* at the end (i.e. ...([^"]+).*""") or B) make the regex .unanchored, in which case you can probably drop the \s* at the beginning.