Question

I have a parrallelized C++ program that does brute force optimization. For some reason I get diminishing returns per core up to ~6 cores, at which point I hit a wall where further cores add ~no speed. This is consistent when run on an 8 or 16 core machine.

When I run strace -f ./progname I get a whole ton of the following that occur specifically during the multithreaded section of the program: [pid 2646] mprotect(0x7ffe7c030000, 4096, PROT_READ|PROT_WRITE) = 0

and a few of these that occur one after the other: [pid 2645] mprotect(0x7ffe78030000, 4096, PROT_READ|PROT_WRITE <unfinished ...> — [pid 2646] <... mprotect resumed> ) = 0

Their not always from the same pid though.

When I decrease the number of cores I get fewer of the above messages, and at 2 or 3 cores I don’t get any.

The only thing I can guess is that maybe it has to do with the massive quantity of vector allocations and accessing that gets done in each thread. I’m not using any other memory management libraries if that’s relevant.

Answer 1

I have a parrallelized C++ program that does brute force optimization.
For some reason I get diminishing returns per core up to ~6 cores, at
which point I hit a wall where further cores add ~no speed. This is
consistent when run on an 8 or 16 core machine.

Can you describe the algorithm? Lots of computation algorithms are memory bound.

Try profiling your application with oprofile, perf, or if those aren’t possible, gprof.

The only thing I can guess is that maybe it has to do with the massive
quantity of vector allocations and accessing that gets done in each
thread. I’m not using any other memory management libraries if that’s
relevant.

A super easy way to relieve TLB pressure is to use huge pages (presumably your hardware supports it). On linux you can use libhugetlbfs and its morecore hook.

HUGETLB_MORECORE=yes LD_PRELOAD=libhugetlbfs.so  ./brute_force_optimization