Testing is all about getting confidence that your code is correct. As you say, this technique can find bugs, and so when it then ceases to find bugs in a given bit of code even as you increase the load it gives you confidence that your code is working.

However, I’ve been doing lots of multi-threaded work in C++ recently, and I’m painfully aware that multi-threaded synchronization is one aspect that is particularly hard to test: you need to carefully think through the synchronization requirements before coding.

This style of testing should be used sparingly and only to assert behavior that can’t be tested some other, more deterministic way. I do think it can be very effective, and while there is always a chance the “bad behavior” doesn’t surface for small sets, that chance rapidly nears zero once you increase the sample and thread sizes. For example, for the project where this technique was first employed, we consistently saw issues with sample sets as small as 10 with 2 threads. When we were close a solution, we cranked the sample set to 1k and 5k with 50-100 threads to see if it would break under extreme load. With that data size and load, we were confident that we’d found the right solution.

There may be scenarios where multi-cpu configurations expose some unexpected behavior, but in the example and sample code, there are no such scenarios. There is no shared state between invocations and you can run as many concurrent threads as the VM can handle, on any number of CPUs without issue. It’s a rather simple solution to fairly common real world problem.

Also, there is the issue of visibility. On single CPU systems, or dual-core systems that share a cache between cores, threads may see changes to data made by other threads even when suitable synchronization is not being used. If the application is then run on a true multi-processor system, such latent bugs may be exposed.

So: this is a worthwhile tool to have, but it’s not a silver-bullet.