Turns out my Threadripper stomps these tests so the arbitrary wait time
actually slows down my unit test time.
Don't want to make it shorter since `strace -f` becomes ugly at that
point.
Instead set up a semaphore style waiting structure that waits until
there are no longer any participants in the atomic