Try to fix flaky syncer test
Created by: eseliger
Under load of all test suites run in parallel, this could have taken more than 50ms apparently, since it was flaky in a CI build. This should hopefully fix it, and in the happy path, never slow down the test.