workerutil: Use correct contexts for handlers
Created by: efritz
This PR fixes the Stop()
method to be properly graceful. We achieve this by using a different context for dequeue operations and handler operations. Stop now cancels the dequeue context but does not affect the root context that is passed into the handlers, so they can remain live while the worker waits for them to finish.
Relevant behavior changes: Currently, a signal will call Stop
on the worker util, which cancels all contexts in the dequeue loop and the handler. After this PR, Stop
will no longer cancel already dequeued jobs, making this method graceful.
Product teams have likely assumed this was the case, but this may be a behavioral change for DevOps. See a related increase of graceful shutdown period here.
Test plan
Added an additional unit test. Also tested with signals on local stack to test desired behavior.