Search backend: simplify search job traces

Created by: camdencheek

This reduces the number of logs added to search job traces. I'd previously implemented a wrapper that logs every time a search event came through, but that ended up being pretty verbose and not super useful. Additionally, I started seeing logs about traces being too large, and I have a feeling that may have been contributing.

Instead, each job logs an event to its trace on the first streamed result, and adds a tag with the total number of results sent when the trace finishes. This removes the "only do this if tracing is enabled" logic since it reduces the overhead to an atomic add for each event.

FYI @tbliu98, this change is in the same domain as #34009 (closed), which I believe is slated to be one of your first issues to look at. No need to take a close look at this until then, but I figured it'd be valuable to provide a link trail.

Test plan

See screenshot for an example of how this looks: