[spike] insights: start collecting backfilling benchmark pings
- Truncate descriptions
Created by: Joelkw
In order to understand and prioritize improvements to our backfill completion times, we should start collecting actual data.
We can collect p75, p90, p95 points.
I wonder if this RFC/design should be led by engineering rather than product/me? You might have the best sense of what performance captures we want to make. Our objective / the goal is primarily to understand how long it takes code insights on an instance to go from "created" to "no longer loading data" (done), so we can prioritize improvements or catch/validate odd behavior on certain instances if customers raise issues.
We can model this somewhat off how we collect these stats for search queries.
We already collect the # of repos on a customer instance, so we can cross-reference with that per customer, but other questions/concepts to keep in mind:
- If we eventually allow you to backfill over "some" repos, and not all repos, is it useful to collect how many repos that was? Probably (so our ping design should allow for this future expansion).
Pings RFC for bizops approval coming this iteration as well.