Code insights critical telemetry not included in ping payload

Created by: dadlerj

This is in response to @Joelkw's question at https://sourcegraph.slack.com/archives/CN4FC7XT4/p1647501550824509

We added critical telemetry to count total # of insights (because that’s the free/paid tier metric) in 3.37, and this work shipped in 3.37 (screenshot attached for branch cut; PR here). We made the following schema update.

But if I look in Bigquery, though I see this field, I see entirely null values, even for the k8s instance (second screenshot; verify k8s remote site id here). I don’t think this is a problem with our pings because I see the ping populated correctly in the “pings object” on the site admin page for k8s (also attached a screenshot if you don’t have admin access). Is this a problem with our schema or something else in our pipeline such that we’re not recording this number correctly, or is this a problem we need to try to fix before the release cut? (edited)

Summary of issue: It looks like the code insights critical telemetry object is correctly being generated on the client side (i.e., on self-hosted instances), correctly being deserialized on the handler side (i.e., on sourcegraph.com), but it's NOT then being marshaled and sent to our data warehouse.

The pingRequest object in the updatecheck/handler.go file includes the CodeInsightsCriticalTelemetry field.

However, the pingPayload object in the same file does NOT include that field. It should be added to that struct linked there, and added to the marshaling that happens at https://sourcegraph.com/github.com/sourcegraph/sourcegraph@366d5e428642a9e6dd2722da6d4eb3abf22c2529/-/blob/cmd/frontend/internal/app/updatecheck/handler.go?L393.

I can't recall if anything else is required here, but I suspect that this is the only issue. This is a tough issue to catch, since there's no actual error happening here—the field is just left empty/null since it's never added to the payload, and it's not a required field in our data warehouse (like all fields, since we have customers on different versions). Thanks for identifying the problem @Joelkw !

CC @kelsey-brown @andersonlauren