batches: webhook delivery tracking backend
Created by: LawnGnome
This PR implements logging for incoming webhooks via a new middleware that stores the incoming requests and the Sourcegraph responses in a database table, and exposes two simple GraphQL queries (and a few associated types) to retrieve them. That table may be encrypted via a new encryption key (since payloads may include secrets), and is purged via a background job in cmd/frontend
(which felt more appropriate than repo-updater
, since this is all frontend-related), which is controlled by a new site configuration option.
To be conservative, sites with any encryption enabled will default to webhook logging being off. Sites without any encryption will default to retaining three days of webhook deliveries, with the idea being that that's about the minimum usable period once you factor in weekends, and we don't want to store any more than we have to. An obvious future enhancement would be to allow for different retention periods for webhooks that resulted in errors versus those that were processed successfully.
The PR looks fairly massive, but it's not as bad as it looks. (I think. If people want the database handling and GraphQL API work separated, I can do that.) A lot of this is tests. About a quarter of the diff is just the generated mock from the new mock methodology that's being phased in for internal/database
. All of the concepts are in the first paragraph of this PR description.
Obviously this doesn't include any frontend work as yet. I'm getting to that.
Part 1 (of probably 2) of #26669 (closed).