[WIP] Codeinsights MVP backend
Created by: slimsag
See #17218 for the general plan / work being done on code insights.
I am working from this codeinsights
branch (which will be unstable/buggy/broken sometimes) and my aim is to have this branch by EOW run a very basic code insights backend in dev environments, including all the app infra/changes we need, running timescaledb, doing the DB migrations to create the schema, and the GraphQL schema/API/backend making requests and serving data from timescale.
This branch itself won't be merged, rather I will be taking changes from here once they are solid and sending individual PRs for review (so they meet our quality bar, are not "one big PR", etc.) I will be posting updates here as I send out those PRs.
TODO:
-
Grafana dashboards -- issue filed elsewhere -
testing: An end-to-end test where we use the GraphQL API to modify user settings/add insights, and the GraphQL API to query data after a while? -- issue filed elsewhere -
Investigate adding more tests for enterprise/internal/insights/background/* - see https://github.com/sourcegraph/sourcegraph/pull/18267 -- issue filed elsewhere -
Basic webhook fetching support -- decided to adopt push model, see other issue -
Deploy TimescaleDB in non-dev envs -
Generate schema.insights.md documentation -
dev/db/squash_migrations.sh
starts Postgres 9.6 if the connected DB version != 9.6, so it doesn't work with these migrations. -
internal/db/schemadoc/main.go
starts Postgres 9.6 if the connected DB version != 9.6, so it doesn't work with this DB schema yet. -
Add code TODO for retroactively updating repo names in DB
Feb 21-26:
-
insights: correct over-reporting / aggregation of data points #18632
Feb 15-19:
-
Basic search query execution & storage -
Investigate using go-mockgen for GraphQL resolver tests: https://github.com/sourcegraph/sourcegraph/pull/18075#discussion_r572359571
Feb 8-9:
-
Implement DB store tests -
Merge DB schema -
Implement GraphQL backend tests
Feb 1-5:
-
Implement DB store -
Implement GraphQL backend -
Investigate replacing repo_names
andmetadata
with https://docs.timescale.com/latest/using-timescaledb/compression (Tomas suggested, looks really cool) -- CONCLUSION: https://github.com/sourcegraph/sourcegraph/pull/17227#issuecomment-773704129 -
Start TImescaleDB in testing env; run migrations #17733 #17586
Jan 18-22:
-
Determine where to store insights (user/org/global settings? likely not DB for now, but file an issue for later?) -
Run migrations on server startup -
Plan DB schema & queries/inserts -
Generate separate schema.md files for codeintel/frontend DBs so we can generate schema.insights.md later -
Tag TimescaleDB Docker image as sourcegraph/codeinsights-db
-
Run codeinsights-db
as part of dev server. -
Add migrations/codeinsights
DB migration foundation -
Add GraphQL backend scaffolding / stubs