code insights: backend MVP implementation
Created by: slimsag
This is a tracking issue for implementing the MVP backend for Code Insights.
General strategy is documented in RFC 273: Code Insights backend.
TL;DR: We have a good amount of information that customers are interested in Code Insights. We have an experimental frontend demo we have been showing to prospective customers (viewable on k8s.sgdev.org) and multiple customers are interested in this. However, we are entirely blocked on having a scalable backend before customers can actually use it and we can get feedback on it to determine the product direction in general (there are still many open questions.) The team has not yet been able to hire a backend engineer, and Felix and Joel have chatted with me extensively about having me step in to contribute a solid foundation which customers can use at scale so we can determine what to do with the product next, and be able to make our first backend engineer hire for the team and make them be productive / have clear direction. The goal here is for @slimsag to work with the code insights team closely to add an MVP backend that can be used at scale and as a good foundation, and help with making the team generally successful in the future (once @slimsag steps back and goes onto other projects.)
Tracked issues
@slimsag
-
internal contributor 3.25 Tracking issue (#17737) -
( 🏁 19 days ago) add codeinsights-db (TimescaleDB) deployment (#1851) :shipit: -
( 🏁 19 days ago) insights: set POSTGRES_PASSWORD (#1856) :shipit: -
( 🏁 18 days ago) codeinsights-db: remove liveness/readiness probes (#1859) :shipit: -
( 🏁 18 days ago) frontend: configure codeinsights DB env var (#1864) :shipit: -
( 🏁 19 days ago) sourcegraph/deploy-sourcegraph-dot-com (#4599) :shipit: -
( 🏁 19 days ago) sourcegraph/deploy-sourcegraph-dot-com (#4600) :shipit: -
( 🏁 19 days ago) sourcegraph/deploy-sourcegraph-dot-com (#4601) :shipit: -
( 🏁 40 days ago) code insights: add Postgres migrations foundation (#17219; PRs:#17432,#17586,#17588) -
( 🏁 27 days ago) code insights: add experimental GraphQL API (#17221; PRs:#17497,#18075) -
( 🏁 today) code insights: document that it is not supported in sourcegraph/server demo deployments (#17222; PRs:#17735) -
( 🏁 18 days ago) code insights: deploy TimescaleDB in Kubernetes deployments (#17224) -
( 🏁 16 days ago) code insights: deploy TimescaleDB in docker-compose deployments (#17225; PRs:#259) -
( 🏁 39 days ago) code insights: deploy TimescaleDB in dev and CI environments (#17226; PRs:#17431,#17733) -
( 🏁 54 days ago) internal/db: generate separate schema.md files for codeintel and frontend DBs (#17228) :shipit: -
( 🏁 47 days ago) docker-images: tag TimescaleDB as sourcegraph/codeinsights-db (#17427) :shipit: -
( 🏁 40 days ago) internal/db: do not run TimescaleDB migrations against singleton database (#17585) :shipit: -
( 🏁 40 days ago) Revert "insights: execute database migrations (#17586)" (#17701) :shipit: -
( 🏁 39 days ago) dev/ci: do not run e2e sourcegraph/server tests with DEPLOY_TYPE=dev (#17731) :shipit: -
( 🏁 39 days ago) Revert "Revert "insights: execute database migrations (#17586)" (#17701)" (#17732) :shipit: -
( 🏁 39 days ago) code insights: add dev option to disable code insights backend entirely (#17738) :shipit: -
( 🏁 39 days ago) code insights: fully qualify Postgres DSN to avoid conflicts (#17742) :shipit: -
( 🏁 39 days ago) dev: give codeinsights-db more time to start (#17744) :shipit: -
( 🏁 34 days ago) code insights: add new GraphQL schema (#17842) :shipit: -
( 🏁 37 days ago) code insights: improve backend testing experience (#17843) :shipit: -
( 🏁 34 days ago) code insights: add new user/org/global settings schema (#17844) :shipit: -
( 🏁 37 days ago) insights: skip integration tests if DB unavailable in dev environments (#17861) :shipit: -
( 🏁 30 days ago) insights: split out DB testing setup code into separate package (#18001) :shipit: -
( 🏁 28 days ago) insights: add very basic GraphQL resolver integration test (#18035) :shipit: -
( 🏁 4 days ago) dev: make drop-entire-db script also drop insights DB (#18081) :shipit: -
( 🏁 26 days ago) insights: add initial DB schema (#18083) :shipit: -
( 🏁 26 days ago) insights: add & prepare to use generated store mocks (#18118) :shipit: -
( 🏁 26 days ago) insights: store: add support for querying series data points (#18129) :shipit: -
( 🏁 26 days ago) insights: resolvers: implement point querying & add integration tests (#18130) :shipit: -
( 🏁 23 days ago) insights: change series_id to string (hash); add index (#18230) :shipit: -
( 🏁 20 days ago) insights: store: query metadata & other minor improvements (#18254) :shipit: -
( 🏁 20 days ago) insights: store: add support for recording data points (#18255) :shipit: -
( 🏁 20 days ago) insights: add new discovery package for locating insights (and use it) (#18264) :shipit: -
( 🏁 20 days ago) migrations: add insights_query_runner_jobs table (#18265) :shipit: -
( 🏁 19 days ago) insights: actually enqueue/record insights from global settings (#18314) :shipit: -
( 🏁 19 days ago) dev: allow codeinsights-db more time to startup (#18359) :shipit: -
( 🏁 19 days ago) insights: turn on for Sourcegraph.com (#18360) :shipit: -
( 🏁 18 days ago) insights: fix job dequeueing (columns mismatch) (#18390) :shipit: -
( 🏁 3 days ago) insights: backfill data (#18398) -
( 🏁 18 days ago) insights: query correct series data (#18401) :shipit: -
( 🏁 17 days ago) insights: turn on in dev environments by default (#18458) :shipit: -
( 🏁 17 days ago) insights: correctly query data even if metadata is not present (#18459) :shipit: -
( 🏁 17 days ago) insights: store: always store data in UTC format (#18460) :shipit: -
( 🏁 17 days ago) insights: store: add fast data existence check (#18461) :shipit: -
( 🏁 16 days ago) internal/vcs/git: add ability to get first ever commit in a repository (#18463) :shipit: -
( 🏁 16 days ago) insights: disable TimescaleDB telemetry (#18464) :shipit: -
( 🏁 today) insights: 3.25 release checklist (#18465) -
( 🏁 16 days ago) insights: query data aggregated over 12h (#18506) :shipit: -
( 🏁 12 days ago) insights: backend: possible double-inserts (over-reporting) if container restarts happen frequently (#18510) -
( 🏁 16 days ago) insights: enable in all production deployments (#18515) :shipit: -
( 🏁 16 days ago) insights: ensure search queries run with a large count: (#18518) :shipit: -
( 🏁 16 days ago) insights: default to 90d display; improve handling of forked/archived repos (#18519) :shipit:
-
Completed
-
( 🏁 54 days ago) internal/db: generate separate schema.md files for codeintel and frontend DBs (#17228) :shipit: -
( 🏁 47 days ago) docker-images: tag TimescaleDB as sourcegraph/codeinsights-db (#17427) :shipit: -
( 🏁 40 days ago) internal/db: do not run TimescaleDB migrations against singleton database (#17585) :shipit: -
( 🏁 40 days ago) code insights: add Postgres migrations foundation (#17219; PRs:#17432,#17586,#17588) -
( 🏁 40 days ago) Revert "insights: execute database migrations (#17586)" (#17701) :shipit: -
( 🏁 39 days ago) dev/ci: do not run e2e sourcegraph/server tests with DEPLOY_TYPE=dev (#17731) :shipit: -
( 🏁 39 days ago) Revert "Revert "insights: execute database migrations (#17586)" (#17701)" (#17732) :shipit: -
( 🏁 39 days ago) code insights: deploy TimescaleDB in dev and CI environments (#17226; PRs:#17431,#17733) -
( 🏁 39 days ago) code insights: add dev option to disable code insights backend entirely (#17738) :shipit: -
( 🏁 39 days ago) code insights: fully qualify Postgres DSN to avoid conflicts (#17742) :shipit: -
( 🏁 39 days ago) dev: give codeinsights-db more time to start (#17744) :shipit: -
( 🏁 37 days ago) code insights: improve backend testing experience (#17843) :shipit: -
( 🏁 37 days ago) insights: skip integration tests if DB unavailable in dev environments (#17861) :shipit: -
( 🏁 34 days ago) code insights: add new GraphQL schema (#17842) :shipit: -
( 🏁 34 days ago) code insights: add new user/org/global settings schema (#17844) :shipit: -
( 🏁 30 days ago) insights: split out DB testing setup code into separate package (#18001) :shipit: -
( 🏁 28 days ago) insights: add very basic GraphQL resolver integration test (#18035) :shipit: -
( 🏁 27 days ago) code insights: add experimental GraphQL API (#17221; PRs:#17497,#18075) -
( 🏁 26 days ago) insights: add initial DB schema (#18083) :shipit: -
( 🏁 26 days ago) insights: add & prepare to use generated store mocks (#18118) :shipit: -
( 🏁 26 days ago) insights: store: add support for querying series data points (#18129) :shipit: -
( 🏁 26 days ago) insights: resolvers: implement point querying & add integration tests (#18130) :shipit: -
( 🏁 23 days ago) insights: change series_id to string (hash); add index (#18230) :shipit: -
( 🏁 20 days ago) insights: store: query metadata & other minor improvements (#18254) :shipit: -
( 🏁 20 days ago) insights: store: add support for recording data points (#18255) :shipit: -
( 🏁 20 days ago) insights: add new discovery package for locating insights (and use it) (#18264) :shipit: -
( 🏁 20 days ago) migrations: add insights_query_runner_jobs table (#18265) :shipit: -
( 🏁 19 days ago) add codeinsights-db (TimescaleDB) deployment (#1851) :shipit: -
( 🏁 19 days ago) sourcegraph/deploy-sourcegraph-dot-com (#4599) :shipit: -
( 🏁 19 days ago) insights: set POSTGRES_PASSWORD (#1856) :shipit: -
( 🏁 19 days ago) sourcegraph/deploy-sourcegraph-dot-com (#4600) :shipit: -
( 🏁 19 days ago) insights: actually enqueue/record insights from global settings (#18314) :shipit: -
( 🏁 19 days ago) dev: allow codeinsights-db more time to startup (#18359) :shipit: -
( 🏁 19 days ago) sourcegraph/deploy-sourcegraph-dot-com (#4601) :shipit: -
( 🏁 19 days ago) insights: turn on for Sourcegraph.com (#18360) :shipit: -
( 🏁 18 days ago) frontend: configure codeinsights DB env var (#1864) :shipit: -
( 🏁 18 days ago) insights: fix job dequeueing (columns mismatch) (#18390) :shipit: -
( 🏁 18 days ago) codeinsights-db: remove liveness/readiness probes (#1859) :shipit: -
( 🏁 18 days ago) code insights: deploy TimescaleDB in Kubernetes deployments (#17224) -
( 🏁 18 days ago) insights: query correct series data (#18401) :shipit: -
( 🏁 17 days ago) insights: turn on in dev environments by default (#18458) :shipit: -
( 🏁 17 days ago) insights: correctly query data even if metadata is not present (#18459) :shipit: -
( 🏁 17 days ago) insights: store: always store data in UTC format (#18460) :shipit: -
( 🏁 17 days ago) insights: store: add fast data existence check (#18461) :shipit: -
( 🏁 16 days ago) insights: disable TimescaleDB telemetry (#18464) :shipit: -
( 🏁 16 days ago) internal/vcs/git: add ability to get first ever commit in a repository (#18463) :shipit: -
( 🏁 16 days ago) insights: query data aggregated over 12h (#18506) :shipit: -
( 🏁 16 days ago) insights: enable in all production deployments (#18515) :shipit: -
( 🏁 16 days ago) code insights: deploy TimescaleDB in docker-compose deployments (#17225; PRs:#259) -
( 🏁 16 days ago) insights: ensure search queries run with a large count: (#18518) :shipit: -
( 🏁 16 days ago) insights: default to 90d display; improve handling of forked/archived repos (#18519) :shipit: -
( 🏁 12 days ago) insights: backend: possible double-inserts (over-reporting) if container restarts happen frequently (#18510) -
( 🏁 4 days ago) dev: make drop-entire-db script also drop insights DB (#18081) :shipit: -
( 🏁 3 days ago) insights: backfill data (#18398) -
( 🏁 today) insights: 3.25 release checklist (#18465) -
( 🏁 today) code insights: document that it is not supported in sourcegraph/server demo deployments (#17222; PRs:#17735)