codeintel: Rewrite janitor

Created by: efritz

This is an attempt to clean up the janitor to better fit Postgres as the data store. This is partially a cleanup effort from the migration (RFC 235). There was a lot of cruft around this piece. The janitor now has two distinct responsibilities, some of which will be permanent:

• Remove all upload files on disk older than a certain age
• Remove all upload part files on disk older than a certain age
• Soft delete lsif_uploads records that are uploading longer than a given time period
• Soft delete lsif_uploads records with no matching repository
• Soft delete lsif_uploads records that are not visible at tip and are older than a given time period
• Hard delete lsif_uploads records transactionally with the data associated with it in the codeintel db
• (temporary) Find all data in the codeintel db that has no matching lsif_uploads record and delete it. This is to catch stuff that falls through the cracks and should be subsumed by the task above in normal operation.

The code for the first two can be taken out entirely if we use S3/GCS with object lifecycle management and are the only file management tasks that remain in code.

This started as an effort to update our monitoring for the bundle manager, but I pulled on too much of the noodle and this meatball fell out.

5 second rule for reviewers starts NOW.

To review: Read the new version of enterprise/cmd/precise-code-intel-bundle-manager/internal/janitor/janitor.go and compare to the list above.

This closes https://github.com/sourcegraph/sourcegraph/issues/12168 and covers any outstanding details of RFC 205.

Created by: sourcegraph-bot

Notifying subscribers in CODENOTIFY files for diff cfdb8d8b38664418ca3815c6cad65a905fe83e57...85514e1e796ce6e69634f60a78fbeffc67912c55.

Notify	File(s)
@beyang	enterprise/cmd/precise-code-intel-bundle-manager/env.go enterprise/cmd/precise-code-intel-bundle-manager/internal/janitor/db_records.go enterprise/cmd/precise-code-intel-bundle-manager/internal/janitor/db_records_test.go enterprise/cmd/precise-code-intel-bundle-manager/internal/janitor/expired.go enterprise/cmd/precise-code-intel-bundle-manager/internal/janitor/janitor.go enterprise/cmd/precise-code-intel-bundle-manager/internal/janitor/metrics.go enterprise/cmd/precise-code-intel-bundle-manager/internal/janitor/old_files.go enterprise/cmd/precise-code-intel-bundle-manager/internal/janitor/old_files_test.go enterprise/cmd/precise-code-intel-bundle-manager/internal/janitor/orphans.go enterprise/cmd/precise-code-intel-bundle-manager/internal/janitor/orphans_test.go enterprise/cmd/precise-code-intel-bundle-manager/main.go
@bobheadxi	monitoring/CODENOTIFY monitoring/precise_code_intel_bundle_manager.go
@nicksnyder	monitoring/CODENOTIFY
@slimsag	monitoring/CODENOTIFY monitoring/precise_code_intel_bundle_manager.go

Created by: nicksnyder

@efritz The @sourcegraph/code-intelligence codenotify mention is not working because the team's name is @sourcegraph/code-intel. Can you update the necessary CODENOTIFY files with the correct team name?

Alternatively, you should maybe discuss with the team whether they do in fact want to be notified of every change (maybe they don't?).

Created by: efritz

@Strum355 Nope, #14695 is stacked on top of this branch.

Created by: codecov[bot]

Codecov Report

Merging #14684 into main will increase coverage by 0.00%. The diff coverage is 54.48%.

@@           Coverage Diff            @@
##             main   #14684    +/-   ##
========================================
  Coverage   52.25%   52.25%            
========================================
  Files        1556     1552     -4     
  Lines       79200    79237    +37     
  Branches     6933     7034   +101     
========================================
+ Hits        41383    41407    +24     
- Misses      34070    34083    +13     
  Partials     3747     3747            

Flag	Coverage Δ
#go	52.57% <54.48%> (+<0.01%)
#integration	30.69% <ø> (+0.01%)
#storybook	22.05% <ø> (-0.01%)
#typescript	51.48% <ø> (+<0.01%)
#unit	33.35% <ø> (ø)

Impacted Files	Coverage Δ
enterprise/internal/codeintel/store/store.go	53.84% <ø> (ø)
...e-intel-bundle-manager/internal/janitor/janitor.go	39.49% <40.51%> (+36.46%)
enterprise/internal/codeintel/store/uploads.go	89.04% <88.88%> (-0.06%)
...e-intel-bundle-manager/internal/janitor/metrics.go	100.00% <100.00%> (ø)
...terprise/internal/codeintel/store/observability.go	97.64% <100.00%> (+0.04%)
client/web/src/tree/TreeRoot.tsx	86.20% <0.00%> (-1.73%)
...b/src/enterprise/codeintel/CodeIntelUploadPage.tsx	56.00% <0.00%> (-1.34%)
...recise-code-intel-worker/internal/worker/worker.go	0.00% <0.00%> (ø)
cmd/frontend/graphqlbackend/zoekt.go	74.80% <0.00%> (+0.25%)
...t/web/src/search/input/interactive/FilterInput.tsx	89.61% <0.00%> (+0.64%)
... and 2 more