This PR is meant to cleanup the mess I have created on dotcom by manually removing repos from the DB with an SQL command. Instead of soft delete I did a hard delete, which seems to cause the repositories never to be removed from disk.
I think this new behavior is actually beneficial and can remove stale data from the disks, in case other inconsistencies between current disk state and what's defined in the DB exist.
Screenshot
This is how the new metric looks like in the grafana dashboard:
Test plan
This feature is behind an env flag SRC_REMOVE_NON_EXISTING_REPOS which by default is false.
So far tested locally and it seems to be working fine in these scenarios:
Flag is not set - nothing happened
Flag is set to false - nothing happened
Flag is set to true - removed repositories, that were on disk, but not defined in the DB
Flag is set to true, 2nd run - nothing happened
Flag is set to true, deleted a row in the repo table - repo deleted from disk as well
