dev/ci/cluster-test: remove cadvisor from test cluster
Created by: bobheadxi
Right now test clusters deploy into the same cluster as buildkite agents, but under a different namespace. This causes some weirdness with daemonsets like cAdvisor, which will deploy one very node that the test cluster has a service on, which can be quite a few due to the number of buildkite agents.
Example from build:
2021-09-24 12:51:40 EDT | cadvisor-2f5kh 1/1 Running 0 6m51s
-- | --
| 2021-09-24 12:51:40 EDT | cadvisor-5krrj 1/1 Running 0 6m50s
| 2021-09-24 12:51:40 EDT | cadvisor-5tz24 1/1 Running 0 6m51s
| 2021-09-24 12:51:40 EDT | cadvisor-5v4wf 1/1 Running 0 6m50s
| 2021-09-24 12:51:40 EDT | cadvisor-6bp77 1/1 Running 0 6m50s
| 2021-09-24 12:51:40 EDT | cadvisor-6ccv2 1/1 Running 0 6m52s
| 2021-09-24 12:51:40 EDT | cadvisor-6cn82 1/1 Running 0 6m52s
| 2021-09-24 12:51:40 EDT | cadvisor-6nxvl 1/1 Running 0 6m49s
| 2021-09-24 12:51:40 EDT | cadvisor-78s4t 1/1 Running 0 4m29s
| 2021-09-24 12:51:40 EDT | cadvisor-7nrjp 1/1 Running 0 6m52s
| 2021-09-24 12:51:40 EDT | cadvisor-86kzt 1/1 Running 0 6m50s
| 2021-09-24 12:51:40 EDT | cadvisor-8ck5n 1/1 Running 0 6m50s
| 2021-09-24 12:51:40 EDT | cadvisor-8ns6h 1/1 Running 0 6m52s
| 2021-09-24 12:51:40 EDT | cadvisor-8w7sh 1/1 Running 0 6m51s
| 2021-09-24 12:51:40 EDT | cadvisor-9pkpl 1/1 Running 0 6m51s
| 2021-09-24 12:51:40 EDT | cadvisor-9s4lr 1/1 Running 0 6m52s
| 2021-09-24 12:51:40 EDT | cadvisor-9s8pv 1/1 Running 0 6m50s
| 2021-09-24 12:51:40 EDT | cadvisor-9xk8d 1/1 Running 0 6m52s
| 2021-09-24 12:51:40 EDT | cadvisor-9xmhj 1/1 Running 0 6m52s
| 2021-09-24 12:51:40 EDT | cadvisor-bnk24 1/1 Running 0 6m51s
| 2021-09-24 12:51:40 EDT | cadvisor-bzb9t 1/1 Running 0 6m50s
| 2021-09-24 12:51:40 EDT | cadvisor-c9pbn 1/1 Running 0 6m51s
| 2021-09-24 12:51:40 EDT | cadvisor-ccj2q 1/1 Running 0 6m51s
| 2021-09-24 12:51:40 EDT | cadvisor-dh747 1/1 Running 0 6m52s
| 2021-09-24 12:51:40 EDT | cadvisor-dnkw9 1/1 Running 0 6m50s
| 2021-09-24 12:51:40 EDT | cadvisor-dpclb 1/1 Running 0 6m52s
| 2021-09-24 12:51:40 EDT | cadvisor-fhw94 1/1 Running 0 6m52s
| 2021-09-24 12:51:40 EDT | cadvisor-fk69b 1/1 Running 0 6m49s
| 2021-09-24 12:51:40 EDT | cadvisor-fzj5b 1/1 Running 0 6m51s
| 2021-09-24 12:51:40 EDT | cadvisor-h9mxv 1/1 Running 0 6m49s
| 2021-09-24 12:51:40 EDT | cadvisor-h9pjv 1/1 Running 0 6m52s
| 2021-09-24 12:51:40 EDT | cadvisor-hd6zz 1/1 Running 0 6m52s
| 2021-09-24 12:51:40 EDT | cadvisor-hpm2c 1/1 Running 0 6m51s
| 2021-09-24 12:51:40 EDT | cadvisor-jbjhh 1/1 Running 0 6m50s
| 2021-09-24 12:51:40 EDT | cadvisor-jw9jt 1/1 Running 0 6m51s
| 2021-09-24 12:51:40 EDT | cadvisor-k4mkv 1/1 Running 0 6m52s
| 2021-09-24 12:51:40 EDT | cadvisor-k9ggc 1/1 Running 0 6m51s
| 2021-09-24 12:51:40 EDT | cadvisor-lcgmr 1/1 Running 0 6m50s
| 2021-09-24 12:51:40 EDT | cadvisor-lhfc4 1/1 Running 0 6m50s
| 2021-09-24 12:51:40 EDT | cadvisor-ltmmh 1/1 Running 0 6m50s
| 2021-09-24 12:51:40 EDT | cadvisor-nhtqk 1/1 Running 0 6m51s
| 2021-09-24 12:51:40 EDT | cadvisor-nr7wr 1/1 Running 0 6m51s
| 2021-09-24 12:51:40 EDT | cadvisor-nscjp 1/1 Running 0 6m49s
| 2021-09-24 12:51:40 EDT | cadvisor-p66vp 1/1 Running 0 6m52s
| 2021-09-24 12:51:40 EDT | cadvisor-pmkzk 1/1 Running 0 6m52s
| 2021-09-24 12:51:40 EDT | cadvisor-px7d8 1/1 Running 0 6m50s
| 2021-09-24 12:51:40 EDT | cadvisor-qdp7k 1/1 Running 0 6m51s
| 2021-09-24 12:51:40 EDT | cadvisor-qjkzn 1/1 Running 0 6m50s
| 2021-09-24 12:51:40 EDT | cadvisor-qjw5j 1/1 Running 0 6m51s
| 2021-09-24 12:51:40 EDT | cadvisor-qkqtx 1/1 Running 0 6m52s
| 2021-09-24 12:51:40 EDT | cadvisor-qnm7s 1/1 Running 0 6m52s
| 2021-09-24 12:51:40 EDT | cadvisor-rht89 1/1 Running 0 6m51s
| 2021-09-24 12:51:40 EDT | cadvisor-s8nj4 1/1 Running 0 6m50s
| 2021-09-24 12:51:40 EDT | cadvisor-th66d 1/1 Running 0 6m50s
| 2021-09-24 12:51:40 EDT | cadvisor-vdj68 1/1 Running 0 6m51s
| 2021-09-24 12:51:40 EDT | cadvisor-w87fx 1/1 Running 0 6m52s
| 2021-09-24 12:51:40 EDT | cadvisor-wgm7k 1/1 Running 0 6m50s
| 2021-09-24 12:51:40 EDT | cadvisor-wpqfl 1/1 Running 0 6m50s
| 2021-09-24 12:51:40 EDT | cadvisor-x4qc8 1/1 Running 0 6m51s
| 2021-09-24 12:51:40 EDT | cadvisor-xbb6v 1/1 Running 0 6m50s
| 2021-09-24 12:51:40 EDT | cadvisor-xh57k 1/1 Running 0 6m51s
| 2021-09-24 12:51:40 EDT | cadvisor-xrh8w 1/1 Running 0 6m49s
| 2021-09-24 12:51:40 EDT | cadvisor-zwwx8 1/1 Running 0 6m50s
| 2021-09-24 12:51:40 EDT | codeinsights-db-7c49c65fc5-znwlh 1/1 Running 1 6m52s
| 2021-09-24 12:51:40 EDT | codeintel-db-66b65c7f99-5c6s9 2/2 Running 0 6m51s
| 2021-09-24 12:51:40 EDT | github-proxy-6649c7d457-vkwlh 2/2 Running 0 6m51s
| 2021-09-24 12:51:40 EDT | gitserver-0 2/2 Running 5 6m51s
| 2021-09-24 12:51:40 EDT | grafana-0 1/1 Running 0 6m51s
| 2021-09-24 12:51:40 EDT | indexed-search-0 2/2 Running 0 6m51s
| 2021-09-24 12:51:40 EDT | jaeger-8dd74897c-6kdgg 1/1 Running 0 6m51s
| 2021-09-24 12:51:40 EDT | minio-667b87dc85-gncm6 1/1 Running 0 6m51s
| 2021-09-24 12:51:40 EDT | pgsql-65fdc5856-988b4 2/2 Running 0 6m50s
| 2021-09-24 12:51:40 EDT | precise-code-intel-worker-6dc64748d5-fl9vh 1/1 Running 0 6m50s
| 2021-09-24 12:51:40 EDT | precise-code-intel-worker-6dc64748d5-txn8s 1/1 Running 0 6m50s
| 2021-09-24 12:51:40 EDT | prometheus-6bd8dc46d7-828pl 1/1 Running 0 6m49s
| 2021-09-24 12:51:40 EDT | query-runner-9ff666c86-ppptq 2/2 Running 0 6m50s
| 2021-09-24 12:51:40 EDT | redis-cache-dc85448b9-sstvn 2/2 Running 0 6m50s
| 2021-09-24 12:51:40 EDT | redis-store-6bbb88bc5-54wwc 2/2 Running 0 6m50s
| 2021-09-24 12:51:40 EDT | repo-updater-86649c7498-thvq4 2/2 Running 0 6m49s
| 2021-09-24 12:51:40 EDT | searcher-68b4fdbfbd-dggsk 2/2 Running 0 6m49s
| 2021-09-24 12:51:40 EDT | searcher-68b4fdbfbd-nk6h8 2/2 Running 0 6m49s
| 2021-09-24 12:51:40 EDT | sourcegraph-frontend-58ff9d846c-p4crc 2/2 Running 5 6m51s
| 2021-09-24 12:51:40 EDT | sourcegraph-frontend-58ff9d846c-vn5wm 2/2 Running 5 6m51s
| 2021-09-24 12:51:40 EDT | symbols-7c98f97758-dzvfx 2/2 Running 0 6m49s
| 2021-09-24 12:51:40 EDT | syntect-server-85d46fbc9f-87dtz 1/1 Running 0 6m49s
| 2021-09-24 12:51:40 EDT | worker-5c89bbd9fd-mtcs6 1/1 Running 2 6m49s
| 202We might want to drop cAdvisor from test clusters entirely. Alternatively, we could deploy test clusters into newly created clusters like we do for https://github.com/sourcegraph/deploy-sourcegraph, which might mitigate issues with pod scheduling as well (e.g. https://github.com/sourcegraph/sourcegraph/pull/25338)