src debug command

Created by: DaedalusG

Src Debug

src debug is a little pet project I've been working on for a long time now as a way to learn golang. The intention here is to create a cli command that collects the data from common diagnostic commands to speed up debugging and make sharing information about a Sourcegraph instance easier for our support team.

The command is divided into three separate sub commands (server,compose, and kube) each specific to their deployment type and with some unique flags and values.

Here is an example of the return from a src debug server command:

warrengifford@Warrens-MacBook-Pro src-debug-test % src -v debug server -c vibrant_ritchie -o serv3
This command will archive docker-cli data for container: vibrant_ritchie
 SRC_ENDPOINT: https://cse-aws-test.sgdev.org
 Output filename: serv3.zip
Do you want to start writing to an archive? [y/N]: y
archiving file "serv3/inspect-vibrant_ritchie.txt" with 12294 bytes
archiving file "serv3/vibrant_ritchie.log" with 571587 bytes
archiving file "serv3/top-vibrant_ritchie.txt" with 84951 bytes
archiving file "serv3/config/siteConfig.json" with 3772 bytes
archiving file "serv3/config/external_services.txt" with 9480 bytes

This is the simplest version of the command and output

For an abridged example of kubernetes --

warrengifford@Warrens-MacBook-Pro src-debug-test % src debug kube -n ns-sourcegraph -o kube-test
Archiving kubectl data for 33 pods
 SRC_ENDPOINT: https://cse-aws-test.sgdev.org
 Context: gke_beatrix-test-overlay_us-central1-c_beatrix-test
 Namespace: ns-sourcegraph
 Output filename: kube-test.zip
Do you want to start writing to an archive? [y/N]: y

... after unzip

warrengifford@Warrens-MacBook-Pro src-debug-test % cd kube-test
warrengifford@Warrens-MacBook-Pro kube-test % ls
config  kubectl
warrengifford@Warrens-MacBook-Pro kube-test % ls config
external_services.txt   siteConfig.json
warrengifford@Warrens-MacBook-Pro kube-test % ls kubectl
events.txt                      persistent-volume-claims.txt    pods
getPods.txt                     persistent-volumes.txt
warrengifford@Warrens-MacBook-Pro kube-test % ls kubectl/pods
cadvisor-44nvf                                  jaeger-56fbb74f8-jmtjs
cadvisor-55b9d                                  minio-54b558cdd6-hlgbt
cadvisor-79pgg                                  pgsql-8565fd54ff-6n8tk
cadvisor-7s8gw                                  precise-code-intel-worker-67946948c5-f9282
cadvisor-9dxct                                  precise-code-intel-worker-67946948c5-xq8pf
cadvisor-cwmpm                                  prometheus-79f68b8d76-x8skc
cadvisor-gnnbq                                  redis-cache-bb84b65cf-j6gfj
cadvisor-lmd6q                                  redis-store-75ffcfb74b-b8pxf
cadvisor-nq75p                                  repo-updater-978f46dff-27b52
cadvisor-zkspf                                  searcher-67fd7f64cc-l6dqs
codeinsights-db-78b8747cdd-bzj72                searcher-67fd7f64cc-tpjj9
codeintel-db-8fb75ddfd-xhncd                    sourcegraph-frontend-6869dbb7d-tf4v8
github-proxy-7bfd897445-tgtxv                   sourcegraph-frontend-6869dbb7d-tr7rk
gitserver-0                                     symbols-95d9c59d9-v6brz
grafana-0                                       syntect-server-6b4478987-tpvsg
indexed-search-0                                worker-6dc84c8cf5-bkmdh
indexed-search-1
warrengifford@Warrens-MacBook-Pro kube-test % ls kubectl/pods/gitserver-0
describe-gitserver-0.txt        jaeger-agent.log                prev-gitserver.log
gitserver.log                   manifest-gitserver-0.yaml

Here you can see the general directory structure outputted in the zip archive and get an idea about the data captured by the command.

Test plan

I'd love to get some testing from friends here 🙏

I tested each command with every iteration against the two AER test environments cse-aws (the command was tested against a local compose instance for docker cli outputs) and cse-k8s. These instances are fairly small so it might be a good idea to test out the kube command against a much larger instance to test out the semaphore throttling, I don't have kubectl access to .com but that would be a good test ground.

Testing amounted to executing the command and checking to make sure that expected outputs were received and written to a zip file.

To run the command checkout the src-debugger and run the command from there:

wglaptop@Warrens-MacBook-Pro-2 src-cli % pwd
/Users/wglaptop/src-cli
wglaptop@Warrens-MacBook-Pro-2 src-cli % go run ./cmd/src debug
'src debug' gathers and bundles debug data from a Sourcegraph deployment for troubleshooting.

Usage:

	src debug command [command options]

The commands are:

	kube                 dumps context from k8s deployments
	compose              dumps context from docker-compose deployments
	server               dumps context from single-container deployments


Use "src debug [command] -h" for more information about a subcommands.
src debug has access to flags on src -- Ex: src -v kube -o foo.zip

Note that the command makes use of your SRC_ENDPOINT to gather siteconfig.json and external service config json. It also uses your local machine kubectl context to target the Sourcegraph kubernetes cluster, and a host machines docker cli.

Feel free to hit me here or DM me with any questions!