k8s.sgdev.org: symbols pod evicted

Created by: ggilmore

A symbols pod was recently evicted from k8s.sgdev.org b/c it was using too much ephemeral storage. Note that we aren't currently mounting a real SSD for the symbols pod to use, and we might want to add one in the future.

However, has the symbols service cache size grown due to all of the recent changes? Is this something that we need to inform our customers about?

cc @sourcegraph/core-services

---

kubectl describe pod symbols-9b56c4b46-tkmnp output:

Name:           symbols-9b56c4b46-tkmnp
Namespace:      default
Priority:       0
Node:           gke-dogfood-full-k8s-dogfood-full-k8s-fe742003-jzv9/
Start Time:     Thu, 15 Aug 2019 00:21:24 -0700
Labels:         app=symbols
                pod-template-hash=561270602
Annotations:    <none>
Status:         Failed
Reason:         Evicted
Message:        The node was low on resource: ephemeral-storage. Container symbols was using 11836Ki, which exceeds its request of 0.
IP:
Controlled By:  ReplicaSet/symbols-9b56c4b46
Containers:
  symbols:
    Image:       index.docker.io/sourcegraph/symbols:3.7.0-rc.1@sha256:9d17b4c95b24996c3184ffc6970e6aa1385cc9d45cf629cd407c796ab9e41735
    Ports:       3184/TCP, 6060/TCP
    Host Ports:  0/TCP, 0/TCP
    Limits:
      cpu:     2
      memory:  2G
    Requests:
      cpu:      500m
      memory:   500M
    Liveness:   http-get http://:http/healthz delay=60s timeout=5s period=10s #success=1 #failure=3
    Readiness:  http-get http://:http/healthz delay=0s timeout=5s period=5s #success=1 #failure=3
    Environment:
      SYMBOLS_CACHE_SIZE_MB:  100000
      POD_NAME:               symbols-9b56c4b46-tkmnp (v1:metadata.name)
      CACHE_DIR:              /mnt/cache/$(POD_NAME)
    Mounts:
      /mnt/cache from cache-ssd (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-2vkvv (ro)
Volumes:
  cache-ssd:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  default-token-2vkvv:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-2vkvv
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:          <none>