Redis OOM error prevents sign in or using server.
Created by: mcdan
- Sourcegraph version: sourcegraph/server:2.11.2
- Platform information:
➜ ~ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 16.04.4 LTS
Release: 16.04
Codename: xenial
➜ ~ docker info
Containers: 24
Running: 8
Paused: 0
Stopped: 16
Images: 55
Server Version: 18.03.1-ce
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 462
Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 773c489c9c1b21a6d78b5c538cd395416ec50f88
runc version: 4fc53a81fb7c994640722ac585fa9ca548971871
init version: 949e6fa
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.4.0-104-generic
Operating System: Ubuntu 16.04.4 LTS
OSType: linux
Architecture: x86_64
CPUs: 16
Total Memory: 15.44GiB
Name: coldplay
ID: TDMA:6IQ6:X2BE:S4NA:RDBL:TGJW:B22K:WAET:5N24:RK3E:CDJT:S7QC
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: No swap limit support
Steps to reproduce:
Run the server for a while, with a bunch fo repos. I think I have about 8 or so configured with 5 language servers.
- Try and login
Expected behavior:
- Login and use site.
Actual behavior:
- Login succeeds but you get into a redirect loop: POST /-/sign-in : 200 OK GET /search?toast=integrations : 302 -> /sign-in?returnTo=%2Fsearch%3Ftoast%3Dintegrations POST.... etc, etc. never allows me in.
Meanwhile inside the server container I see 10s of these message per second:
16:06:24 repo-updater | t=2018-10-05T16:06:24+0000 lvl=warn msg="failed to execute redis command" cmd=SETEX error="OOM command not
allowed when used memory > 'maxmemory'."
16:06:24 repo-updater | t=2018-10-05T16:06:24+0000 lvl=warn msg="failed to execute redis command" cmd=SETEX error="OOM command not
allowed when used memory > 'maxmemory'."
16:06:24 repo-updater | t=2018-10-05T16:06:24+0000 lvl=warn msg="failed to execute redis command" cmd=SETEX error="OOM command not
allowed when used memory > 'maxmemory'."
16:06:24 repo-updater | t=2018-10-05T16:06:24+0000 lvl=warn msg="failed to execute redis command" cmd=SETEX error="OOM command not
allowed when used memory > 'maxmemory'."
16:06:24 repo-updater | t=2018-10-05T16:06:24+0000 lvl=warn msg="failed to execute redis command" cmd=SET error="OOM command not al
lowed when used memory > 'maxmemory'."