add --ulimit nofiles=10000:10000 to docker quickstart command?
- Truncate descriptions
Created by: ggilmore
In the course of trying out the AWS docs for https://github.com/sourcegraph/sourcegraph/issues/72, I ran into the following error when I tried to run the docker image on an EC2 instance running Amazon Linux 2:
9:58:17 frontend | !!!!! Rlimit is required
19:58:17 frontend | Problem: Insufficient file descriptor limit
19:58:17 frontend | Possible fix: Please increase the open file limit by running "ulimit -n 10000".
19:58:17 frontend | Skip this check by setting the env var SRC_SKIP_REQS="Rlimit" (separate
19:58:17 frontend | multiple entries with spaces). Note: Sourcegraph may not function
19:58:17 frontend | properly without Rlimit.
That error message comes from here:
Inside the docker container running on Amazon Linux, ulimit
reports a file descriptor hard limit of 4096
and a soft limit of 1024
:
[ec2-user@ip-10-0-0-146 ~]$ docker exec -it sourcegraph /bin/sh
/ # ulimit -Hn
4096
/ # ulimit -n
1024
/ #
These are the same settings that you get when checking the limit outside of the container:
[ec2-user@ip-10-0-0-146 ~]$ ulimit -Hn
4096
[ec2-user@ip-10-0-0-146 ~]$ ulimit -n
1024
I can't reproduce the issue when I run the docker image on a GCP VM running Ubuntu 18.04 (or locally). On Ubuntu, the docker image reports that it has both a hard limit and a soft limit of 1048576
:
ggilmore@geoffrey-docker-test:~$ docker exec -it eager_hofstadter /bin/sh
/ # ulimit -Hn
1048576
/ # ulimit -n
1048576
/ #
Outside the container, Ubuntu reports a hard limit of 1048576
but a soft limit of 1024
:
ggilmore@geoffrey-docker-test:~$ ulimit -Hn
1048576
ggilmore@geoffrey-docker-test:~$ ulimit -n
1024
Manually lowering the hard limit on file descriptors caused the "Insufficient file descriptor limit" issue to show up on Ubuntu as well:
ggilmore@geoffrey-docker-test:~$ docker run -d --publish 7080:7080 --publish 2633:2633 --volume ~/.sourcegraph/config:/etc/sourcegraph --volume ~/.sourcegraph/data:/var/opt/sourcegraph --restart=unless-stopped --ulimit nofile=1024:1024 sourcegraph/server:3.0.0-beta
ggilmore@geoffrey-docker-test:~$ docker logs elegant_minsky
23:03:59 zoekt-indexserver | 2019/01/25 23:03:59 Post http://127.0.0.1:3090/.internal/repos/list: dial tcp 127.0.0.1:3090: connect: connection refused
23:03:59 nginx | nginx: [alert] could not open error log file: open() "./logs/error.log" failed (2: No such file or directory)
23:03:59 postgres | 2019-01-25 23:03:59.404 UTC [141] LOG: listening on IPv4 address "127.0.0.1", port 5432
23:04:00 frontend | !!!!! Rlimit is required
23:04:00 frontend | Problem: Insufficient file descriptor limit
23:04:00 frontend | Possible fix: Please increase the open file limit by running "ulimit -n 10000".
23:04:00 frontend | Skip this check by setting the env var SRC_SKIP_REQS="Rlimit" (separate
23:04:00 frontend | multiple entries with spaces). Note: Sourcegraph may not function
23:04:00 frontend | properly without Rlimit.
23:04:00 frontend | t=2019-01-25T23:04:00+0000 lvl=eror msg="System requirement checks failed (see above for more information)." failed=[Rlimit]
This behavior looks like a duplicate of https://github.com/sourcegraph/enterprise/issues/10157#issuecomment-429572199.
It seems that different operating systems have different file descriptor limits set, so I think that we need to add --ulimit nofile=10000:10000
to the docker run
quickstart command to cover all cases (this setting can only be set in the docker run
command since containers themselves can't control this setting for security reasons).
The only concern that I can think of is that some people might consider --ulimit nofile=10000:10000
to be a security/ops concern - and they might not appreciate that just being silently added to the quickstart command without us explicitly calling that out.

