Skip to content
Snippets Groups Projects
Closed add --ulimit nofiles=10000:10000 to docker quickstart command?
  • View options
  • add --ulimit nofiles=10000:10000 to docker quickstart command?

  • View options
  • Closed Issue created by Warren Gifford

    Created by: ggilmore

    In the course of trying out the AWS docs for https://github.com/sourcegraph/sourcegraph/issues/72, I ran into the following error when I tried to run the docker image on an EC2 instance running Amazon Linux 2:

    9:58:17           frontend |  !!!!!  Rlimit is required
    19:58:17           frontend |     Problem: Insufficient file descriptor limit
    19:58:17           frontend |     Possible fix: Please increase the open file limit by running "ulimit -n 10000".
    19:58:17           frontend |     Skip this check by setting the env var SRC_SKIP_REQS="Rlimit" (separate
    19:58:17           frontend |         multiple entries with spaces). Note: Sourcegraph may not function
    19:58:17           frontend |         properly without Rlimit.

    That error message comes from here:

    https://github.com/sourcegraph/sourcegraph/blob/cac576ae749b04dce921fc883a6ba338efd45c11/pkg/sysreq/unix.go#L13-L25

    Inside the docker container running on Amazon Linux, ulimit reports a file descriptor hard limit of 4096 and a soft limit of 1024:

    [ec2-user@ip-10-0-0-146 ~]$ docker exec -it sourcegraph /bin/sh
    / # ulimit -Hn
    4096
    / # ulimit -n
    1024
    / #

    These are the same settings that you get when checking the limit outside of the container:

    [ec2-user@ip-10-0-0-146 ~]$ ulimit -Hn
    4096
    [ec2-user@ip-10-0-0-146 ~]$ ulimit -n
    1024

    I can't reproduce the issue when I run the docker image on a GCP VM running Ubuntu 18.04 (or locally). On Ubuntu, the docker image reports that it has both a hard limit and a soft limit of 1048576:

    ggilmore@geoffrey-docker-test:~$ docker exec -it eager_hofstadter /bin/sh
    / # ulimit -Hn
    1048576
    / # ulimit -n
    1048576
    / #

    Outside the container, Ubuntu reports a hard limit of 1048576 but a soft limit of 1024:

    ggilmore@geoffrey-docker-test:~$ ulimit -Hn
    1048576
    ggilmore@geoffrey-docker-test:~$ ulimit -n
    1024

    Manually lowering the hard limit on file descriptors caused the "Insufficient file descriptor limit" issue to show up on Ubuntu as well:

    ggilmore@geoffrey-docker-test:~$ docker run -d --publish 7080:7080 --publish 2633:2633 --volume ~/.sourcegraph/config:/etc/sourcegraph --volume ~/.sourcegraph/data:/var/opt/sourcegraph --restart=unless-stopped --ulimit nofile=1024:1024 sourcegraph/server:3.0.0-beta
    ggilmore@geoffrey-docker-test:~$ docker logs elegant_minsky
    23:03:59  zoekt-indexserver | 2019/01/25 23:03:59 Post http://127.0.0.1:3090/.internal/repos/list: dial tcp 127.0.0.1:3090: connect: connection refused
    23:03:59              nginx | nginx: [alert] could not open error log file: open() "./logs/error.log" failed (2: No such file or directory)
    23:03:59           postgres | 2019-01-25 23:03:59.404 UTC [141] LOG:  listening on IPv4 address "127.0.0.1", port 5432
    23:04:00           frontend |  !!!!!  Rlimit is required
    23:04:00           frontend | 	Problem: Insufficient file descriptor limit
    23:04:00           frontend | 	Possible fix: Please increase the open file limit by running "ulimit -n 10000".
    23:04:00           frontend | 	Skip this check by setting the env var SRC_SKIP_REQS="Rlimit" (separate
    23:04:00           frontend | 		multiple entries with spaces). Note: Sourcegraph may not function
    23:04:00           frontend | 		properly without Rlimit.
    23:04:00           frontend | t=2019-01-25T23:04:00+0000 lvl=eror msg="System requirement checks failed (see above for more information)." failed=[Rlimit]

    This behavior looks like a duplicate of https://github.com/sourcegraph/enterprise/issues/10157#issuecomment-429572199.

    It seems that different operating systems have different file descriptor limits set, so I think that we need to add --ulimit nofile=10000:10000 to the docker run quickstart command to cover all cases (this setting can only be set in the docker run command since containers themselves can't control this setting for security reasons).

    The only concern that I can think of is that some people might consider --ulimit nofile=10000:10000 to be a security/ops concern - and they might not appreciate that just being silently added to the quickstart command without us explicitly calling that out.

    Activity

    • All activity
    • Comments only
    • History only
    • Newest first
    • Oldest first