syntect-server freeport detection failing

Created by: slimsag

A customer https://app.hubspot.com/contacts/2762526/company/693777200 / https://sourcegraph.slack.com/archives/C011899KB6U/p1597363626348700 reports syntect-server is failing in their EKS cluster with nothing other than:

2020/08/14 00:06:27 worker command: env ROCKET_PORT={{.Port}} /syntect_server
2020/08/14 00:06:27 failed to find free port
2020/08/14 00:06:27 failed to find free port
2020/08/14 00:06:27 failed to find free port
2020/08/14 00:06:27 failed to find free port
2020/08/14 00:06:28 failed to find free port

This behavior happened only recently with no prior changes, indicating that freeport detection was working before in the same deployment but no longer is.

Most likely, this is caused by syntect-server's workers dying and allocating new ports - leading to ephemeral port exhaustion on the EKS host node. However, the logs which indicate which files are problematic appear to not be present (most likely they were paged out due to the number of "failed to find free port" messages.)

https://github.com/sourcegraph/syntect_server makes use of the freeport Go package to spawn worker processes. That package has a number of reported failures 1 2 3 about its localhost-based resolution.

We should adopt a different package and/or https://github.com/phayes/freeport/pull/8 for port allocation and see if that resolves the issue.