Core Services: 3.7 Tracking Issue

Created by: mrnugget

Goal

99th percentile latency of indexed search at 42req/s with 20k repos is under 2s.

Availability

Team:

• Hackweek(s) July 22nd - Aug 2nd (see details here)
• @tsenart & @mrnugget in San Diego for GopherCon from July 22th-27th
• @tsenart, @mrnugget & @keegancsmith in Berlin for Core Meetup August 5th-10th

Invidividuals:

• @mrnugget: Not working August 12th-16th
• @tsenart Available
• @keegancsmith Available

Background

A requirement for future work on Zoekt performance is to define a performance model of Zoekt's current architecture that allows us to predict Zoekt's performance to a certain extent.

After doing back-of-the-napkin math, we came up with the following formula:

"With N cores and M shards it can take up to X to search a single shard when aiming to do 42req/s (T)"

With variables plugged in, that gives us:

• "With 384 cores and 30000 shards and 42 req/s it can take a maximum of 300us to search a single shard"
• "With 1 core and 30000 shards and 42 req/s it can take a maximum of 0.79us to search a single shard"

Planned work

• Global repository permissions caching (#4812 (closed))
• Determine resource allocation "equation" and update docs for admins to provision them accordingly for their Sourcegraph installation (e.g. https://confluence.atlassian.com/bitbucketserver/scaling-bitbucket-server-776640073.html)
• Experiment with more efficient posting lists in Zoekt https://github.com/sourcegraph/zoekt/pull/10
• Scale Zoekt horizontally
  • Write proposal, derive implementation plan from it.

Moar Ideas

• Reduce contention in Zoekt
• Improve indexData.simplify in Zoekt
• Improve parsing of newlines in Zoekt
• Shard compaction
• Shard compression (are we I/O bound?)