Skip to content

symbols: replace pcre lib with a Go function

Administrator requested to merge symbols-regexp-in-go into main

Created by: chrismwendt

Previously, the symbols service used sqlite3-pcre for the REGEXP function, which doesn't work on a Mac M1 (not entirely sure why, but REGEXP is simply not found at query time) and requires a lot of incantations to fit into Sourcegraph's build processes.

After this change, the symbols service will register a Go function that uses the regexp package for the REGEXP function in SQLite queries. Surprisingly it's 3.5x faster ⚡ I verified it works on a fresh Ubuntu machine and on a Mac M1.

Benchmark results BEFORE

[nix-shell:~/sourcegraph]$ env CGO_ENABLED=1 go test -v -bench=. -test.run=sdfsfd ./cmd/symbols/internal/symbols
goos: linux
goarch: amd64
pkg: github.com/sourcegraph/sourcegraph/cmd/symbols/internal/symbols
cpu: Intel(R) Xeon(R) CPU @ 2.30GHz
BenchmarkSearch
BenchmarkSearch/indexing_go-langserver@391
BenchmarkSearch/indexing_go-langserver@391-16         	      30	  36438669 ns/op
BenchmarkSearch/indexing_moby@6e5
BenchmarkSearch/indexing_moby@6e5-16                  	       1	2924069406 ns/op
BenchmarkSearch/searching_go-langserver@391_^sortedImportRecord$
BenchmarkSearch/searching_go-langserver@391_^sortedImportRecord$-16         	    1473	    770081 ns/op
BenchmarkSearch/searching_go-langserver@391_1234doesnotexist1234
BenchmarkSearch/searching_go-langserver@391_1234doesnotexist1234-16         	    1562	    787795 ns/op
BenchmarkSearch/searching_moby@6e5_^fsCache$
BenchmarkSearch/searching_moby@6e5_^fsCache$-16                             	    1495	    796550 ns/op
BenchmarkSearch/searching_moby@6e5_1234doesnotexist1234
BenchmarkSearch/searching_moby@6e5_1234doesnotexist1234-16                  	    1530	    766157 ns/op
PASS
ok  	github.com/sourcegraph/sourcegraph/cmd/symbols/internal/symbols	9.396s

Benchmark results AFTER

[nix-shell:~/sourcegraph]$ env CGO_ENABLED=1 go test -v -bench=. -test.run=sdfsfd ./cmd/symbols/internal/symbols
REGISTER
goos: linux
goarch: amd64
pkg: github.com/sourcegraph/sourcegraph/cmd/symbols/internal/symbols
cpu: Intel(R) Xeon(R) CPU @ 2.30GHz
BenchmarkSearch
BenchmarkSearch/indexing_go-langserver@391
BenchmarkSearch/indexing_go-langserver@391-16         	      31	  35134148 ns/op
BenchmarkSearch/indexing_moby@6e5
BenchmarkSearch/indexing_moby@6e5-16                  	       1	2885099785 ns/op
BenchmarkSearch/searching_go-langserver@391_^sortedImportRecord$
BenchmarkSearch/searching_go-langserver@391_^sortedImportRecord$-16         	    5198	    219659 ns/op
BenchmarkSearch/searching_go-langserver@391_1234doesnotexist1234
BenchmarkSearch/searching_go-langserver@391_1234doesnotexist1234-16         	    4472	    232458 ns/op
BenchmarkSearch/searching_moby@6e5_^fsCache$
BenchmarkSearch/searching_moby@6e5_^fsCache$-16                             	    4720	    237519 ns/op
BenchmarkSearch/searching_moby@6e5_1234doesnotexist1234
BenchmarkSearch/searching_moby@6e5_1234doesnotexist1234-16                  	    5226	    218710 ns/op
PASS
ok  	github.com/sourcegraph/sourcegraph/cmd/symbols/internal/symbols	8.828s

PCRE was added in https://github.com/sourcegraph/sourcegraph/pull/2418

cc @sourcegraph/distribution

Merge request reports