Skip to content

search: log search inputs for differential testing

Administrator requested to merge rvt/diff-test-queries-dot-com into master

Created by: rvantonder

I want to log some live DotCom search queries to build a test corpus of inputs. Once collected, we will run these inputs through both the old and new parsers and then test that all outputs of the query interface are equal. Any discrepancies will indicate a potential bug in how the new parser interprets input. The thought occurred to me to do the equivalence testing live, and log discrepancies, but I would rather not and just do this offline.

Chatted with @slimsag about doing it this way, and it'll work. Even though we could log everything according to him, I'll just sample 1 in 10, because I noticed there's a large volume of type:symbol queries that don't matter as much.

I'll pipe the output to some file once this is live and then remove the instrumentation in a weeks time probably, we want to close #12477 this iteration 🤞

Merge request reports

Loading