Skip to content

Add integration-tester utility

Administrator requested to merge rvt/search-integration-testing into master

Created by: rvantonder

I need something to anchor the behavioral changes for basically all the search things I touch. e2e/regression tests are a pain. This PR adds the barebones integration-testing that suits my needs. I'm putting it up for review and not super attached to merging this in (i.e., going overboard during review is really not the intended outcome, I've not taken ultra care to polish). It just seems generally useful. I will be using it locally either way (better than some rando bash commands I used before. There's only one guiding principle: testing needs to be super duper easy. Easy to specify a test, easy to observe changes, and easy to update test values.

How it works:

  1. You set up your local instance or something that has the current search repos we use for testing. Wait for it to all clone.

  2. You add a test like the others you see in this PR:

	{
		Name:  `Global search, repo search by name, nonzero result`,
		Query: `repo:auth0/go-jwt-middleware$`,
	},
  1. Then you run ./integration-tester. You see some output. Either:
  • You decide it looks good to you. Run ./integration-tester update to add/update the golden test files
  • You decide it looks bad. Great, tests are working.

When there's a diff, what you see is just the expected GQL JSON in all it's fullness (though colorized), and what you got instead:

Screen Shot 2020-05-15 at 1 58 33 AM

Everything else is set up to streamlined to easily submit a query and see the GQL results. To wit:

Things I don't do:

  • No defining result types and marshalling the GQL JSON response to them. This means that yes, I can't inspect values/compare values at a finer granularity, but the base GQL definition gqlSearch is good enough for my purposes and a whole bunch of integration tests I've now gone through. Because it's so stupidly easy to update the expected test content in files, this definition (and only this definition) needs to change to accommodate other parts of the response that I may be interested in.

Things I do not really care about right now

You know you can have nice JSON/structural diffing instead of the plain JSON text?

I just need to see whether there was a difference and the rough shape of the GQL response at a glance. Don't want to define GQL types to marshal to and comparing to those values.

Wouldn't this be nice to add to CI?

Maybe but that's not on my mind right now. That needs more effort to figure out the set of frozen repos and such to test against in CI.

Search-specific tests and migration

I've started going through the search tests and migrated the ones that make sense. You can have a look at the notes there. In many cases, search results are nondeterministic (and because I'm not interested in comparing values), I use some combinations of count:1 and stable:yes to get a deterministic ordering--in most of the tests we only care about "did we get results yes or no?"

I'm not committing golden test data at this time--I believe we should make the set of repos to test against smaller than what we have in the search regression tests (and what I used when working on these test cases) and tune the tests accordingly.

It takes about 3.5 seconds to run all the tests included here. That's a tad longer than I like, but it's mostly because I add stable:yes to the queries. 3.5 seconds is a heck of a lot better than >30s timeouts for individual tests and browser stuff though.

@mrnugget it should be straightforward to run campaign-related queries against the GQL API and use the same testing/updating. You can basically just copy search.go and search_tests.go and make it campaigns.go... and go at it. Don't know if this is realistically useful to you though.

Merge request reports