Skip to content

A/B test: Visitors onboarding tour with search example notebook links

Created by: erzhtor

Motivation

During discussion on semgrep playground and their playground editor with pre-existing examples, there came a hypothesis to make similar onboarding for newcomers:

image

  1. I like their editor with all pre-existing examples. I think this can be helpful in our case. Our “quick tour” shows a few examples of queries, but we could make a more advanced one with multiple syntax examples. The easiest solution, I could think of is to make an intro/example search notebook with search + description.

NOTE: There seem to be a dedicated web portal for that https://learn.sourcegraph.com. Clarify who owns, it and what purpose it is used for.

Metric and experimental design

Metric: # of activated (or signed up) users, + retention ? Smaller significant change: 10% ? Significance threshold: 5% Duration/size: ?

Descriptive analytics

Other things we will be interested in:

  • Individual task completion
  • % of visitors who engage with the checklist?
  • % of visitors who complete the entire checklist flow?
  • % of visitors who complete closes checklist

Test

Experiment/test group: visitors who will get a new tour content with notebook links https://github.com/sourcegraph/sourcegraph/issues/36779, https://github.com/sourcegraph/sourcegraph/pull/36490 Control group: visitors who will get old existing tour content

Flag key: ab-visitor-tour-with-notebooks