A/B test: Search notebook examples

Created by: rrhyne

Motivation

Test

Hypothesis, search notebooks will show the unique value propositions of Sourcegraph search, which will increase signups and retention.

Metric and experimental design

Metric: Retention Smaller significant change: Current baseline is 10.4%, if retention is improved by 12%, thus reaches 13.9% the test is a success.

Significance threshold: 5% Duration/size: 1 week or 7,200 users

Descriptive analytics

• Impact on conversion ratio of new visitors to sign-ups.
• Percentage of traffic clicks on a notebook vs. example searches
• Search success funnel: Homepage -> search results -> search success

Flag

AB-test/notebook-examples