In the world of website optimization, A/B testing is used to determine which user experiences are most effective. This requires calculating how long a test will need to run based on the traffic volume to the page/site being tested, the desired/expected observed difference, and the tolerance for Type I and Type II errors.
This calculator enables the test planner to input different values as part of the planning process and then visualize the impact on the test duration and the likelihood of a Type I or Type II error as a means of providing business users who do not have a strong statistical background a degree of intuition regarding the different levers they can pull in their test design.
Key technical details:
- Ultimately, this is just a bunch of interactivity and visualizations built around a simple use of
power.prop.test() - It uses
flexdashboard, so, technically, is combining both RMarkdown and Shiny - Obviously, there is heavy use of
ggplot2
Links:
- RStudio Cloud project: Posit Cloud
- The app on shinyapps.io: https://gilligan.shinyapps.io/sample-size-calculator/
- The app embedded in an iFrame: Thought Leadership On Data, Cloud & AI | Further
- A brief video demo on YouTube: https://www.youtube.com/watch?v=xlaxe_BDuXE
- A blog post that walks through the different myths about A/B testing that many marketers have that this calculator was designed to help overcome: Think Further Blog | Data, Cloud, AI Thought Leaders
A screen cap of the app:
