The use of A/B testing and massive online experimentation to understand user interaction has helped many organizations refine their digital presences, branding, business models, and user experiences. A neat overview of some recent work at Microsoft, Amazon, and others can be found here in the most recent Harvard Business Review.
The framework that the authors describe is applicable for all organizational experimentation, so, as that’s our ‘jam’ at interstitio, I thought I’d share the highlights here.
- Define experimental metrics that are aligned with strategic goals
What are you going to measure and how is it connected to what you’re trying to do? It’s super simple to state, and understand, but it requires discipline to execute effectively. Why? Because it has to happen as part of a conversation, and one that requires trade-offs (usually the easiest place for organizations to fuck up). It’s also important to not just get up the energy to do this once, you have to do this periodically.
- Decompose experimental metrics where you can
Having an understanding of the fine-grained components of your experimental metrics will help you better rationalize why an experimental change delivers the outcome you observe.
- Be skeptical
Experiments are a wonderful way of navigating the unknown, but if you’re not careful, you can lose sight of ‘what’s true’. Running a simple experiment to demonstrate ground truth is a useful control to ensure you don’t get lost. Also, if a result is different or unexpected, it’s usually wrong - at least start with that assumption.
Online experimentation can realize significant annual incremental revenue generation. Organizations are either developing internal infrastructures to routinely enable this practice, or being supported by external platforms such as Optimizely or VWO.
The principles of experimentation described above are as true for offline organizational experiments as they are for online; and the potential revenue generation opportunities are no less significant. We love this stuff, and if you want to start experimenting internally, give us a shout, we’d be more than happy to help.