The 7 Broader Topics of Discussion in Experimentation

The narrative I have followed till date on this blog started off with the fundamentals of controlled experimentation (1,2) and slowly dived into statistics (1,2,3). From there I built upon one of the most interesting debates in experimentation statistics which was the Bayesian vs Frequentist debate (1,2,3,4,5). Since the blog has reached this far, I think it is time that I review and carefully draw out the road ahead in experimentation. This note is an overview of the 7 broader topics on experimentation that I will spend time discussing in the next few weeks.

I eagerly look for feedback and request topics that the interested reader feels I should cover and have missed my list. In subsequent notes, I will dig one layer deeper into each topic and share the blog topics I have in mind corresponding to each topic. My aim is to create a comprehensive blog on experimentation that helps a reader get introduced to the field. I also want to state that I expect this list to evolve and mature over time.

The 7 Major Topics of Discussion

After careful thought, I have outlined the following major ideas that I want to discuss in the coming few weeks in this blog. These ideas create a major part of the discussion in experimentation.

1. Statistical Nuances: While our statistical journey up till now explores two methods of calculating statistical significance, I have not started to discuss the various complexities of statistics that plague both the Bayesian and the Frequentist methods. These are the general problems of statistics such as multiple testing (testing for more than one pair of comparisons), false positives (seeing significance when actually there is not), and non-stationarity (when the real world does not follow the assumptions of testing). These will be valid for those who are wanting to level up their statistical rigor in experimentation.
2. Complexities of Reality: While statistics are forced to assume some simplifications in the data, the real world is much more chaotic and turbulent. In this part of the discussion, I will explore concepts such as the Primacy/Novelty effects, Weekday-Weekend Effect, and Twyman's Law (a concept I learned from Ronny Kohavi). These are the harsh realities of the real world that an experimenter should know.
3. Components of an Experimentation Engine: In this side of the discussion, I will explore the various components of a full-fledged experimentation engine that large-scale companies have started building. These include components such as a Hazard Detector, Framework for Concurrent Experiments, Institutional Memory, and enablement of discussions on experiments.
4. Culture of Experimentation: One part of the discussion on experimentation is about the culture of experimentation and how can one evolve experimentation in their own organizations. Experimentation usually starts with a very deep philosophical change of realizing that we live in a highly uncertain world and we know much lesser than we think we know. This section will deal with the journey from there on and provide you with the concepts to understand experimentation in a philosophical and a business context.
5. Deeper into Experimentation: While experimentation still remains one of the most powerful scientific tools, the tech industry with the help of data and data science has made advanced tools beyond experimentation. These are tools like Personalisation, Multi-armed bandits, and Multi-variate testing. This side of the debate explores the advanced innovations that have been done in the experimentation field and those that remain uncovered now.
6. Tips, tricks, and interesting experiments: While the theory is dense, I regularly see digital marketers and data scientists sharing their hard-earned thumb rules and the stories of interesting experiments they have done. I intend to compile a list of that practical wisdom under various logical heads to help you remember the core concepts of experimentation.
7. Discovering Causality from data: Moving beyond experimentation: Experimentation is the best tool we have to date to establish causality in scientific inquiry. However, the recent developments in computer science and AI have convinced me that the road ahead is the discovery of causality through observational data. The dream seems to be inching slowly towards realization yet it is hard to predict the bumps in the road ahead. In this part of the discussion, I will share various learnings I have about thinking about experimentation from a causal angle.

Note that each of the above broader topics will be a collection of blogs. In the upcoming few notes, I will continue to dive deeper into each topic and give an overview of the ideas I have for each of them. Later, I will start to go deeper one by one into each topic.

On days when I am not able to write

To be honest, on days I do not have the fuel to write (which I will try are less), I try to add various book summaries that I have written through the course of reading books in the past year (1,2,3). Please do let me know if the diversion in the narrative is jarring to readers. I will try and create a separate list out of it. Also, if you have interesting book recommendations, drop me a message I will try to read it and write a summary on it.

I hope I am able to provide value to my readers in the days to come with this blog. I look forward to your comments on my plan and generally on the blog (directly mail them to me, or put them on LinkedIn). For those interesting in commenting here, apologies for not adding the comments feature yet in my blog. (it is costing me more money, as subscribers increase I will make that investment).