Zach Flynn

What interactions do, why they are just like any other change in the environment post-experiment, and some reassurance Zach Flynn · Follow Published in Towards Data Science · 5 min read · 15 hours ago — Photo by Uriel Soberanes on Unsplash Experiments do not run one at a time. At any moment, hundreds to thousands of experiments run on a mature website. The question comes up: what if these experiments interact with each other?

Decide what to do with experiment results in code Zach Flynn · Follow Published in Towards Data Science · 5 min read · 8 hours ago — Photo by Cytonn Photography on Unsplash The experiment lifecycle is like the human lifecycle. First, a person or idea is born, then it develops, then it is tested, then its test ends, and then the Gods (or Product Managers) decide its worth. But a lot of things happen

We have to draw the line somewhere Zach Flynn · Follow Published in Towards Data Science · 5 min read · 5 hours ago — Photo by Siora Photography on Unsplash It’s become something of a meme that statistical significance is a bad standard. Several recent blogs have made the rounds, making the case that statistical significance is a “cult” or “arbitrary.” If you’d like a classic polemic (and who wouldn’t?), check out: https://www.deirdremccloskey.com/docs/jsm.pdf. This

Why your experiments might never reach significance Zach Flynn · Follow Published in Towards Data Science · 6 min read · Jul 17, 2024 — Photo by Andrik Langfield on Unsplash Introduction Experiments usually compare the frequency of an event (or some other sum metric) after either exposure (treatment) or non-exposure (control) to some intervention. For example: we might compare the number of purchases, minutes spent watching content, or number of clicks on a call-to-action.