Let’s start with a simple fact: no one likes to fail.
Now, if you happen to also be a product manager, fully accountable for your team’s motivation and your outcomes and the product and possibly – indirectly – the outcome of the business, then you dislike it even more.
As product managers, we tend to run a lot of experiments (a/b tests, MVPs, etc.) so it’s important to take a step back and think about what we should consider failure.
For each and every experiment, we (should) have a hypothesis.
The purpose of any experiment is to test a hypothesis and draw conclusions.
From that perspective, I consider an experiment a failure when I am unable to draw conclusions and establish if I have proven or disproven the hypothesis.
Let me give you 2 examples:
1. A product manager runs an experiment that results in a 5% increase in revenue. When asked what learnings or conclusions did he get out of that experiment, he doesn’t have any. We could argue that in this case, that 5% is a one time bonus. Without any learnings, we don’t have any chance of reproducing the success.
2. A product manager runs an experiment that results in a 7% drop in revenue. However, the experiment and the process of building and running it yields several key learnings that other teams can reuse to avoid making similar mistakes.
Don’t get me wrong, I love experiments with positive business/user outcomes as much as any other product person out there.
What I’m arguing for is that the focus should be on the learnings and you should be concerned if you don’t get any from an experiment.