What gets measured gets managed

“What gets measured gets managed” was coined by Peter Drucker, regarded as the father of modern management, in 1954. It is an often quoted saying which is actually part of a larger, and I think more powerful quote “What gets measured gets managed — even when it’s pointless to measure and manage it, and even if it harms the purpose of the organization to do so.”

Drucker’s insight underscores that, while gathering and measuring data is essential, the real challenge lies in identifying and prioritizing the right metrics that will drive a business in the right direction. By focusing on and prioritizing the right metrics, you can ensure that what gets measured and managed is truly impactful.

This blog focuses on product analytics in technology companies, however this idea rings true for all businesses and types of analytics. Below is a summary of what I’ve learnt and applied working as a data professional in a start-up (Digivizer), a scale-up (Immutable), and a big tech company (Facebook) across a range of different products.

How should you prioritize metrics?

The most important metrics for a company change over time. Uber was not profitable for around 15 years, yet the company is considered one of the most successful businesses in recent time. Why? Uber focused intensely on rapid growth in its initial years rather than immediate profitability. The company prioritized metrics like user growth and user retention to establish a dominant presence in the ride-sharing market. Then, once Uber became the dominant ride-sharing company, its focus shifted towards profitability and financial sustainability. It, like many others, anchored their metrics to the stages of the product lifecycle.

You should prioritize metrics based on the product lifecycle stages.

The metrics you focus on during each stage should help answer the urgent problems that each stage presents. The tactical problems can vary but will derive from the following high level questions:

• Stage 1 — Introduction: Do we have product-market-fit?
• Stage 2 — Growth: How do we scale effectively?
• Stage 3 — Maturity: How can we be profitable?
• Stage 4 — Decline: How do we maintain user interest and slow decline?

Stage 1 — Introduction: Do we have product-market-fit?

The first and most crucial stage in the product lifecycle is the Introduction stage, where the primary focus is on achieving product-market-fit. At this stage, product owners must determine whether their product meets a genuine market need and resonates with the target audience. Understanding product-market-fit involves assessing whether early adopters are not only using the product but also finding value in it. Being confident in product-market-fit sets the foundation for future growth and scalability.

There are 3 metrics that can provide clarity on whether you have achieved product-market-fit. These are, in order of importance:

1. Retention: Do users find value in the product? Example metrics: D30 Retention, Cohort Retention Curves.
2. Active Users: How many users does the product have? Is this increasing? Example metrics: Daily Active Users (DAU), Monthly Active Users (MAU), Growth Accounting.
3. Stickiness: Is the product engaging and used frequently? Example metrics: DAU/MAU, Activity Frequency Histogram (sometimes called L28 Histogram).

Used together, these three metrics can quantitatively measure whether there is product-market-fit or point to the most critical product issue. There are 5 potential scenarios you will fall into:

1. No long-term retention and low user growth (worst case): In this scenario there is no product-market-fit. Users are not returning to use the product and there is a small market. This requires large changes in the product and/or the target market.
2. No long-term retention but high user growth: This is the leaky bucket problem. Users are being acquired, using the product for a short period, but all eventually churn. Focus here is on fixing retention and slowing down growth.
3. Long-term retention but low user growth: Focus in this scenario is to either adjust the acquisition funnel to improve user growth or, if the market proves to be small, pivot to a larger market.
4. Long-term retention, high user growth, but low stickiness: This is a utility product that users find value in, but are using infrequently. Examples include tax preparation apps, travel websites and event ticketing sites. Focus should be exploring new features that make the product more engaging.
5. Long-term retention, high user growth, and high stickiness (ideal state): Users are returning to the product, using it frequently and the user numbers are growing. This shows product-market-fit.

Once an organization has confidence in product-market-fit, the attention can shift to growth. This approach avoids spending large amounts on user acquisition only to have to pivot the product or market, or have the majority of users churn.

Stage 2 — Growth: How do we scale effectively?

The Growth stage is where a product has the potential to move from promising to dominant. A perfect example of effective scaling is Facebook’s famous “8 friends in 10 days” rule. By using funnel analysis and experimentation, Facebook discovered that new users who connected with at least 8 friends within their first 10 days were far more likely to remain active on the platform. This insight led to focused efforts on optimizing user onboarding and encouraging friend connections, significantly boosting user retention and stickiness. In this stage, the key question is: how do we scale effectively while maintaining product quality and user satisfaction?

Analytics in this stage should broaden to include 3 types:

1. User Journey Analysis: How do we optimize the user experience? Example metrics: Conversion Rate, Time to Convert, Funnels.
2. Experimentation: How can we determine whether a change will positively improve key metrics? Example methods: A/B Testing, Multivariate Testing.
3. ‘Aha’ Analysis: What is the moment that causes a step-change in a users retention and stickiness. Example metrics: A combination of user journey analysis, experimentation and product-market-fit metrics.

When implementing user journey analysis, less is more. The temptation may be to instrument every page and every button in a product, but this can often be onerous for engineering to implement and difficult to maintain. Instead, start with just a beginning and end event — these two events will allow you to calculate a conversion rate and a time to convert. Expand beyond two events to only include critical steps in a user journey. Ensure that events capture user segments such as device, operating system and location.

Experimentation is a muscle that requires exercise. You should start building this capability early in a product and company’s lifecycle because it is more difficult to implement than a set of metrics. Build the muscle by involving product, engineering and data teams in experiment design. Experimentation is not only crucial in ‘Stage 2 — Growth’ but should remain a fundamental part of analytics throughout the rest of the product lifecycle.

‘Aha’ Analysis helps identify pivotal moments that can turbocharge growth. These are the key interactions where users realize the product’s value, leading to loyalty and stickiness. Facebook’s 8 friends in 10 days was their users ‘aha’ moment. This analysis requires analysts to explore a variety of potential characteristics and can be difficult to identify and distil down to a simple ‘aha’ moment. Be sure to use the hypothesis driven approach to avoid boiling the ocean.

Stage 3 — Maturity: How can we be profitable?

In the Maturity stage, the focus shifts from rapid growth to optimizing for profitability and long-term sustainability. This phase is about refining the product, maximizing efficiency, and ensuring the business remains competitive. Companies like Apple, Netflix and Amazon have successfully navigated this stage by honing in on cost management, increasing user revenue, and exploring new revenue streams.

Focus in this stage shifts to:

1. Monetization Metrics: How can we be profitable while maintaining a high-quality product and satisfied customer base? Example metrics: Customer Acquisition Cost (CAC), Customer Lifetime Value (LTV), LTV:CAC Ratio, Monthly Recurring Revenue (MRR).

Monetization metrics have clear objectives in terms of trying to increase revenue and decrease costs. Marketing and Go-To-Market teams often own CAC reduction and product teams often own LTV and MRR improvement. Strategies can range from optimizing advertising spend, reducing time to close sales deals through to cross-selling and bundling products for existing users. Broadly, a LTV:CAC ratio of 3:1 to 4:1 is often used as a target for B2B software companies while B2C targets are closer to 2.5:1.

Stage 4 — Decline: How do we maintain user interest and slow decline?

“Your margin is my opportunity” — Jeff Bezos. As products mature, profitability inevitably declines. Competitors identify your opportunity and increase competition, existing users migrate to substitutes and new technologies, and markets become saturated, offering little growth. In this phase, maintaining the existing user base becomes paramount.

In Stage 4, there are a broad set of useful metrics that can be adopted. Some key types are:

• Churn Prediction Modelling: Can we identify users likely to churn and intervene? Example models: Logistic Regression, Tree Models, Neural Networks.
• Power User Analysis: What can we learn from the most engaged users? Example metrics: Stickiness, Feature Usage, Transaction Volume.
• Root Cause Analysis: What are the root cause drivers of key metrics? Example analysis: Quarterly Business Reviews, Issue Driver Trees.

By creating churn prediction models and analyzing the feature importance, characteristics of users who are likely to churn can be identified and intervention measures deployed. Given new user growth has slowed, retaining existing users is critical. This analysis may also help resurrect previously churned users too.

Power user analysis seeks to understand the most engaged users and their characteristics. These users are the highest priority to retain, and have the product-usage behavior that would ideally be shared across all users. Look for users active every day, who spend long periods of time in the product, who use the most features and who spend the most. Deploy measures, such as loyalty programs, to retain these users and identify pathways to increase the number of power users.

Root cause analysis is essential for delving into specific problem areas within a mature product. Given the complexity and scale of products at this lifecycle stage, having the capability to conduct bespoke deep dives into issues is vital. This type of analysis helps uncover the underlying drivers of key metrics, provides confidence in product changes that are costly to implement and can help untangle the interdependent measures across the product ecosystem.

A product or company who finds themselves in this final stage may choose to create new products and enter new markets. At that point, the cycle begins again and the focus shifts back to product-market-fit at the start of this blog.

Conclusion

“Focus is about saying no.” — Steve Jobs. Product analytics is a bottomless pit of potential metrics, dimensions and visualizations. To effectively use product analytics, companies must prioritize metrics down to a few focus areas at any one time. These metrics can be supported by a range of other measures, but must have the following:

• Teams aligned on which metrics should be prioritized
• Teams who deeply understand the definition of key metrics
• Metrics that are tied to a key product question
• A tangible action which can be taken to improve the metric

This can be achieved by prioritizing the right metrics at each product lifecycle stage — Introduction, Growth, Maturity, and Decline. From achieving product-market fit to scaling effectively, optimizing for profitability, and maintaining user interest, each phase demands a clear focus on the most relevant problems to solve.

Remember, it’s not about measuring everything; it’s about measuring what matters. In the words of Steve Jobs, let’s say no to the noise and yes to what truly drives our products forward.