Framework for Success Metrics Questions | Facebook Groups Success Metrics

As I geared up for my Product Data Scientist interviews, I scoured the web for tips and frameworks on handling the “Success Metrics” interview question. Despite finding bits and pieces, a complete, end-to-end guide was missing. That’s why I’m excited to share the ultimate framework I crafted during my preparation, which landed me an offer from Meta! Dive in, and I hope it will work for you too!

Framework — Assume you are part of the DS team for Facebook Groups, how would you define success metrics?

Clarifying Questions — Always start by asking clarifying questions. Make sure that you flesh out each and every word of the question and most importantly the product in scope. If you don’t ask any questions, that’s definitely a red flag, so please do!

Let me ask a few clarifying questions. Are we taking about the Groups in the Facebook core app? Is my understanding correct that Groups can be private or public?

Once the question is clear, then take a big breath and start your answer by taking a big detour. Yes, you read that right — don’t just jump into answering the question. It is paramount, and most importantly expected, to talk about the product, the mission of the company and how these two tie together before anything else. So please make sure that you talk about the points below.

Start your answer by taking a big detour — don’t just jump into answering the question right away but talk about the product, the mission of the company and how these two tie together before anything else.

Company’s Mission

First of all, Meta’s mission is to bring people closer together and give them the power to build communities.

Goal of the Product + how it ties to Company’s Mission

FB Groups goal is to bring people with common interests closer together and it is a very important product of Facebook since its goal ties with the overall mission of Meta to bring people closer together.

Users — Always talk about the users. Almost for every product there are 2 sides of users: the producers and the consumers. It is very important to talk about the user journey of both and even more important to give metrics later in your answer that cover both sides.

For FB Groups we have two sides of users, the admins of the groups who are the producers and the members of the groups that are the consumers.

The admins create a group, decide whether it will be pubic/private, send invitation for users to join, post links/media/info and get the conversation started in the group.

The members either see on feed, search or get invited to a group (depending on if it’s public or private) and once joined, they can engage with the group through posting, commenting, liking, sharing etc.

Benefits+Costs (both for the Users and Company) — before jumping into the metrics is good to briefly talk about the benefits and costs of the product both for the users and the company.

One of the main benefits of FB Groups is that users with common interests can get together, which ties with Meta’s mission and contributes to FB app overall engagement. FB Groups also allow Meta to get a better understanding of the users’ interests by looking at the groups they are members of, which in turn can help in making better recommendations and more engaging feed for the users.

On the flip side, FB Groups could potentially make users to engage less with FB’s Newsfeed, which is the “heart” of the FB app and the place where revenue is generated through ads. Another potential con is when groups do not get sufficient members or engagement, which could discourage the admins and create “empty shell” groups.

Types of Metrics to focus on — Now it’s time to start taking about metrics. Choose 2 out of Acquisition/Activation/Retention/Engagement/Monetization to focus on.

Now going to metrics, since FB Groups is a mature product, I believe it makes sense to focus on Engagement + Retention.

Mention Company’s NSM (North Star Metric) + reporting numbers — in order to choose the primary metric, it’s important to have the NSM and reporting numbers at the back of our mind.

Before jumping into the success metrics of the product, let me quickly talk about Meta’s NSM, which is the number of sessions per user per day. On top of that, Meta reports to Wall Street DAUs and MAUS. Hence when we talk about success metrics, and specifically when picking the primary metric, it’s important to keep the above in mind.

Metrics — Now it’s finally time to give the metrics. We’ll give metrics from the two areas we mentioned above we’d focus on and it’s important to not over do it — 2 or 3 to-the-point metrics per area are more than enough. Please note how we make sure to have metrics that cover both the producer and consumer sides of users.

It’s important to have metrics that cover both the producer and consumer sides of users.

Engagement:

# of groups created per week with at least 3 members per user

# of interaction within Groups per user per week

# of sessions that involved Groups per user per week

Retention:

# of active days per user within the past 7 days (Active = used FB Groups)

2nd week retention = # of users that were active at least once a week for 2 consecutive weeks / # of users active only the first week

** I chose the week as timeframe since I believe FB Groups is not meant to be used daily.

Choose Driver/Secondary/Guardrail metrics — Now it’s time to choose the primary, secondary and guardrail metrics from our list of metrics above. Don’t forget to talk about the trade-offs!

Driver Metric:

# of sessions that involved group per week

[# of sessions is easy to explain, captures users behavior and ties with Meta’s NSM. If sessions involving groups are up then more interaction among users → creating communities.]

Secondary Metrics:

# of groups created per week with at least 3 members per user (to capture supply)

# of interaction within Groups per user per week (to capture demand)

# of users active 2 weeks in a row (this + last week) / # of users active last week (2nd week retention)

Guardrail Metrics:

# of users removed from groups

# of groups shut down or reported

# of offensive/inappropriate posts posted on Groups per week

time spent on Newsfeed (we don’t want users to stop using Newsfeed as much)

Wrapping up — Always a good idea to quickly go through the story you just put together and show how/why it answers the question.

So that’s it — this is the framework that worked for me and helped me land an offer from Meta! The same framework can be used for any Product/Success Metrics questions asked for Data Science positions. Having a well-structured response that thoroughly scopes out all the important components and takes the interviewer along your thought process is key. Hope you enjoyed it and would love to hear your feedback in the comments below!