Preparing for a Root Cause Analysis Interview: A Framework

A common interview step for an aspiring Product Manager is the Root Cause Analysis. The goal of this article is to give you the basics of a product management root cause analysis and a framework for acing the interview.

What is a root cause analysis interview?
A root cause analysis is a framework for discovering the most likely cause (or causes) of a problem in a software experience. It allows you to play real-time digital detective in pursuit of the answers to a struggling product.

A root cause interview will usually last 30 to 60 minutes in which the interviewer will present you with a hypothetical scenario and ask you to diagnose the issue and resolve it. Note that in this interview, unlike a Product Sense interview, there is a “right” answer to the problem. By asking questions, you are deducing your way to that answer.

Scenarios include prompts like:

• We noticed spend is down 20% this week. How would you find out what is going on?
• Order cancellations on Amazon are up 10%. Why is this happening?
• Facebook Groups are seeing a 30% decrease in usage. What’s the problem?

A root cause analysis is commonly part of a product manager interview because:

1. It’s a part of a Product Manager’s everyday job.
2. The exercise is a great way for an interviewer to assess how you think and work your way through a problem.

This type of interview is a conversational, real-time attempt for the interviewer to understand your process as a product manager.

What is the interviewer listening for?

The interviewer wants to understand how you approach ill-defined problems without a clear answer. .

They want to find out:

• Can you approach an ambiguous problem in an organized and efficient manner?
• How do you gather information?
• How do you prioritize the information you want to gather?
• Once you have the information, how do you determine what is valuable and what is not?
• How do you measure your success?

Some Root Cause Analysis tips:

• Take Notes: Bring a notebook and a pen to take notes with throughout the interview. There will be a lot of information to sift through — don’t try to keep it all straight in your head.
  And yes, you could use your computer if you prefer — but I find it distracting for myself and the interviewer. .
• Talk Out Loud: The purpose of the interview is to give insight into your process — so guide the interviewer through your process with your words. (Also, reasoning out loud almost always produces better results).
• Don’t Be Afraid to Ask for Clarification: In a root cause analysis you will ask a lot of questions. But if you don’t understand a concept or a term, be sure to ask the interviewer. Don’t muddle your way through. There is no shame in asking questions.

Framework for Root Cause

Whether it’s in an interview or real-life, I tend to tackle a root cause problem with five basic steps:

1. Define the Problem
2. Identify Possible External Factors
3. Identify Possible Internal Problems
4. Zero in on a cause (or causes) and suggest a fix (fixes)
5. Measure the “success” of the fix

Step 1: Define the problem
Before you can solve a problem you have to know what the problem is. In this step you will want to ask your interviewer clarifying questions to narrow down the problem as much as possible. I try to look at the prompt and ensure I understand every part of the scenario.

For example, in the scenario “We noticed spend is down 20% this week. How would you find out what is going on?”, I would start out defining each of the parts of the stated problem. Listen closely and take notes on the interviewer’s response.

• When is this scenario taking place?
  Are there holidays or other calendar-based events that could explain differences?
• How long has this problem been going on? You mentioned spend being down 20% this week. Was there a decrease from baseline spend during previous weeks?
  The problem could have existed longer than the initial prompt let on. Something likely changed to cause this problem. By defining the timeline we can better focus on possible causes.
• When you say 20% this week — is that 20% from last week? Or 20% from average? What is that 20% drop based on?
  Again, defining the language of the prompt is important. A 20% drop in spend may be less dire if we understand its a 20% drop from record spend driven by a huge sale.
• Is this decrease platform agnostic? Are we seeing the same drop across web and mobile?
  Another attempt at narrowing the focus of our inquiry.
• If the problem is focused on mobile-only, is there a difference between ecosystems — ie Apple vs Android?
  Further attempts to narrow down the problem.
• Are we seeing any other corresponding dips in key metrics? Spend is down, but are daily user counts down? Are people adding less items to their carts?
  Corresponding data points can often narrow down a problem.

I usually end this step by asking if the presented information in the prompt is reliable. I know this may sound silly, but it’s an important step in the real-life product management playbook so I always include it here. Analytics tools are made by humans and can break and/or be fooled.

• Are we sure the analytics tools are behaving as expected?
• Can we confirm the data with any other data source?

At this point, if you have a hunch or two you might consider voicing them while still moving on with your analysis. I usually go with something like “Well, spend is down 20% from the baseline across the product regardless of platform, but the average number of items being added to the carts remains the same. This sounds like maybe a payment issue?”.

Step 2: Identify Possible External Factors:
Sometimes your product doesn’t do well because of issues not directly related to your handling of the product. In this step we want to work through possible external factors that could lead to the prompt.

• What have our competitors been doing? Have they launched something new that might account for the change in spend on our platform?
  Sometimes your competitors eat your lunch.
• And are we seeing this problem located to any specific area of the world or the country?
  Example: If there are major power outages in a region, it might account for less people signing on to a digital experience.
• Do we have any specific group of users that have been affected most?
  New Users vs Established Users?
  Women?
  Men?
  Users utilizing a specific app version?
  Again, in order to narrow down possible causes you’ll want to narrow down possible affects.
• Are there any external events that could be causing changes in how our app is used?
  Cell network downages?
  Third-party partners experiencing issues?

Finding that your issue is an external cause is always both a relief and a challenge. On one hand, I am always happy when the problem is not “our” fault. On the other hand external factors mean that the solution to the problem is largely out of your control.

Step 3: Identify Possible Internal Factors
With our problem defined and external forces considered, it’s time to turn the gaze inward. What have we as a company done that could cause this problem?

Bugs and other problems usually don’t appear out of nowhere — so I start by concentrating on the most recent events.

• Have any internal changes been made?
• When was our last release?
• What features or changes were included in that last release?
• Have we changed how we interact with any of our third party providers?
• New integrations?
• Updated integrations with third party providers?

From here I will also look more closely at the problem in a step-by-step manner. For example, we know there is an issue in overall spend being down — but is that where the problem starts? Look at each step in the product experience and look at the corresponding metrics to see where outliers occur.

Step 4: Zero in on Cause and Suggest a Fix
We’ve gathered our information, now is the time to form our hypothesis regarding the root cause of the problem. This is where your notes will come in handy — I like to review the process and guide my interviewer through my line of thinking.

“So we know that spend is down 20% over baseline across all the platforms — web and mobile.”
“This is unique to the last week.”
“Our customers seem to be adding items to the cart at the same rate prior to the downturn.”
“There are no identified external factors — such as holiday slumps — that would explain this decrease in spend…”
“And our third party partners have not reported any known issues that would affect our pipeline…”
“However, with the latest release we did change our integration with our credit card payment processor.”
“Based on what we know, I would start by investigating our integration with our credit card payment processor — our user’s behavior is staying the same, but the expected outcome is off. We should alert our partner that we believe this is an issue and ask them to investigate on their end, while they do that we should ensure the fault is not on our side.”

And please note: depending on the problem you are given you may arrive at a few possible hypotheses and solutions. Share them all.

Step 5: Measure “Success” of the Fix
Once you’ve given your hypothesis and suggested fix it is important to share how you will confirm whether or not the fix is a success.

In the example we’ve been using, the team would likely concentrate on spend metrics ensuring this number returns to its previous baseline.

Summary
A root cause analysis can seem daunting to the uninitiated but it is easy enough to ace when you have a framework. But note that there is no one right way to tackle these problems.

I suggest watching some mock interviews on YouTube and reading up on other frameworks and example prompts. The more familiar you are with this exercise the more comfortable you will be the day of the interview!

Good luck!