HEART UX metrics for startups


Implementing Google’s UX metrics framework at a Brazilian food delivery startup



Role
UX strategy consultant
Client
DeliveryMuch Brazil
Duration
2 months
February to April 2018
Deliverables
UX metrics framework
Knowledge Management System — WikiMedia





Like many growing startups, Deliverymuch has pains to incorporate UX into their product development process, driven mainly by business and engineering needs. Deliverymuch is a food delivery company with over 100 million users in Brazil — and zero designers on their team.

Deliverymuch hired me to work with their only Product Manager (shout out to Renato Peixe) to establish ground rules before building their design team.

After some background research, we chose Google’s HEART metrics framework to structure our project. Although complicated at first sight, HEART is an industry-validated framework.

However, how to establish UX metrics that work for the entire company, and foremost, help to deliver better products and services for customers?



What is the HEART framework?


HEART is the acronym for Happiness, Engagement, Adoption, Retention and Task success. It was created by Kerry Rodden, Hilary Hutchinson and Xin Fu of Google’s research team to measure user experience quality for digital products. Google researchers realized that they lacked a simple and objective framework following rules and standards to find insights on how to improve user experience.

Happiness is used to measure how satisfied users are with the product. Engagement is used to measure how much the users are using the product. Adoption is used to measure how likely the users are to adopt a product after signing up for it. Retention is used to measure how many users stick to the product. Task success is used to measure how much time users take complete specific tasks with the product. See the breakdown below.

Metric
Definition
Example

Happiness
Measures of user attitudes, feedback, and resolutions, often collected through surveys. For example, how are users dealing with changes made to the product? Are they satisfied or disapproved?
Satisfaction
Ease of use
Net-Promoter Score
Engagement
Level of user engagement, usually, this data comes from analytics and user-triggered events.
Number of visits/week
Number of uploads/day
Number of Shares
Adoption
Reaching new users for the product or feature. How many users are using the new versions or registering for a feature? This is a great metric to understand how the public is responding to the news.
Updates for new versions
New signatures created
Purchases made by users
Retention
Rate of return of registered users. One of the most critical features for startups is whether users are returning and using the product again. Keeping a satisfied user is hard work. 
Active user number
Renewal or withdrawal fee
Recurring purchases
Task Success
It is related to efficiency, effectiveness, and failure rate. “Performance matters,” this phrase summarizes the importance of quick interactions. 
Success in return search
Time to upload a photo
Time to create a full profile



How did we implement HEART?


First, Renato and I set up weekly meetings. I would consult with my contacts in US companies and read articles before meeting Renato. Renato consulted with stakeholders and users on site, to hear what they cared about and how to craft contextual metrics or survey questions.

Early on, we decided to focus on a subset of users in our project. Deliverymuch has 3 main groups of users: customers, those who use their app to order food; restaurant workers, who get orders and deliver food to customers, and franchise managers, who onboard restaurants and run marketing campaigns to expand the network. To pilot the HEART metrics, we focused on end customers in this project.

These were the metrics that we created:

Happiness. We want users to enjoy the experience of ordering food. We decided to ask the following question right after users place an order: Please indicate how much you agree with the following statement: Ordering from our app is easy. Answers are on a 7-point Likert scale.

This question should be continuously shown to 5% of users, chosen randomly. Users who answer to this survey should not get any request to participate in user research for 6 months.

Aside from the quantitative signal above, we decided to systematically track qualitative feedback from app stores (Apple’s app store and Google Play).

Engagement. Level of user engagement as the frequency of use over a given period. We want users to order more often, so we created a metric to track increases or decreases in the average number of orders from active users in a month.

Adoption. We want the app to be standard on people’s smartphones, just like WhatsApp is in Brazil. Therefore, we defined this metric as the population of cities where we have franchises divided by the number of downloads in app stores (App store + Google Play) in that location. We also established that cohorts of new users should be tracked to analyze retention after 1 month of the app download.

Task success. As this category is most applicable to specific actions, we selected the most common activity performed by customers on the app: ordering food. We want users who access the app to place their orders quickly.
Therefore, we proposed tracking the time between starting the app and finalizing an order.

Read this paper to learn more about crafting metrics using the Goal—Signals—Metrics trio.



A rising challenge was to help establish a data-driven culture at the company. For example, not even the engineering team had experience with data-driven development. Noticing the opportunity to bring engineers and business partners onboard, we decided to incorporate more engineering quality metrics to the Task success category. We also created another category of metrics related to overall Business goals:

Task success appendix. We decided to track and create projects to move metrics such as Active applications time, Page loading time, Number of error pages on the website, Number of mobile app crashes.

Regarding the Business side, there is the obvious goal of upselling orders whenever possible. Here we started tracking and designing for purely transactional metrics such as Average Order in Reais (R$), and Percentage Visits vs. Orders.



Interestingly, we decided to amend the HEART framework with an indirect organizational metric to impact the user experience. We noticed a trend —somewhat common in tech startups — of engineering team members having conversations blaming users for “not knowing how to use the app.” Deliverymuch wanted employees to develop empathy with their users, so we created another metric category called Partnering.

The goal of Partnering is to increase the involvement of the team with users. We proposed measuring the face-to-face amount of time that the product team spends with users. We established the minimal cutoff in 1 hour per month. Product team members have to fill out a type of timesheet, describing when, how long did the meeting with a user last for, the topics they discussed, and what they learned from their meeting.



A rising challenge was the dissemination of the UX metrics into the organization’s day to day operations. To disseminate the results of this project, I created a knowledge-base using Wikimedia’s MediaWiki project. In the shared knowledge-base, teams can access, edit, or add new resources. I believe that the establishment of the knowledge-base was a contribution in itself that demands another post!

HEART is an efficient framework to help product teams to identify and establish user experience metrics. Engaging multiple stakeholders in the planning and execution process can help the entire organization to focus on what really matters.

In this project, I learned that aside from its primary goal of measuring user experience quality for digital products, the HEART framework can also help young organizations to be more customer-centric in different areas, from business to engineering and design.



Read more about HEART in this excellent paper published by Google’s researchers Kerry Rodden, Hilary Hutchinson, and Xin Fu.




More projects