HEART UX metrics
UX strategy consultantFebruary to April 2018
DeliveryMuch
Implementing Google’s UX metrics framework at a food delivery startup
DeliveryMuch is a food delivery company with over 100 million users in Brazil — and zero designers in their team at the time of this project. Like many growing startups, DeliveryMuch has pains to incorporate UX into their product development process, which is driven mainly by business and engineering needs.
DeliveryMuch hired me to work with their Product Manager Renato Peixe to establish ground rules before building their design team. After some background research, we chose Google’s HEART metrics framework to structure our project.
Although this framework is widely used and validated at a large tech company, many adaptations had to be made in order for it to work in the context of DeliveryMuch.
DeliveryMuch hired me to work with their Product Manager Renato Peixe to establish ground rules before building their design team. After some background research, we chose Google’s HEART metrics framework to structure our project.
Although this framework is widely used and validated at a large tech company, many adaptations had to be made in order for it to work in the context of DeliveryMuch.
What is the HEART framework?
HEART is an acronym for Happiness, Engagement, Adoption, Retention and Task success. It was created by Google’s Kerry Rodden, Hilary Hutchinson and Xin Fu who were tasked to create a simple and objective framework to measure and improve user experience.Metric
Definition
Examples
Happiness
How satisfied users are with the product
How satisfied users are with the product
Measures of user attitudes and feedback, often collected through surveys.
Satisfaction
Ease of use
Net-Promoter Score
Ease of use
Net-Promoter Score
Engagement
How much the users are using the product
How much the users are using the product
Level of user engagement, usually measured via analytics and user-triggered events.
Visits per week
Uploads per day
Share count
Uploads per day
Share count
Adoption
How likely the users are to adopt a product after signing up for it
How likely the users are to adopt a product after signing up for it
Measures of how well the company is reaching new users for a product or feature.
Updates for new versions
New accounts created
Purchases made by users
New accounts created
Purchases made by users
Retention
How many users stick to the product
How many users stick to the product
One of the most critical features for startups is whether users are returning and using the product again. Keeping users satisfied is hard work!
Active user number
Return rates
Recurring purchases
Return rates
Recurring purchases
Task Success
How much time users take complete specific tasks with the product
How much time users take complete specific tasks with the product
Related to efficiency, effectiveness, and failure rate.
Time to upload a photo
Time to create a full profile
Time to create a full profile
How we implemented HEART
Renato and I set up weekly meetings. I would consult with my contacts in US companies and read articles before meeting Renato. Renato consulted with stakeholders and users on site, to hear what they cared about and how to craft new metrics or reuse existing metrics.
Early on, we decided to focus on a subset of DeliveryMuch users for our project.
DeliveryMuch has 3 user groups: customers who use the app to order food; restaurant workers who receive orders and deliver food to customers; and franchise managers who onboard restaurants and run marketing campaigns to expand their network.
To pilot the HEART metrics, we focused on end customers.
These were the metrics that we created:
Happiness. We want users to enjoy the experience of ordering food. We decided to ask the following question right after users place an order:
- Please indicate how much you agree with the following statement: Ordering from our app is easy. Answers are on a 7-point Likert scale.
This question should be continuously shown to 5% of users, chosen randomly. Users who answer to this survey should not get any request to participate in user research for 6 months.
Aside from the quantitative signal above, we decided to systematically track qualitative feedback from app stores (Apple’s app store and Google Play).
Engagement. Level of user engagement as the frequency of use over a given period. They want users to order more often, so we created a metric to track increases or decreases in the average number of orders from active users in a month.
Adoption. DeliveryMuch wants the app to be installed on every single smartphone in Brazil. Therefore, we defined the metric as a) the population of cities where DeliveryMuch has franchises, which we divided by b) the number of downloads in app stores (App store + Google Play) in that location. We also established that cohorts of new users should be tracked to analyze retention after 1 month of the app download.
Task success. As this category is most applicable to specific actions, we selected the most common activity performed by customers on the app: ordering food. We want users who access the app to place their orders quickly. Therefore, we would track the time between starting the app and finalizing an order.
Extending HEART
Of course, Deliverymuch is a rather different company than Google. Certain adaptations were necessary, especially considering that the entire company still struggled with data-driven design.
We noticed the opportunity to bring engineers and business partners onboard, creating more sinergy between different areas of the company. We decided to incorporate these 3 areas in our UX metrics framework:
1. Engineering quality
We decided to systematically track and move key product metrics such as:
- Active application time
- Page loading time
- Number of mobile app crashes
2. Business goals
Here we proposed tracking and designing for specific transactional metrics related to upselling order:
- Average Order in Reais (R$)
- Percentage of Visits / Orders
3. Partnering
The goal of partnering is to increase the involvement of the team with users. We proposed measuring the face-to-face amount of time that product team members spend with users (baseline being 1 hour per month).
Product team members would have to fill out a timesheet describing when, how long did the meeting with a user last for, what topics they discussed, and what learning came out of the meeting.
We noticed a trend —somewhat common in tech startups — of product team members having conversations blaming users for “not knowing how to use the app.” Obviously, we want to encourage designers and developers to empathize with customers which is why we created the Partnering metric.
Conclusion
Establishing UX metrics can help entire organizations to focus on what really matters, which is to deliver value to users.
In this project, I learned that aside from measuring user experience quality for digital products, metrics frameworks can be extended to help nascent organizations to be more user-centric — not only in their design efforts, but also in engineering and sales.
In this project, I learned that aside from measuring user experience quality for digital products, metrics frameworks can be extended to help nascent organizations to be more user-centric — not only in their design efforts, but also in engineering and sales.