HEART UX metrics

UX strategy consultant
February to April 2018
DeliveryMuch

Implementing Google’s UX metrics framework at a food delivery startup



DeliveryMuch is a food delivery company with over 100 million users in Brazil — and zero designers in their team at the time of this project. Like many growing startups, Deliverymuch has pains to incorporate UX into their product development process, which is driven mainly by business and engineering needs.

DeliveryMuch hired me to work with their Product Manager Renato Peixe to establish ground rules before building their design team. After some background research, we chose Google’s HEART metrics framework to structure our project.

Although this framework is widely used and validated at a large tech company, many adaptations had to be made in order for it to work in the context of DeliveryMuch.


What is the HEART framework?

HEART is an acronym for Happiness, Engagement, Adoption, Retention and Task success. It was created by Kerry Rodden, Hilary Hutchinson and Xin Fu of Google’s research team to measure, who realized that they lacked a simple and objective framework to measure and improve user experience.


Metric
Definition
Examples

Happiness
How satisfied users are with the product
Measures of user attitudes and feedback, often collected through surveys. 
Satisfaction
Ease of use
Net-Promoter Score
Engagement
How much the users are using the product
Level of user engagement, usually measured via analytics and user-triggered events.
Visits per week
Uploads per day
Share count
Adoption How likely the users are to adopt a product after signing up for it
Measures of how well the company is reaching new users for a product or feature. 
Updates for new versions
New accounts created
Purchases made by users
Retention How many users stick to the product 
One of the most critical features for startups is whether users are returning and using the product again. Keeping users satisfied is hard work!
Active user number
Return rates
Recurring purchases
Task Success How much time users take complete specific tasks with the product
Related to efficiency, effectiveness, and failure rate.
Time to upload a photo
Time to create a full profile





How we implemented HEART


Renato and I set up weekly meetings. I would consult with my contacts in US companies and read articles before meeting Renato. Renato consulted with stakeholders and users on site, to hear what they cared about and how to craft new or reuse existing metrics.

Early on, we decided to focus on a subset of Deliverymuch users for our project. Deliverymuch has 3 user groups: customers who use their app to order food; restaurant workers who receive orders and deliver food to customers, and franchise managers who onboard restaurants and run marketing campaigns to expand their network. To pilot the HEART metrics, we focused on end customers.

These were the metrics that we created:

Happiness. We want users to enjoy the experience of ordering food. We decided to ask the following question right after users place an order: Please indicate how much you agree with the following statement: Ordering from our app is easy. Answers are on a 7-point Likert scale.

This question should be continuously shown to 5% of users, chosen randomly. Users who answer to this survey should not get any request to participate in user research for 6 months.

Aside from the quantitative signal above, we decided to systematically track qualitative feedback from app stores (Apple’s app store and Google Play).

Engagement. Level of user engagement as the frequency of use over a given period. We want users to order more often, so we created a metric to track increases or decreases in the average number of orders from active users in a month.

Adoption. We want the app to be standard on people’s smartphones, just like WhatsApp is in Brazil. Therefore, we defined this metric as the population of cities where we have franchises divided by the number of downloads in app stores (App store + Google Play) in that location. We also established that cohorts of new users should be tracked to analyze retention after 1 month of the app download.

Task success. As this category is most applicable to specific actions, we selected the most common activity performed by customers on the app: ordering food. We want users who access the app to place their orders quickly. Therefore, we proposed tracking the time between starting the app and finalizing an order.


Extending HEART


Of course, Deliverymuch is a rather different company than Google. Certain adaptations were necessary, especially considering that the entire company still struggled with data-driven design.

We noticed the opportunity to bring engineers and business partners onboard, creating more sinergy between different areas of the company. We decided to incorporate these 3 areas in our UX metrics framework:


1. Engineering quality
We decided to systematically track and move key product metrics such as:
  • Active applications time
  • Page loading time
  • Number of error pages on the website
  • Number of mobile app crashes

2. Business goals
Here we proposed tracking and designing for specific transactional metrics related to upselling order:
  • Average Order in Reais (R$)
  • Percentage of Visits / Orders

3. Partnering
The goal of partnering is to increase the involvement of the team with users. We proposed measuring the face-to-face amount of time that product team members spend with users (baseline being 1 hour per month).
Product team members would have to fill out a timesheet describing when, how long did the meeting with a user last for, what topics they discussed, and what learning came out of the meeting.

We noticed a trend —somewhat common in tech startups — of product team members having conversations blaming users for “not knowing how to use the app.” Obviously, we want to encourage designers and developers to empathize with customers which is why we created the Partnering metrics.




Conclusion

Establishing UX metrics can help entire organizations to focus on what really matters.

In this project, I learned that aside from measuring user experience quality for digital products, the HEART and other metrics frameworks can be extended to help nascent organizations to be more customer-centric — not only in their design efforts, but also in engineering and sales.












© 2019 Lucas Colusso.
Built with 💜in Seattle.