In early 2014, we found ourselves working for an organization with a large product portfolio built up over decades of acquisitions. To create a cohesive branded experience, it became clear that many of the products we were suddenly supporting existed at very different levels of UX maturity. Our challenge was to create a comparable scorecard for an array of products across UX frameworks, usability maturity, and technology platforms. Our first, somewhat obvious step, was to leverage existing UX maturity models to build a custom framework for our organization, following the advice of Rich Buttiglieri in his presentation on maturity models at UXPA Boston 2012. The model we built focused on the following for each level:

• What will the user see and experience?
• How will the experience impact brand perception?
• What level of UX process support is needed?
• How would product management, as a whole, react towards the framework proposed?

Table 1. Measures of Experiential Maturity

We began introducing our initial experience maturity model to leadership, both within the product organization and throughout IT. This proved invaluable in helping our internal stakeholders understand why our processes were a little different for each product. It also served as a communication tool to help our UXers understand how to relate to internal product management teams.

In addition to introducing our maturity model to key stakeholders, we also needed to provide senior leadership with an executive summary so they could make comparisons across products. This would allow them to gain insight across a large portfolio of varying user experiences and to identify where to prioritize UX in the organization.

We experimented with multiple methods to create this type of summary for products and kept returning to the idea of a UX debt scorecard. What if we could combine the concept of UX debt with our experiential maturity model? This could not only help us communicate across the organization on a level playing field, but would also provide us with a way to measure how products were decreasing their debt, thereby increasing their experience maturity.

In this article, we discuss UX debt as a way to measure how easy and intuitive a product is to use so we can use the UX debt score as a way to determine the organization’s level of experiential maturity. For example: when the Ad-Hoc UX Debt=0, the organization is ready move to Intentional UX. The same formula works for the move from each level, all the way to a UX-driven organization.

We are now calculating UX debt relative to the product’s current maturity level to provide senior leadership with a product-by-product comparative dashboard of UX debt specific to a given product’s maturity level.

Due to the organization’s numerous, diverse products, we created a calculation worksheet meant to consistently calculate UX debt across our portfolio. As we began to communicate this concept throughout the organization, it was apparent that this was an effective communication and planning tool across multiple stakeholders: sales, product teams, technology teams, and senior leadership.

Defining UX Debt

The term “technical debt” was coined by Ward Cunningham as a metaphor used to describe the increased cost of maintaining technology due to shortcuts taken during the development process. Joshua Kerievsky later extended the metaphor to user experience design using the term “User Experience (UX) Debt.” Kerievsky explained that, like technical debt, UX debt will eventually come due, usually in the form of less customer satisfaction and possible customer defects.

In an enterprise setting, studies show that the decision to accumulate technical debt is often made by non-technical stakeholders. In our current enterprise, we wondered if the same is true for UX debt? If we develop a practical way to measure UX debt for products in the enterprise, what would that look like and how would it be received?

From Theory to Practice

We were brought into a company that had previously tried to ”perform UX,” but teams were unable to make much visible progress across multiple diverse, distinct product lines. Non-UXers (largely product managers) were often in charge of making UX decisions. Across this wide variety of products, customer satisfaction scores were showing that user experience, along with a large UX debt, was hindering products’ usability and creating unhappy customers in the process.

How could a small UX team of five people begin to describe at a high level how more than 100 products compared to each other? How could we communicate to senior managers the amount of UX work that we might need to perform to improve them? We quickly shifted our conversations to whether the notion of UX debt could help not only prioritize our UX work, but could pave the way to communicate and develop a more data and user-centric approach to product development at an enterprise level. Our team was familiar with the term UX debt from a theoretical perspective, but we soon began to wonder if we could actually create a long-term UX debt calculation and track it over time for our products. We wanted to create a comparable scorecard across product lines, and to help explain to our senior leadership how much work needs to be done. This was the point where we began to develop the UX debt calculator spreadsheet.

Table 3. A summary of UX Debt by Maturity Level

The UX Debt Calculator

UX debt can be calculated with data from either heuristic evaluations or prior usability testing.

Calculating UX debt with heuristic data

We noticed that many products have little or no formative usability data available; products are mostly at the Ad-Hoc level of maturity. To calculate UX debt in these instances, we use heuristic analysis. The UX team performs a heuristic evaluation of the product using the following categories: Findable, Accessible, Clear & Communicative, Useful, Credible, Controllable, Learnable, Overall Aesthetics, Persuasive, and whether the product is currently tracked using our web analytics tool.

In the Template-Heuristics tab in the calculator, the categories correspond to a heuristic. As we complete the evaluation of a product or service, we update the value of the Assessment field for each measure. This field controls the Actual Score and Score Percentage. Possible values for the measure include:

• Unusable: imperative to fix – Unusable
• Major issue: important to fix with high priority – Critical
• Minor issue: should be given low priority – Moderate
• Cosmetic issue only – Minor
• Meets Criteria No Issues (default value)

If the product has failed to take accessibility into account, we assign a severity of Major issue. In addition, if we do not currently track the product’s usage, do not complete formative and/or summative testing, and do not implement company branding, we assign a severity of Major issue. This helps drive a culture making product decisions based on user data and beginning to help “paydown” UX debt.

We use the fields Problems and Steps to Resolve to communicate what needs to be addressed in the product for it to move up in its UX maturity. Once all columns are completed, we calculate the current UX debt and any projected paydown.

Calculating debt with prior usability data

In our organization, usability tests were previously conducted on a few products at higher levels of UX maturity. Our initial proof of concept for our UX debt calculator was to translate these third-party studies into a standard debt score. This approach quickly became our preferred approach to calculating debt, as it takes direct user feedback into account. However, when there is no formative data (as was the case with many of our products at lower levels of UX maturity), we default to heuristic evaluation to calculate UX debt.

Figure 1. The UX debt calculator is a spreadsheet with measures and scores

Next Steps

This first phase of the UX debt calculator has been useful in helping us assess products. As a small UX team intending to create a branded experience across over 100 products, we must first determine where that product lies in our experiential maturity model and, second, calculate a debt score. As we began to use this calculator for many of our products, we discovered where our gaps lie in terms of user data, user experience, and customer satisfaction. From this vantage point, we can help product teams move the product forward in its UX maturity, as well as lower its long-standing UX debt.

As we move forward with these concepts across our organization, we expect that our company’s products will advance in their UX maturity and, consequently, pay down their debt. The calculator is a great tool for modeling this CX/UX transition in a product and reinforcing a user- and data-centric product culture.

Download the UX debt calculator

Topics: Usability Testing, Management, UX Methods
Published in: February, 2015 in Open Topics

Comments are closed.