This set of notes is distilled from NN/g's *UX Metrics and ROI* report + a real-world case library spanning 5 editions, and answers only one question that PMs are most often pressed by their bosses to answer:"You say the experience has improved — how much is that worth?"
Its answer has three layers, which correspond exactly to the three main sections of this article:
- Metric system checklist: There are only a few categories of things UX can quantify (how effortful it is to complete tasks, how satisfied users are, whether they come back, whether they convert). For each category, what off-the-shelf metrics exist, how to calculate them, when to use them, and where the pitfalls are — you need a "menu" first before you can talk about selection.
- UX → ROI conversion methodology: Translating "time on task decreased by 14 seconds" into "saves 1.5 million yuan a year" is essentially unit conversion. It provides you with the improvement score formula + the three-step method + monetary conversion recipes for four types of business models + a correlation model that connects soft metrics to revenue.
- Expectation management + reporting talking points: How much UX redesigns improve on average (75%), why it is getting harder to deliver lifts now (shrinking from 247% to 75%), and how to report to bosses in a credible way that does not get you in trouble.
One core principle: ROI is not a financial forecast; it is a directional indicator— "Once you figure out how much money is saved and how much is spent, the boss will naturally understand." Do not tie yourself in knots chasing precision; telling a relatable story about money matters more than decimal points.
Division of labor with NNGroup Analytics & Metrics (choosing the right metrics / conversion funnels / statistical significance and confidence intervals / four-step UX ROI / sampling and confounders / triangulation / vanity metrics / benchmarking): That piece presents NN/g's "analytics and measurement perspective" (choosing the right metrics, conversion funnels, statistical significance and confidence intervals, the four-step UX ROI principle, sampling and confounders, triangulation, vanity metrics, benchmarking); this piece is a "complete metric system checklist + improvement score math + multi-business-model monetary conversion recipes + empirical baseline magnitude data + reporting talking points". Content covered in both (ROI steps, HEART, confidence intervals) is only cross-referenced here and not rewritten.: That article covers NN/g's "analytical measurement perspective" (vanity metrics, conversion rate quirks, confidence intervals, statistical significance, sampling, triangulation, and the four-step principle of ROI); this article covers "a complete set of metric lists + the mathematics of score improvement + money conversion formulas for multiple business formats + empirical evidence of benchmark ranges + presentation talking points". Content covered in both articles (ROI steps, HEART, confidence intervals) is only cross-referenced here and not rewritten.
1. Why PMs must force themselves to quantify UX (the source of persuasiveness)
Why: Qualitative research (interviews, stories, verbatim user quotes) has always been the mainstream of UX, because it can explain "what the problem is and how to fix it". But it has a fatal shortcoming —it cannot articulate "scale": How many users does this problem affect? How severe is it? Should it be prioritized? Only numbers can answer these questions.
The real value of quantitative data for PMs is not that it is "more scientific", but that it can persuade the people who control the budget:
| What quantitative data helps you do | Why it is critical for PMs |
|---|---|
| Determine the scale of the problem (how many users are affected) | Decide whether to schedule it in this iteration and how high its priority should be |
| Provide a basis for design experiments such as A/B testing | Turn "I think B is better" into "B increased add-to-cart by 12%" |
| Persuade stakeholders and demonstrate UX value | Most bosses come from business/finance backgrounds and prefer numbers; visualized numbers are easiest to circulate within the organization and report upward |
| Tie UX outcomes to corporate KPIs(revenue / cost) | The most effective way to secure more UX resources and expand the team |
Real quote (Head of Research at HelloFresh): "We made a very small adjustment based on problems found in qualitative testing, which brought a 12% conversion rate lift— for a company of our scale, that is already astonishing. With good results, the company is more willing to invest money; the more projects there are, the larger the investment becomes."
Counterexamples / pitfalls: Do not force in quantitative data just for the sake of "hard data". A UX Writer at Marketade reviewed a project: the quantitative part was purely insisted on by the client,and in the end it was the qualitative interviews that actually delivered insights and were most recognized by the client— "Quantitative data did not bring many insights, but numbers can be made into charts, and clients are really sold on that." So the value of quantitative data is sometimes persuasion rather than discovery; do not confuse the two.
How to use quantitative methods — two timing points(borrowing the formative/summative terminology):
- Formative evaluation: When the project is still in the ideation/prototype stage, "checking as you build whether the direction is right". Qualitative methods are the mainstay, but quantitative methods can assist —A/B testing is a typical formative quantitative method(launch two versions simultaneously and compare conversion rates/click-through rates).
- Summative evaluation: At the end of a design cycle, evaluate "whether this change actually worked". Quantitative methods are the mainstay; collect UX metrics to measure success or failure, and you can compare the new version against the old one or against competitors.Using quantitative methods to measure experience quality in summative evaluation is called UX Benchmarking— see Section 4.
2. Core metric system checklist: what to measure / how to calculate / when to use / limitations
Why: You cannot manage what you do not measure. But the dimensions UX can quantify are actually limited — just those few categories. Lay out the "menu" first, so that when selecting metrics you will not miss anything or use the wrong metric to measure the wrong thing.
2.1 What exactly UX metrics are
A UX metric = a numerical value that reflects one aspect of the user experience. Using coffee as an analogy: "The milk foam looks like a leaf, the cup is very retro" is qualitative(describing feelings); "12 ounces, 120 degrees Fahrenheit, $7.35" is quantitative(quantifying things). UX quantitative research mainly focuses on the following categories of experience details:
| Experience dimension | Corresponding common metrics | One-sentence description |
|---|---|---|
| How much effort/time it takes to complete a task | Time on task | Measures process efficiency |
| How difficult the task feels to the user | Ease-of-use score | Subjective difficulty |
| How many people can complete it successfully | Success rate | The most common usability metric |
| How satisfied users are | Satisfaction score / CSAT / NPS / SUS | Collected via ratings or questionnaires |
| How often users come back | Return visit rate / repeat visits | Reflects stickiness |
| Total number of users | Total user count | Measures popularity (note: an isolated total count easily becomes a vanity metric; see NNGroup Analytics & Metrics (choosing the right metrics / conversion funnels / statistical significance and confidence intervals / four-step UX ROI / sampling and confounders / triangulation / vanity metrics / benchmarking)) |
| What proportion proceeds to the next step of a key flow | Conversion rate / completion rate of each funnel step | Core of funnel analysis |
2.2 Understanding metrics systematically by "metric category" (selection scaffold)
The report groups all UX metrics into 6 major categories —this table is the most practical classification framework for building metric dashboards, writing research reports, and tracking redesigns:
| Metric category | Which aspect of experience it measures | Example metrics |
|---|---|---|
| Adoption / Retention Adoption/Retention | Acquiring new users + retaining existing ones (including traffic visits) | Lead submissions, application form submissions, subscription sign-ups, renewals, new visitors, returning users, users from organic search |
| Effectiveness Effectiveness | Whether users can successfully complete tasks | Success rate, completion rate |
| Efficiency Efficiency | How quickly and smoothly tasks are completed | Time on task, average session duration, tree test score, completion time, number of errors |
| Engagement / Usage Engagement/Usage | How frequently, deeply, and broadly the product is used | Sessions per user, bounce rate, page value, feature usage rate, active user count |
| Revenue Revenue | How much money is actually earned | Monthly revenue, revenue per session, average order value (AOV) |
| Satisfaction / Perception Satisfaction/Perception | What users subjectively think | Ease-of-use score, CSAT, NPS, app store rating, subjective sense of success |
When to use: If the goal is user acquisition and revenue growth → look at adoption/retention + revenue; if the goal is to reduce customer complaints → look at effectiveness/efficiency; if you want to know "are users happy" → you can only look at satisfaction/perception (and this can only be done via questionnaires; neither analytics tools nor operational testing can measure attitudes).
2.3 Standardized questionnaire metrics: SUS / NPS / CSAT — the three easiest to take to reports
These three are "off-the-shelf scales" that PMs can directly adopt and use, comparable across products and across time:
| Metric | Full name / What it is | How to calculate | When to use / Limitations |
|---|---|---|---|
| SUS | System Usability Scale | 10 standard questions, converted to a 0–100 score(note: not a percentage, but an industry-standard scale; ~68 is considered average) | Use when you want a cross-product comparable total usability score; limitation = it only gives a total score and does not tell you specifically what is broken |
| NPS | Net Promoter Score | Ask "How likely are you to recommend (0–10)?",%Promoters (9-10) − %Detractors (0-6), resulting in a score from −100 to +100 | Measures loyalty, easiest to communicate to executives; limitation = a single number, high volatility, does not explain causes |
| CSAT | Customer Satisfaction | Questionnaire rating (e.g., 1–5), take the average value | Measures satisfaction with a specific product/interaction; limitation = subjective, influenced by emotion and timing |
Ease-of-use score / Satisfaction score: Ask users to select on a 1–5 or 1–7 (Likert) scale, then calculate the average. Often asked immediately after a task(rated right after task completion), collecting both "whether it was completed" and "how difficult it felt" together.
2.4 Four major data sources: what questions each answers
Why: For the same metric obtained from different sources, credibility and meaning differ. First recognize "where this number comes from", so you know whether it can be trusted and whether conclusions can be drawn.
| Source | What question it answers | Typical metrics | Limitations |
|---|---|---|---|
| Surveys | What users think(subjective) | Satisfaction, ease of use, NPS, SUS | Self-reported, may not match actual behavior |
| Quantitative usability testing | Whether users can complete tasks and how efficient they are(given assigned tasks) | Time on task, success rate, post-task ease-of-use score | Small sample size (~40 people), expensive, hard to generalize |
| Analytics tools(GA / Adobe / Pendo) | What users actually did(real-world environment) | conversion count/rate, returning users, churn rate, bounce rate, revenue per user | lacks context(knowing what but not why); uncontrollable (competitors’ major promotions will also shift your conversion) |
| customer service + marketing data(non-research channels) | circumstantial evidence of whether the product is easy to use | customer service ticket volume, error report volume, CLV, app store ratings | indirect, requires ruling out other causes |
Tree Testing is a dedicated method for evaluating information architecture (IA, i.e., menu/directory structure): show users only the menu hierarchy without letting them enter pages, give them a task ("I want to register as an employer, where do I click?") and see if they can click the correct option. There are three metrics:overall success rate(found the correct option in the end),direct success rate(got it right on the first try),indirect success rate(got it wrong first then right).
When to use customer service/app store data: they are free and exist naturally; after a redesign, check "did customer service tickets decrease, did app store ratings improve" — this is the easiest entry point to connect UX to costs/growth (higher ratings also bring more downloads).
2.5 Having data ≠ only looking at data: quantitative and qualitative research must cycle back and forth
Why: analyzing data can never tell you "why". The same behavior of "flipping through many pages, staying for a long time" could mean immersive enjoyment, or it could mean getting lost and frustrated (for details see the Greek travel site example in NNGroup Analytics & Metrics (choosing the right metrics/conversion funnels/statistical significance and confidence intervals/four steps of UX ROI/sampling and confounding/triangulation/vanity metrics/benchmarking)).
Standard cadence: first look at data to find clues → then conduct interviews to find causes → keep testing → repeat the cycle. You can start from either direction:
- Qualitative first: For the PetSmart charity site, qualitative testing first found problems in the subscription flow → then Google Analytics was used to quantify that "73% of users abandoned midway" → proving "how severe" the impact scope was.
- Quantitative first: first use analytics to find "high task failure rate" → then interview users to ask exactly where they got stuck.
III. UX → ROI: Converting experience into monetary terms (core methodology of this article)
3.1 First step math: improvement score formulas (positive vs negative metrics)
Why: Reporting "success rate went from 25% to 100%" is less impactful than saying "it improved by 300%". But different metrics have opposite directions of rise and fall, so the formula cannot be applied uniformly; calculating it the wrong way will make you look ridiculous.
First divide metrics into two categories:
- Positive metrics (bigger is better): sales, traffic, satisfaction, success rate, conversion rate —we want these to go up.
- Negative metrics (smaller is better): error count, customer service tickets, task completion time (usually) —we want these to go down.
- Note the scenario: in entertainment scenarios, "time on page" is actually positive (mobile games want users to play longer).
Improvement score formulas (be sure to remember the reverse direction):
| Metric type | Improvement score formula |
|---|---|
| Positive metric | (post-change value ÷ pre-change value) − 1 |
| Negative metric | (pre-change value ÷ post-change value) − 1 |
Calculations for you:
- Conversion rate 2% → 5% (positive): 5 ÷ 2 = 2.5, i.e., 150% improvement.
- Task time 3 minutes → 2 minutes (negative): 3 ÷ 2 = 1.5, i.e., 50% efficiency improvement(in the same 1 hour, the old version completes 20 3-minute tasks, the new version completes 30 2-minute tasks, 50% more work done).
- HelloFresh success rate 25% → 100% (positive): 100 ÷ 25 = 4, i.e., 300% improvement; task time 28 seconds → 14 seconds (negative): 100% improvement.
3.2 Three-step conversion method: turn UX metrics into KPIs into money
Core metaphor (the single most important sentence in the whole article): Calculating ROI is essentially unit conversion— it is the same thing as "how many liters are two gallons", except what you are converting is "task time → cost savings".
Step 1 Choose KPIs: first ask "what does the company actually care about?" Profit? Cost? Churn rate? Satisfaction? Choose based on organizational goals + based on reporting audience(executives/clients/investors have different focuses).
Real lesson (Marketade researcher Emily): The same finding that "engineers account for 70% of monitoring" could be linked to "production targets" and was very useful at her first oil company; at the second company no one cared, they only cared about "data quality" — she changed the angle and said "UX improvements reduced production error by 10%, avoiding millions of dollars in losses". Conclusion: "The data people pay attention to is closely related to their role." The art of ROI is "telling a resonant story".
Common KPI menu: profit / cost / customer lifetime value CLV / employee turnover rate / employee productivity / number of donors.
When you can skip this step: In organizations with high UX maturity, KPIs and UX metrics are already aligned (everyone already values "task completion time"), in which case calculating ROI may be unnecessary— the value is already recognized, don’t calculate just for the sake of calculating.
Step 2 Convert UX metrics to KPIs: find the conversion ratio. Example: customer service process saves 30 seconds, hourly wage is 100 yuan → saves 0.83 yuan per interaction. You often need to ask other departments for data(analytics team/HR/finance); when getting the data, be sure to clearly explain what you need it for.
Step 3 Report responsibly(5 iron rules, see 3.5).
3.3 Signature case: health insurance site, saves 1.5 million yuan a year (all numbers retained)
UX metrics for the registration flow can be obtained from four sources: success rate (usability testing)/ completion rate (website analytics)/ ease-of-use score (survey)/ customer service ticket count (customer support data). When actually calculating ROI,picking only one core metric is the easiest — here we pick customer service tickets, because they are the easiest to convert into money.
工单减少量 = 改版前工单数 − 改版后工单数 = 23,000 − 1,100 ≈ 21,900 张
成本节省 = 工单减少量 × 每张工单成本 = 21,900 × $6
月省 ≈ $131,400 → 年省 ≈ $1,576,800(约合一年省 150 多万美元)
PM implementation: The methodology of this case can be directly applied — any project where "customer complaints/tickets drop after a redesign" can be
(降低的工单数 × 单张工单成本)converted into money saved, which is the ROI calculation most easily recognized by finance teams.
3.4 Monetary conversion formulas for four types of business
Why: Different businesses have different "things that are valuable", so how to calculate ROI depends on the business type. First identify which category your business belongs to.
| Business type | Where value is anchored | Monetary conversion formula |
|---|---|---|
| E-commerce / sites with clear target behaviors | Conversion (purchase, application, registration, subscription) | Conversion lift × average order value. The more users, the more valuable it is (Amazon sales doubled = revenue doubled). If a design change makes it harder and sales drop, the boss will make you change it back |
| Content sites (news type) | Traffic (PV/UV → advertising) | PV/UV increase after redesign × ad unit price |
| Enterprise intranet / tools | Employee work hours = wages | Person-days saved × average salary. Efficiency improvement = reduced labor cost = ROI |
| Nonprofit / government | Non-monetary (traffic, service coverage, number of donors) | ROI doesn’t have to be money, it can be "how many people were helped" |
Three mini cases (all numbers retained):
- Financial subscription site The Deal (e-commerce type): After the redesign, free trial applications increased from 76 to 187 per month over three months, adding $20,000 in monthly revenue76 → 187; assuming 18% convert to paid, $1,000 per customer:
(187−76) × 18% = 20 个新付费用户 → 20 × $1,000 = 月增收约 $20,000(and trial customers keep accumulating, more in the future). - HR tool (intranet type): Optimized a high-frequency time-consuming task, saving $6,112 per month32 person-days; HR annual salary $50,000 ÷ 261 working days ≈ $191/day →
$191 × 32 = 月省 $6,112. Honest note: What is saved for fixed-salary employees is not cash, but "this time can be invested in other tasks", which creates value in the long run — this is how you should phrase it in reports to be credible. This high-frequency task reduced HR time spent overall by 60%. - Intranet scale amplification effect: For the same intranet usability optimization,ROI for a 1,000-person company ≈ 8 times cost; 10,000 people ≈ 20 times; 100,000 people ≈ 50 times— because every user benefits from the improvement, and headcount amplifies the returns.
3.5 Five iron rules for reporting (talking points for reporting to the boss)
| Iron rule | One-liner | Why |
|---|---|---|
| ① Estimate, not financial report | "You don’t need to calculate ROI down to the last cent, just estimate roughly" | ROI is a directional indicator to help you judge which project is more worthwhile, not a financial forecast |
| ② Compare against R&D cost | Compare the money saved/earned against the R&D investment for this improvement | This is the core of ROI; only saying you saved money without saying how much you spent is meaningless |
| ③ Tell stories, don’t pile tables | "String numbers into a logically clear story line" | Answer "why we did it, how we did it, was the result good", don’t dump a bunch of tables no one will read |
| ④ Be transparent about assumptions | Clearly state the sources of numbers and the basis for assumptions | If the audience thinks you are "just fighting for budget for yourself", they will treat you as if you are pushing a personal agenda |
| ⑤ Extend to 2–5 years | "A good design saves money for years, don’t only look at the immediate term" | UX improvements are cumulative (save 300,000 yuan this year, save again next year), a longer time horizon is more persuasive |
Counterexample: Calculating every UI component and every variable into ROI for the sake of "rigor" — takes weeks, and the boss has already zoned out.Overanalysis = no analysis(see the Don't Overthink example of "the design system convinced the boss by only calculating one video plugin" in NNGroup Analytics & Metrics (choosing the right metrics/conversion funnels/statistical significance and confidence intervals/four steps of UX ROI/sampling and confounding/triangulation/vanity metrics/benchmarking)).
3.6 Advanced: Use correlation to connect "soft metrics" to revenue KPIs
Why: Executives often feel that "soft" metrics like satisfaction and ease of use "have nothing to do with money". How to prove they ultimately affect revenue? — Use statistical correlation to build a logical chain.
Athenahealth chain model (medical SaaS empirical evidence): Collected over 50,000 complete surveys on one product over two years, asking about "ease of use" and "reliability". Statistically verified a significant positive chain (p < 0.000001):
易用性 + 可靠性 → 产品满意度 → 推荐意愿(NPS) → 付费推荐 + 用户留存
（用户感知） （满意度） （忠诚度） （营收 KPI）
How to use this chain: When problems occur with the product,work backwards— Retention dropped? Is there a satisfaction problem? Is ease of use/reliability not good enough? This gives PMs a provable path of "improve ease of use → ultimately affect KPIs", backed by a closed data loop.
The universal version is called the UX-Revenue Chain, which can be applied to any product: ① Experience investment(usability + customer service + product quality)→ ② Overall satisfaction → ③ Loyalty(willingness to recommend/reuse/reduced churn)→ ④ Revenue(upsells/renewals/referrals/reduced customer acquisition cost). The front segment is "experience KPIs", the back segment is "business KPIs" — this diagram itself can serve as the design basis for a KPI system.
IV. UX Benchmarking: the "ruler" for quantifying whether a redesign is worth it
Why: Without a baseline it’s like losing weight without weighing yourself first — after the change you have no idea if it worked. Benchmarking = using quantitative data to measure a product’s performance relative to a certain standard (its own history / competitors / industry), and track improvements.It is the data source for ROI calculation(the metrics for Step 1 of ROI come from here).
4.1 Seven-step process: first three steps for launch, last four steps for iteration
| Step | What to do |
|---|---|
| 1 | Choose what to test — lock in the product + user group, find "the tasks most critical to users" |
| 2 | Choose how to test — survey / analytics / quantitative usability testing (choose one or combine) |
| 3 | Collect first round of data — use it as the Baseline |
| 4 | Redesign the product |
| 5 | Test again |
| 6 | Interpret results |
| 7 | Calculate ROI (optional) |
Steps 4–7 are a repeatable cycle for long-term tracking.Two easy pitfalls:
- Don’t rush to test right after changes: Users resist change; major changes need an adaptation period — wait 2–3 weeks for products used daily, wait 1 month+ for low-frequency products.
- Sample size threshold: There are cases in reports that were rejected because only 5 people were used;At least 20–40 people are needed to be persuasive (for details on statistical significance and confidence intervals see NNGroup Analytics & Metrics (choosing the right metrics/conversion funnels/statistical significance and confidence intervals/four steps of UX ROI/sampling and confounding/triangulation/vanity metrics/benchmarking)).
4.2 Use the Google HEART framework to choose metrics (selection scaffold)
The most commonly used scaffold when choosing "what to test". It breaks UX into 5 dimensions, each with ready-made metrics — preventing clients/bosses from only fixating on shallow data like "page views".
| HEART dimension | Meaning | Example metrics |
|---|---|---|
| Happiness Happiness | User attitude | satisfaction score, ease-of-use score, NPS |
| Engagement Engagement | Activity level | average task time, feature usage rate, conversion rate |
| Adoption Adoption | First-time use | new registrations, visit count, sales |
| Retention Retention | Long-term use | returning users, churn rate, renewal rate |
| Task success Task success and efficiency | success rate + efficiency | Number of errors, success rate, task time |
When to use: When the client cannot clearly articulate the "success criteria" (and only says "we want to do it well"), use HEART to help them turn vague goals dimension by dimension into measurable metrics.
4.3 Find comparison benchmarks: Establish a baseline even without historical data
The first test gives you the "baseline score". Even if you have just started and have no historical data, you can still find benchmarks:
- Competitor comparison: Run a round of quantitative testing on competitors' apps as well (HelloFresh vs BlueApron).
- Industry standards: The industry average usability score for "finding recipes" is 4.5 (out of 5); compare your own product against that.
- Stakeholder goals: The boss requires "finding a recipe ≤ 20 seconds"; a measured result of 14 seconds = meets the target.
4.4 Empirical example: HelloFresh redesign data (fully retained)
| Metric | Before redesign | After redesign | Improvement score |
|---|---|---|---|
| SUS (out of 100) | 75 | 90 | +20% |
| Task time (seconds) | 28 | 14 | +100% (twice as fast) |
| Subjective success rate | 63% | 100% | +59% |
| Success rate | 25% | 100% | +300% |
| Usability score (/7) | 6.1 | 6.9 | +13% |
| Confidence score (/7) | 5.9 | 6.9 | +17% |
4.5 Don't declare failure even if "results are not significant" (Shopify counterexample)
For the Shopify billing page redesign, the data were all not significant, and even slightly worse(completion time +8% slower, usability −3%, success rate 0 change):
| Metric | Before redesign | After redesign |
|---|---|---|
| Completion time | 59 seconds | 64 seconds (+8% slower) |
| Usability (/7) | 6 | 5.8(−3%) |
| Success rate | 90% | 90% (no change) |
But qualitative observations found that users understood the information sections more clearly, had shorter exploration paths, and made fewer misclicks. The researcher summarized: "Some design changes will not immediately show up in metrics, but you must use quantitative methods to break them down, to know whether 'nothing really changed' or 'it got better, just in a different way.'" Moreover, this "non-blockbuster" project forced the team to build a set of UX benchmark testing toolkit, which was rolled out to the entire Shopify design team——the redesign with no significant improvement instead laid the cornerstone for long-term improvement.
Five, Expectation Management: How much can a UX redesign actually improve + Why it is getting harder
Why (for PMs): When the boss asks "How much improvement will the redesign bring?", you need an evidence-based expected range, neither overpromising nor underselling yourself. NN/g's 14-year case library provides the answer.
5.1 Average improvement is 75%, but the range is wide
- The average improvement across all cases = 75%, with a 95% confidence interval 8% ~ 104%(confidence interval = the range where the true mean is very likely to fall, see NNGroup Analytics & Metrics (choosing the right metrics/conversion funnels/statistical significance and confidence intervals/four steps of UX ROI/sampling and confounders/triangulation/vanity metrics/benchmarking)).
- Don't expect to hit the jackpot every time: Half of the projects have improvement ranges between 8% and 104%13% ~ 157%, some high and some low. Improvements of 500% or 5000% exist but are extremely rare(statistical outliers).
Why do some redesigns double performance while others stay completely unchanged? Five factors:
- How bad the original experience was——The more problems there are, the greater the room for improvement; for an already excellent product, the room for improvement is small.
- How capable the team is——Experience/skills/strategy determine depth; but even top teams cannot be perfect every time.
- How much was changed, and how critical the changes were——Large projects usually have a big impact, but there is no statistically significant association between project scale and improvement effect(small, focused changes can also produce outsized results).
- Whether the methods/metrics are sensitive enough——A/B tests with thousands of data points can detect very small changes; usability tests with 40 participants require larger differences to be significant (Shopify's small changes may have been missed by insensitive metrics).
- How hard the target behavior is to change——The harder a habit is to change, the smaller the design impact.
Important reminder: Some redesigns will make metrics worse——The report includes a company where form submissions actually dropped after a redesign by 20%65%. Even with research done, the strongest teams can still fail.As long as you keep measuring data, you can detect issues promptly and revert——the earlier you detect, the smaller the harm.
5.2 Break down by metric category: Which types are easiest to improve, which are hardest
| Metric category | Average improvement | 95% confidence interval | PM interpretation |
|---|---|---|---|
| Effectiveness Effectiveness | 131% | −9% ~ 271% | Design has the largest impact on "helping users complete tasks", but also has high volatility (has a negative lower bound, may make things worse) |
| Efficiency Efficiency | 104% | 31% ~ 177% | Improving "how fast it is" is also very effective |
| Adoption/retention | 93% | 45% ~ 140% | Relatively stable |
| Engagement/usage | 87% | −3% ~ 177% | Highly volatile, may make things worse |
| Satisfaction/perception | 19% | 6% ~ 33% | Improvement magnitude is the smallest (perception is hardest to change), but the most stable and controllable |
Reporting usage: If you want to commit to improving "task success/efficiency", you can be confident ("the industry average can improve by 100%+"); if you want to commit to improving "satisfaction/NPS", you should lower expectations ("perceptual metrics generally only improve by ~19%, but it is very stable")——this helps PMs set realistic OKRs and avoid making unrealistic promises to the boss.
5.3 The "narrowing improvement" pattern: From 247% down to 75% (don't misinterpret this as "UX is useless")
14-year longitudinal comparison: The average improvement score dropped from 247% in 2006–2008 to 75% today, a decrease of69%, which is statistically significant (p = 0.017). Breakdown of the decrease by category:
| Metric category | 2006–2008 | 2020 | Trend |
|---|---|---|---|
| Effectiveness | 324% | 131% | Significant decrease |
| Engagement/usage | 269% | 87% | Significant decrease |
| Adoption/retention | 221% | 93% | Significant decrease |
| Efficiency | 109% | 104% | Basically flat |
| Satisfaction/perception | 33% | 19% | Slight decrease |
Correct interpretation (Jakob Nielsen): "Lower improvement scores do not mean we are doing a bad job; on the contrary, it shows that we have solved many problems very well over these years. Early designs were terrible (messy splash screens, useless search, screens full of images), so finding UX problems was extremely easy."
Implications for PMs (key):User expectations rise in lockstep with average experience quality——If you put a website that was "decent" in 2006 in front of users today, they will close the page immediately. Therefore:Even if a single improvement is only a few percentage points, it is still worth investing in, especially because competitors are also continuously optimizing——you either move forward or fall behind. This is the strongest argument to persuade the boss to "keep investing in UX": it is not for blockbuster improvements, but to avoid being left behind by rising expectations and competitors.
Quick reference: One-pager for PMs to take to reports
| What you want to do | Use this |
|---|---|
| Select metrics (afraid of missing something) | 6-category table (2.2) or Google HEART five dimensions (4.2) |
| A total score comparable across products | SUS(0–100) |
| A loyalty metric easiest to communicate to executives | NPS (Promoters% − Detractors%) |
| Convert "task time↓" into money | Improvement score formula (3.1) + three-step method (3.2), negative metric = before ÷ after − 1 |
| Convert reduced customer complaints into cost savings | Reduced number of tickets × cost per ticket (3.3 health insurance case) |
| ROI for intranets/tools | Person-days saved × average salary (3.4 HR case) |
| Connect "soft metrics" to revenue | UX-Revenue chain model (3.6) |
| Give the boss a realistic improvement expectation | Average 75% (range 8–104%); by category see 5.2 |
| Establish a baseline even without historical data | Competitors / industry standards / boss-set goals (4.3) |
| Avoid getting criticized in reports | 5 iron rules (3.5): estimates are not financial reports · compare costs · tell stories · be transparent · take a 2–5 year view |
Sourced from Will's Design Notes "UX Metrics and ROI" (based on the NN/g report of the same name + 5 editions of the case library). Complementary reading:NNGroup Analytics & Metrics (choosing the right metrics/conversion funnels/statistical significance and confidence intervals/four steps of UX ROI/sampling and confounders/triangulation/vanity metrics/benchmarking) (NN/g analytics and measurement perspective: vanity metrics/conversion rates/statistical significance/sampling/triangulation/four-step ROI principles), NNGroup Quantitative Research (A/B testing · sample size · confidence intervals · UX metrics · conversion analysis · benchmarking · learning curves). The original PDF (64MB) and accompanying images are retained in sources (already gitignored), and all charts have been converted to text + table descriptions.
Sources and related materials
- NNGroup Analytics & Metrics (choosing the right metrics/conversion funnels/statistical significance and confidence intervals/four steps of UX ROI/sampling and confounders/triangulation/vanity metrics/benchmarking)
- NNGroup Quantitative Research (A/B testing · sample size · confidence intervals · UX metrics · conversion analysis · benchmarking · learning curves)