Quantitative Research = compressing user experience compressing user experience into comparable numbers(success rate, task time, conversion rate, satisfaction score) to answer three questions: Is this version better than the last? Should we choose A or B? How much is UX investment worth (ROI)?
Its benefits are clarity, traceability, and the ability to persuade data-focused bosses; its pitfall is that numbers look objective, but are actually very fragile — too small a sample is unreliable, too large a sample produces "significant but irrelevant" results, too many variables produce "spurious correlations", and lax execution leads to systematic misdirection. NNGroup's consistent position:Quantitative research supplements, rather than replaces, qualitative research — first use qualitative research (5 participants) to find problems and generate hypotheses, then use quantitative research (40 participants / high traffic) to validate direction and quantify impact.
The content below is divided into eight sections: A/B and MVT (what direction to test) → the three core statistical concepts (significance/confidence interval/sample size) → two types of error and measurement error → metrics and scales (success rate/learnability/NPS/Likert/vanity metrics) → conversion and analysis (macro-micro/Analytics) → benchmarking → data quality → IA research (card sorting/tree testing).
1. A/B Testing and Multivariate Testing: Validate Direction First, Then Optimize Details
Four articles on A/B testing are highly overlapping (A/B Testing 101 two versions + Putting A/B Testing in Its Place + Define Stronger A/B Variations), and have been merged; MVT is covered in a separate paragraph.One-sentence guidance: Use A/B testing to set direction for major redesigns, then use MVT to refine details on winning designs.
| Method | What it tests | How it is tested | When to use it | Core limitations |
|---|---|---|---|---|
| A/B Testing | Which of two (or more) full versions performs better on a single business metric (conversion rate/click-through rate) | Randomly split real traffic into groups assigned to A (control group, current state)/ B (experimental group, 1 element changed),run during the same time period to exclude external variables such as seasonality and market effects | Validating the overall direction of major overhauls and complete redesigns | ① Only one variable can be tested at a time ② Requires high traffic (small sites will need to run tests for a very long time) ③ Only tells you "what changed", not "why it changed" — must be paired with qualitative research |
| Multivariate Testing (MVT) | Which variant combinations of multiple UI variables perform better, as well as the interactions | Test all combinations simultaneously (e.g. image/video × buy now/add to cart) to reveal the specific contribution of each variable | Incremental optimization after a winning design is identified Incremental optimization of details (button copy, image style, etc.) after a winning design is identified | Number of combinations = number of variants per variable multiplied together, which explodes and consumes large amounts of traffic; all combinations must be semantically and visually reasonable |
Do not confuse the three MVT terms: Variable (UI element, e.g. "image") → Variant (specific version, e.g. "image / video") → Variation (the final product that takes one variant from each variable, e.g. "video × add to cart").
The math of combinatorial explosion: 2 variables × 2 variants each = 2×2=4 combinations; adding a 3rd variant to one variable = 2×3=6. The more combinations, the more traffic and time are required to reach statistical significance.
Core principle: A/B testing answers "which overall version is better", MVT answers "which element, and how elements work together, performs better". Example: A/B testing can only say "the video version beat the image version", but cannot distinguish whether the video attracted users or the changed button copy performed better; MVT can break this down to "video × add to cart is optimal, while button copy actually has very little impact".
Counterexample (Garbage in, garbage out): A/B testing is a powerful tool, but if the variants themselves are garbage, the conclusions will also be garbage. Three common failures: ① The variant does not deliver conceptual value (added option descriptions written like advertisements → ignored by users); ② The variant does not solve the real problem (simplified the form, but users are actually stuck on "cannot find the loan interest rate"); ③ The variant is based purely on guesswork.Solution = define variants only after conducting UX research — Einstein said "Experiments can prove a theory, but they cannot give birth to a theory (Chinese quote)".
4 steps to set up an A/B test: ① Formulate a hypothesis (research-based, e.g. "changing 'Purchase' to 'Buy Now' will increase conversions") ② Define the change (only change 1 element, small and precise) ③ Select outcome metrics + guardrail metrics (when testing click-through rate, also monitor purchase rate/average order value to prevent one metric from rising while others collapse) ④ Determine test duration (calculated from sample size, requires three parameters:baseline value, minimum detectable effect, significance threshold typically 95%/p=0.05). Pair with A/A testing (testing two identical designs) to verify the tool is set up correctly. Common mistakes: vague goals, stopping the test too early, no strong hypothesis, focusing only on a single metric.
2. The Real Role of A/B Testing: Solid Benefits, but Short-Sighted and Blind to Causes
Putting A/B Testing in Its Place places A/B testing back in its position as a "supplementary tool" — this is NNGroup's most important attitude correction regarding A/B testing.
| 4 major strengths of A/B testing | 4 major blind spots of A/B testing |
|---|---|
| ① Real user behavior in real contexts (credible) ② High traffic can detect 1% level tiny differences ③ Resolves design tradeoffs (e.g. found that "removing the prominent promo code input box increased sales by 20–50%") ④ Low cost, lightweight technically | ① Can only evaluate a single clear KPI (cannot measure B2B brand/PR goals) ② Can only test fully implemented designs (paper prototypes can cut bad ideas much earlier) ③ Only looks at short-term metrics (promotions boost short-term sales, but long-term page clutter reduces loyalty) ④Lacks behavioral insight — does not know why the big button won, does not know if a better solution exists, cannot detect unexpected trust/information issues |
Conclusion: A/B testing should not be the first choice for improving conversion rates, nor should it be the only research method for a project. The correct process = qualitative research (user testing + paper prototypes) to rapidly generate hypotheses and concepts → after qualitative iteration, use A/B testing to verify whether the final design is truly better than the status quo.
3. Statistical Significance ≠ Practical Significance (The One Thing Product Managers Must Understand)
Statistical Significance Isn't the Same as Practical Significance is listed separately — because this is where numbers are most deceptive.
| Concept | What it answers | One-sentence explanation |
|---|---|---|
| Statistical Significance | Is the result reliable (was it produced by random chance)? | p < 0.05 = the probability this result occurred randomly is < 5% |
| Practical Significance | Is the result worth acting on | Is the difference large enough to affect user perception/product metrics/business outcomes? |
Intuitive formula for p-value (unknown to many PMs):p ≈ 差值 ÷ 随机噪声(标准误差),再换算成"这么大的差有多罕见"的概率.
Full worked example: 1000 users each in A and B, 850 successes in A (85.0%), 852 successes in B (85.2%) → difference 0.002 → standard error SE=√(p(1-p)/nₐ + p(1-p)/n_b) ≈ 0.0159(p=average success rate 0.851) → Z = 0.002/0.0159 ≈ 2.17 → look up in table p ≈ 0.03. statistically significant, but the difference is only 0.2% — users cannot feel it at all, and it is not worth the development cost.
How sample size distorts "significance":
- Large samples (tens of thousands/millions) → even extremely tiny differences (form error rate reduced by 0.03%, p<0.001) are "significant", but significant but unimportant.
- Small samples (common in UX, 10–15 participants) → even very large differences may fail to reach significance. Example: testing two dashboards, A failed for 10/12 users, B failed for 1/12 — this may not be statistically significant, but A clearly has UX problems — in this case practical significance outweighs statistical significance.
Practical significance is context-dependent: the same "+$2 per conversion" is unimportant for a small product; but for a large system with $1000 per transaction and millions of annual conversions → $2 × 1,000,000 = $200 万, it represents enormous business value.
Core principle: Do not judge UX conclusions by p-value alone; look at three things — statistical significance (is it reliable?) + practical significance (is it worth doing?) + qualitative evidence (where are users stuck, and why?).
4. Confidence Interval / Margin of Error / Confidence Level: How to Honestly State "I Didn't Test All Users"
Confidence Intervals... + Measurement Error in UX Research are merged — their core is all "estimating a population from a sample necessarily involves error, which must be quantified and communicated".
| Concept | Definition | Example |
|---|---|---|
| Confidence Interval | The range where the true population metric is likely to fall | 79% of Canadian adults willing to get vaccinated, margin of error ±2.1% → true value lies between 77%–81% |
| Margin of Error | The one-sided distance from the observed value to the interval bound | Lower interval bound = observed value − margin of error; upper bound = observed value + margin of error;Interval width = margin of error × 2 |
| Confidence Level | The probability that the interval contains the true value | 95% = if the study were repeated 100 times, 95 of the intervals would contain the true value, 5 would not |
Narrower intervals carry more information: 40% success rate with an interval of 0%–80% → uninterpretable; interval narrowed to 35%–45% → can guide design.
3 factors that affect interval width: ① Larger sample → narrower; ② Higher sample variability → wider; ③ Higher confidence level → wider.
Adjust confidence level by task importance: Critical tasks (aircraft landing instrument panels) require high confidence levels; non-critical tasks (in-flight entertainment reset) can use lower levels. Tradeoff = higher confidence level is more accurate but produces wider intervals, requires larger samples, and is more expensive.
Measurement Error: Cannot be eliminated no matter how hard you try; sources = small number of participants / individual differences / test environment / external interference. The solution is not to eliminate it, but to increase sample size to reduce it + use confidence intervals to communicate it honestly — tell the boss "the result is an approximation, but the true value is roughly within this interval".
5. Sample Size: Why 40 Participants for Quantitative, 5 for Qualitative
Why Use 40 Participants + How Many Participants... + Why 5 Are Okay in Qualitative but Not Quantitative Three articles merged — this is the most frequent practical question in quantitative research.
Why 40 (for binary metrics): Target = 95% confidence level + 15% margin of error (the balance point between UX accuracy and cost). Statistical formula n = Z²·p(1−p) / E², where Z=1.96 (95%), p=0.5 (most conservative estimate), E=0.15 → calculates to 39, rounded up to account for mid-study dropouts = 40.
- Cannot go lower: 10 participants → margin of error around 30%, results are too broad to guide decisions (if observed success rate is 70%, the interval may span 40%–90%).
- Cannot go higher (diminishing returns): reducing the margin of error from 15% to 10% requires 100+ participants, with steeply rising costs.
Participant-Saving Reference Table (Binary Metrics):
| Confidence Level | Margin of Error | Required Participants |
|---|---|---|
| 95% | 15% | 39 |
| 95% | 20% | 21 |
| 90% | 15% | 28 |
| 90% | 20% | 15 |
Continuous Metrics (Task Time / Satisfaction): Sample size depends on the population standard deviation; the recommended assumption is standard deviation = 52% of the mean(mean 1 minute → SD≈0.52; mean 10 minutes → SD≈5.2).
| Confidence Level | Margin of Error (% of Mean) | Required Participants |
|---|---|---|
| 95% | 15% | 47 |
| 95% | 20% | 26 |
| 90% | 15% | 33 |
| 90% | 20% | 19 |
Optimization Strategy: If budget is tight, start with 20–25 participants, calculate the margin of error, then decide whether to add more.
Why 5 Participants Suffice for Qualitative Research but Not for Quantitative Research:
| Research Type | Goal | Sample Size | Use of Results |
|---|---|---|---|
| Qualitative | Discover interface problems, drive iteration | 5 participants usually suffice (lower problem probability requires more participants) | Only provides a list of problems,does not generalize to overall user behavior |
| Quantitative | Measure metrics, predict overall user behavior | Typically 30–40 participants | Data must pass confidence interval and significance checks |
Mathematics of the 5-user rule (Nielsen & Landauer): Assuming the probability that each user encounters a problem is 31%, then 5 users can discover 85% of usability problems.Point of criticism: The 31% figure comes from 1990s data; modern interfaces are more usable → lower problem probability means more participants are needed (20% probability → 9 participants to find 85% of problems; 10% probability → 18 participants). For quantitative research, because means/success rates/confidence intervals must be calculated, intervals from 5 participants are so wide as to be meaningless (70% success rate may yield 40%–90%).
VI. Two Types of Errors: False Positives and False Negatives
Common Errors in Quantitative Research Single section — directly tied to sample size.
| Error | Formal Name | Meaning | Primary Cause | Solution |
|---|---|---|---|---|
| False Positive | Type I Error | Data shows a difference when none actually exists(mistakenly believing a change was found) | Outliers, confounding variables (attributing higher satisfaction to the new design when it is actually caused by unrelated factors) | Recruit representative users + control for confounding variables |
| False Negative | Type II Error | Data does not show a difference when one actually exists(missing a real change) | Insufficient statistical power— sample too small | Increase sample size (quantitative usability testing recommends at least 40 participants) |
Core principle: Both types of errors lead you to misjudge a design — only carefully designed studies + controlled variables + sufficient sample size let you trust your conclusions.
VII. Metrics and Scales: Success Rate / Learnability / NPS / Scales / Vanity Metrics
1. Success Rate: The Simplest Usability Metric
Success Rate Single section. Success rate = percentage of participants who complete the task; it is binary, simple, and intuitive,but cannot explain why failures occur, nor does it measure efficiency. Importance: If a task cannot be completed, all other metrics are meaningless.
Success Levels (How to Report Partial Success) — Example "Order 12 yellow roses for mother":
| Success Level | Number of Participants (out of 100 total) | Report (including 95% confidence interval) |
|---|---|---|
| Complete Success | 20 | 20%(CI 13%–29%) |
| Success with Minor Problems (order placed but greeting card omitted) | 35 | 35%(CI 26%–45%) |
| Success with Major Problems (order placed but address incorrect) | 30 | 30%(CI 22%–40%) |
| Failure | 15 | 15%(CI 9%–23%) |
Key Counterexample (Common Mistake): Assigning numerical values to levels (complete=1/minor=0.66/major=0.33/failure=0) and calculating a weighted average to get a 53% success rate —Wrong! Success levels are ordinal variables; the distances between labels are not equal, so they cannot be treated as continuous variables for mathematical operations. Correct approach: use text labels + report confidence intervals separately for each level; do not calculate an average.
2. Learnability and Power Law of Learning
How to Measure Learnability + Power Law of Learning Merged.
Learnability = One of the five components of usability (the others: efficiency, memorability, error rate, satisfaction), focusing on ease of first use + how many repetitions are needed to become efficient. Three dimensions correspond to different user groups:
| Dimension | Definition | Suitable Users |
|---|---|---|
| First-use learnability | Ease of use on the very first attempt | One-time users (e.g., registration) |
| Learning curve steepness | Speed of improvement with repetition | Occasional users |
| Final plateau efficiency | Maximum productivity after full proficiency | High-frequency core users |
Power Law of Learning: "Task completion time decreases with the number of repetitions, following a learning curve." Formula Y = A·X^(−B)(Y=task time, X=number of repetitions, A/B are constants). Curve characteristics: slowest on first attempt → faster each time → after a certain point (e.g., the 4th repetition) approaches a saturation plateau, with minimal further improvement from additional practice.
Trade-off Counterexample: A wizard flow is friendly to novices, but proficient users’ task time stays stuck at 16 minutes and cannot go lower —learnable but not efficient.
Why Redesigns Draw Backlash (Most Important Design Takeaway): Longtime users have already reached the plateau (high efficiency) on the old interface; switching to a new interface = resets the learning curve back to the 1st repetition → real drop in productivity (not just "dislike"). Responses: ① Roll out small changes incrementally ② Explain the value of changes ③ Allow switching between old and new versions.
4 Steps to Conduct a Learnability Study: ① Select metrics (commonly task time; can also use error count/success rate) ② Define number of trials and intervals (recommend 5–10 trials to reach saturation; intervals match real usage frequency: daily tasks over consecutive days / monthly tasks every 4 weeks) ③ Have the same group of participants perform the same task multiple times, plot an average time line chart ④ Analyze (use one-way repeated measures ANOVA to test whether the trend is real; steep slope = fast learning, plateau efficiency = proficiency ceiling). Applicable to complex high-frequency systems; for low-frequency one-time tasks (annual tax filing), standard usability testing is sufficient.
3. NPS (Net Promoter Score): Loved by Management, but Not a Usability Metric
NPS: Understanding Its Insights + NPS: What a Customer-Relations Metric Can Tell You Two largely overlapping articles, merged.
Definition and Calculation: Ask "How likely are you to recommend this to a friend?" (0–10 scale) → sort into three groups:Promoters 9–10 / Passives 7–8 / Detractors 0–6. Formula NPS = 推荐者% − 批评者%, range −100% to +100%. Origin: Frederick F. Reichheld, 2003, Harvard Business Review; chosen because the "recommend" question correlates most strongly with repeat purchase and word of mouth.
International Differences: Mathematically the mean should be 5, but only 9–10 count as Promoters and 6 and below count as Detractors, which appears strict; India/Mexico score high (>60%), Japan/Korea score low and even negative; the US prioritizes quality, the UK prioritizes ease of use, the Netherlands prioritizes innovation. Recommend keeping the standard algorithm but accounting for regional differences in interpretation.
6 Major Limitations (Why It Cannot Be Used Alone): ① It is only a subjective score and cannot capture the full experience (users may struggle badly but still give a high score) ② No statistical significance with small samples ③ Binning loses information (a jump from 2 to 5 is a big improvement but still counts as Detractor) ④ Customer loyalty ≠ usability (affected by pricing/brand) ⑤ Cannot measure satisfaction with specific designs (changing one page may not move overall NPS) ⑥ It can be manipulated (only send surveys when customers are satisfied, offer coupons for positive reviews).
Why It Is Still Used: Management accepts it (tied to profit), it is simple to collect (single question, high response rate), and it has strong correlation with UX metrics such as SUS. Conclusion: NPS is not a qualified UX metric, but combined with other UX metrics it can track long-term trends in overall perception.
4. Likert Scales vs Semantic Differential Scales
Rating Scales in UX Research Single section. Two subjective rating tools with subtle differences.
| Characteristic | Likert Scale | Semantic Differential Scale |
|---|---|---|
| Invention | Rensis Likert,1930s | Osgood et al., 1957 |
| Information Captured | Express agreement/disagreement with a single statement("I find the system easy to use") | Select a position between a pair of opposite adjectives("Easy — Difficult") |
| Number of Options | Typically 5–7 | Typically a 7-point spectrum |
| Labels | Every option has a label | Only the two poles are labeled; the middle is unlabeled |
| Limitations | Acquiescence bias (tendency to agree) + social desirability bias (choosing socially acceptable answers) | Higher cognitive load (requires abstract comparison); results are inaccurate if users do not understand the opposite terms |
SUS (System Usability Scale) is a typical Likert-style example: 10 statements about system use, each with 5 points (Strongly Agree → Strongly Disagree).Selection Guidance: Measuring attitude/agreement → Likert (straightforward and easy to understand); measuring perception (ease of use/aesthetics) → Semantic Differential (more flexible spectrum). Tip: If unsure, measure both; prefer validated standard scales (such as SUS) over self-created ones; for Semantic Differential, ensure opposite terms are truly antonyms and understandable to users.
5. Vanity Metrics: Meaningful Only With Context
Vanity Metrics: Add Context to Add Meaning Single section.Vanity Metrics = Numbers that only ever grow, look good but provide no actionable guidance (total users, total downloads, total page views). Problem: Out of context, they do not reflect UX changes.
Four Ways to Add Context:①Add a time frame(short-term is fast but has random fluctuations / long-term is stable but slow to respond → weekly/monthly new users are more usable) ②Calculate ratios per user or per visit(proportion of users performing a behavior, per-user frequency; for behaviors that occur multiple times per visit, calculate per visit, e.g., page views per visit) ③Cohort Analysis(compare subsequent behavior ratios of users who registered in the same period; sudden changes = signal to investigate) ④Ratios between metrics(unique page views / total visits;stickiness ratio DAU/MAU, 10% = 1 daily login per 10 monthly active users).
Example: Looking only at total downloads (red line) shows no UX change; looking at the "new active users / downloads ratio" (blue line) clearly shows that the Week 4 update significantly increased the proportion of new users who became active.Tracking Principle: Stable ratios/proportions are good metrics (any change reflects real system changes rather than random fluctuation); if a metric change cannot drive actionable outcomes, it is a vanity metric — do not track it.
VIII. Conversion and Analysis: Macro & Micro Conversion + Three Uses of Analytics
1. Macro Conversion vs Micro Conversion
Macro vs Micro Conversions Single section.
| Type | Definition | Example |
|---|---|---|
| Macro Conversion | Behaviors directly tied to business goals, usually KPIs | Completing a purchase / submitting a lead form / completing event registration / creating an account; completing tax filing on a government website also counts |
| Micro Conversion | Indirectly related to KPIs but more frequent, divided into two categories | See below |
Two types of micro conversions: ①Process Milestones(key steps leading to macro conversion, e.g., flight booking flow: enter flight info → select flight → enter passenger details → add services → pay; the earlier steps are all micro conversions, tracking them finds drop-off points) ②Secondary Actions(do not directly convert but predict future conversion, e.g., subscribing to emails / downloading the App).
Non-profit Goals: Required behaviors (applying for leave in an internal system) are not suitable for conversion rates; use ease of use/error rate/customer service data instead; optional behaviors (signing up for non-mandatory events) are suitable for conversion rates.
2. Three Uses of Analytics in UX Practice
Three Uses for Analytics + Translating UX Goals into Analytics Measurement Plans Merged.
| Use | What It Does | Reports/Tools |
|---|---|---|
| ① Problem Indication | Monitor key goals, quickly diagnose potential problems/leads | Page reports filtered by URI, source as secondary dimension |
| ② Problem Investigation | Validate hypotheses about macro conversion problems, find root causes | Traffic problems → source-filtered page reports; technical problems → event reports; content/visual issues → in-page analysis + heatmaps (CrazyEgg); navigation issues → navigation summary |
| ③ Triangulation | Use quantitative data to supplement and validate qualitative findings | Users cannot find information → search term reports; expected links not clicked → navigation summary + in-page analysis; users unwilling to fill forms → event reports |
Measurement Plan Structure: Goal/macro conversion (50 consultation leads per month) → Expected actions/micro conversions (visit service page, read content, download whitepaper) → Web metrics (unique page views, average time on page, events).
Ask "Why" First (Work backward from goals to metrics, do not start from easy-to-measure metrics): Common mistake is treating "reduce bounce rate" as a goal — bounce rate is not a goal; the real goal is "make content more engaging + help the right users find what they need". Correct process: ① Clarify high-level goals → ② Identify behavioral signals (read to the end, interact with components, land on correct page from search) → ③ Select measurement methods (scroll heatmaps, event tracking, source/keyword analysis, return visit rate). For videos, do not just count plays; ask for play ratio(plays / unique page visits) and conversion rate difference between viewers and non-viewers.
Core principle: Design success is not about more clicks/PVs, but about UX improvement (saving time, simplifying flows). Adding a bulk edit feature → you should measure decreased task time, increased completion rate, decreased errors, increased engagement, not just count feature usage. Used poorly, Analytics becomes a "data black hole that consumes resources without yielding insights."
IX. 7 Steps of UX Benchmarking (Connect UX to Business Goals)
7 Steps to Benchmark Your Product's UX + Benchmarking UX: Tracking Metrics Merged.Benchmarking = evaluating UX with quantitative metrics and comparing them against meaningful standards; it is a form of summative evaluation(conducted at the end of a design cycle), typically requiring quantitative usability testing/analytics/surveys, with repeated testing across multiple versions to track progress.
4 types of comparison benchmarks: against earlier versions (58 seconds → 43 seconds) / against competitors (our form completion rate 86% vs competitor 62%) / against industry standards (account creation ease of use 5.3/7 vs industry 6.5/7; average hotel website NPS 13%) / against target values (8-week retention 8% vs target 15%).
7 steps: ① Choose what to test (define product type/target users/key tasks; for metrics refer to Google's HEART framework: Happiness / Engagement / Adoption / Retention / Task success) ② Decide how to test (quantitative testing + survey / analytics / survey, based on time/cost/skills/tools) ③ Collect first-round data to establish a baseline (run a pilot test first, avoid major promotions/economic fluctuations) ④ Redesign (apply the 10 usability heuristics) ⑤ Retest (analyze data; testing can be run continuously; for task-related data give users an adaptation period: 2–3 weeks for daily-use products, 4–5 weeks for weekly-use products) ⑥ Interpret findings (statistical validation + contextualization) ⑦ Calculate ROI (optional, connect UX metrics to KPIs).
Key case (task time barely changed but the redesign was successful): Smart speaker setup — average task time 6.28→6.32 minutes (almost unchanged), but success rate 70%→95%, SEQ ease of use 5.4→6.2, indicating the redesign was successful.Core principle: Don't fixate on a single metric; look at a set of metrics; whether a task time is good or bad depends on the specific task.
X. Data Quality: Don't Be Fooled by a Single Data Point + Inherent Risks of Quantitative Research
6 Dimensions for Assessing Usability Data + Risks of Quantitative Studies merged — one covers "how to judge if a data point is credible", the other covers "why over-superstition in numbers can be harmful".
6 dimensions for evaluating data points(why AI struggles with usability analysis: it lacks contextual judgment):
| Dimension | What to check |
|---|---|
| Authenticity | Is the feedback genuine? Is the participant pleasing the moderator / being forced to give a rating? |
| Consistency | Are words and actions consistent? (Saying "it's easy" but making errors and retrying → trust behavior over words) |
| Repetition | How often does the same feedback/error occur? Multiple people making the same mistake = a real problem |
| Spontaneity | Was it spontaneous or guided? (Avoid revealing the purpose too early, or implying features in questions) |
| Appropriateness | Are users representative? Are tasks close to real-world scenarios? |
| Confounds | Are there task order effects? Are instructions ambiguous and affecting completion? |
Counterexample: 4 out of 5 people said they liked a feature → it seems to test well, but if ① actually only 1 person really used it and encountered difficulties ② the positive feedback only came after the moderator actively asked ③ no one mentioned it again in follow-up discussion ④ users did not compare other options — this "4/5 like" result is not credible.Words and actions often do not match; multiple data points + cross-validation with multiple methods are needed.
6 pitfalls of "number superstition" in quantitative research(Risks of Quantitative Studies):
- Random results: p<0.05 means randomness < 5%, but 1 out of every 20 analyses may still produce a purely random result.
- Arbitrary correlation (data dredging): The more variables, the more accidental correlations — 7 variables produce 21 potential correlations;53 variables → 1378 possible correlations, of which about 69 will be "significant" but have no real connection. You must define hypotheses before the study, rather than "dredging" for significance after the fact.
- Ignoring covariates: Apparent correlations may be driven by third variables (intelligence and birth order are actually determined by parental economic/educational status; long link text correlates with success rate, but the real cause is whether user-centered design was followed).
- Overly simplistic analysis: Overly controlled experiments / using college student samples → conclusions cannot be transferred to real-world scenarios.
- Measurement distortion: Poor task design/guidance skews results (when testing ads, asking users to rate web design → forces users to look at ads more).
- Publication bias: Journals prefer "novel" findings over stable replications.
Conclusion (NNGroup's consistent position): Quantitative research requires high-level expertise + strict execution, is high-cost, and easily misleading;qualitative research is faster, deeper, and more cost-effective(5 users can find most problems, and even flawed methods can provide reliable insights). Three questions to judge anomalous results: Does it rely on arbitrary correlation? Is it affected by design/bias? Is it hyped only because it is novel? Characteristics of reliable research = replicable + consistent conclusions across multiple studies (e.g., Nielsen's "discount usability engineering/5-user testing" was initially questioned, but became accepted practice after repeated replications). When resources are limited, prioritize qualitative research; reserve quantitative research for complex problems or when precise data is needed.
XI. IA Research Methods: Card Sorting + Tree Testing (Quantified Findability)
Card Sorting + Quantifying UX Improvements: A Case Study merged — card sorting builds structure, tree testing quantitatively validates it.
Card Sorting: Participants group labeled cards (pages/links/images) according to their own understanding, revealing users' mental models of information architecture (IA)mental models(sportswear example: grouping hoodies + tank tops as "tops", gloves + hats as "accessories").
| Sort type | Definition | Applicable scenario |
|---|---|---|
| Open | Users create their own groups and labels | Exploring natural categorization, discovering new IA patterns |
| Closed | Users sort cards into preset categories | Validating existing IA (it is more recommended to use tree testing) |
| Hybrid | Partially preset + allows user-created groups | When some categories are clear but others still need exploration |
Implementation: Cards30–50 cards(more than 50 causes fatigue and "miscellaneous" categories); labels should be concise, do not use brand names as clues (avoid "Toyota Corolla" making users only recognize the brand); run a pilot test first. Recruitment: qualitative ≥15 participants(to observe grouping logic), quantitative ≥30–50 participants(for statistical significance). Tools such as OptimalSort automatically generate similarity matrices/dendrograms/tree diagrams. Limitations: lacks context (only looks at individual content items, ignores navigation cues), only single-level, fatigue with over 50 cards, large mental model differences make interpretation difficult.
Tree Testing (quantification of findability) + case Marketade × Baileigh Industrial(B2B metal/woodworking machinery, single orders often >$15,000):
- Background: Sales interviews found that customers frequently called because they could not find information due to poor website IA, taking time away from high-value sales work.
- Qualitative: Usability testing found too many top-level categories and unintuitive subcategory names.
- Quantitative: Used Treejack for tree testing,64 participants, 8 top-level category findability tasks, initial navigation average score4/10(only 34% took the correct path for one task).
- IA revision: Redesigned via card sorting — simplified top-level categories, renamed ambiguous subcategories, added contextual descriptions for tools.
- Result: New navigation average score7.4/10, an improvement of 85%85% compared to 4/10; 6 out of 8 tasks scored ≥7. Revenue and leads rose sharply after launch.
- Takeaway: Baileigh's 85% improvement was higher than Marketade's recent average improvement75% — complex design problems often conceal huge room for improvement.Quantitative UX metrics are effective tools for evaluating and demonstrating results to clients, and must be combined with qualitative methods (finding problems) + quantitative methods (quantitative validation).
Sourced from NNGroup Topics / Quantitative Research (28 files / deduplicated and merged into 24 topic articles). For qualitative methods and the other side of the 5-user rule, see NNGroup Qualitative Research and Research Methods Toolbox (Method Landscape/Interviews/Empathy Maps/Sample Size/Task Design/Moderation/Expert Evaluation/Workshops/Ethics); for compressing digital users into empathizable individuals, see NNGroup Personas (Essence/Three Types/Scope Hierarchies/Creation/Comparison with JTBD-Prototypes-Segmentation/Application/Maintenance/5 Major Failures); it complements Will's notes for the Google UX Certificate course Google UX Certificate / UX Research and Empathy-Define (the HEART framework is also covered there). Images/original files are retained in sources.
Sources and related materials
- NNGroup Qualitative Research and Research Methods Toolbox (Method Landscape/Interviews/Empathy Maps/Sample Size/Task Design/Moderation/Expert Evaluation/Workshops/Ethics)
- NNGroup Personas (Essence/Three Types/Scope Hierarchies/Creation/Comparison with JTBD-Prototypes-Segmentation/Application/Maintenance/5 Major Failures)
- Google UX Certificate / UX Research and Empathy-Define