NNGroup Quantitative Research (A/B Testing · Sample Size · Confidence Interval · UX Metrics · Conversion Analysis · Benchmarking · Learning Curve)
Nielsen Norman Group quantitative research knowledge base: 24 curated pieces (merged and deduplicated from 28 files). Quantitative research = compressing user experience into comparable numbers (success rate / task time / conversion rate / satisfaction score) to track progress, make tradeoffs, and prove ROI to stakeholders — but numbers are fragile and require professional execution, otherwise they will mislead design. Core methods: A/B Testing (randomly assign groups to test overall direction · only one variable at a time · requires high traffic · cannot identify causes, must be paired with qualitative research)/ Multivariate Testing MVT (tests interactions of multiple variable combinations · optimizes details · number of combinations = product of variant counts, which explodes)/ Key statistics: statistical significance ≠ practical significance (p-value = difference ÷ random noise · large samples produce 'significant but irrelevant' results)/ confidence interval + margin of error + confidence level (95%/15% is the default balance point for UX · interval width = 2 × margin of error)/ Sample size: quantitative studies need 40 participants (for binary metrics at 95% confidence +15% margin of error · formula Z=1.96 p=0.5), not the 5 participants used in qualitative studies (which find 85% of problems · based on the 31% problem probability assumption)/ Type I false positive + Type II false negative / measurement error / quantitative risks (number worship · spurious correlation · covariates · publication bias). Metrics and scales: success rate (simplest binary metric · success levels use ordinal variables and cannot be numerically averaged)/ learnability + power law of learning (learning curve · redesign resets the curve and causes productivity drops)/ NPS Net Promoter Score (9-10 promoters / 0-6 detractors / formula · customer loyalty ≠ usability · 6 major limitations)/ Likert scale vs semantic differential scale + SUS / vanity metrics require added context / macro conversion vs micro conversion / HEART framework. Analysis and benchmarking: three uses of Analytics (problem indication · investigation · triangulation)/ UX goals → measurement plan (ask why first · set goals before selecting metrics)/ 7-step UX benchmarking + 6 dimensions of data quality (authentic/consistent/repeated/spontaneous/appropriate/unobtrusive)/ card sorting + tree testing (case study of 85% findability improvement).