Quantitative UX Methods: How to Read and Interpret Numbers Correctly

"Is our new design better?" sounds like five usability tests and a week of analysis. In reality, the question requires an A/B test with a sufficiently large sample size; otherwise, any answer remains speculation. "How satisfied are our users?" sounds like a few interviews. What’s needed is SUS, NPS, or CSAT on a representative sample; only then can the results be compared later and tracked over time. And “Why do users abandon the checkout process?” sounds like torturous interviews, even though funnel analysis and A/B testing must first show where and how often this happens. The “why” comes only afterward.

Quantitative UX methods answer “how many” and “how often,” not “why.” Collecting the numbers is usually the easier part. The challenge comes when interpreting them: If you confuse mean, significance, and correlation, you’ll make the wrong decisions—even if the data is sound.

It is precisely this interpretation that is missing from most method overviews. I’ve been analyzing quantitative UX data for over 25 years, and the mistake I encounter most frequently rarely occurs during data collection. It happens afterward, at the desk, when interpreting the table. This article explains what matters most: from the correct sample size to the mean and median, to p-values, confidence intervals, and a brief look at multivariate methods.

📌 Key Takeaways

• Qualitative research answers the “why,” while quantitative research answers the “how many.” The two complement each other; neither replaces the other.

• The sample size is determined by the research question: 30 for a cell comparison, 100 as a minimum, 400 for robust results.

• For skewed distributions, the median is more informative than the mean. Look at the distribution first, then do the math.

• A p-value below 0.05 indicates that a difference is likely real. However, this does not necessarily mean it is relevant.

• Confidence intervals indicate the uncertainty of a measurement and are often more meaningful than the p-value alone.

• The average SUS score is 68 (Sauro, 2018). Only by comparing it to others can you put your own score into perspective.

• Correlation indicates an association, not a cause. To establish causality, you need an A/B test.

When should you use qualitative methods, and when should you use quantitative methods?

Qualitative and quantitative methods answer different questions. Qualitative research explores the “why” and “how”: deep insights from interviews, observations, or diary studies, usually with small samples, open to surprises, and highly interpretive. Quantitative research answers “how many” and “how often”: broad data from surveys, A/B tests, or analytics, with large samples, structured, hypothesis-driven, and statistically analyzable.

A key phrase from my workshops sums it up: What do users do? We observe that. Why do they do it? We ask that through qualitative research. How many do it? We measure that quantitatively.

Practical example: When it comes to checkout abandonment, funnel analysis first shows at which step and how often users drop off. The interview that follows clarifies why. Guessing first and then looking for numbers, on the other hand, costs time and often leads in the wrong direction.

What makes a quantitative study scientifically sound?

Three concepts determine whether a quantitative study is meaningful at all: hypothesis, variable, and operationalization. The hypothesis is a testable assumption, for example: Users find the checkout button faster when it’s green instead of gray. The variable is what is measured or changed; in this case, the button color is the independent variable and the time taken to find it is the dependent variable. Operationalization means making something abstract measurable: “satisfaction” becomes a Likert scale from 1 to 5 or a SUS score.

For the results to be meaningful, three additional quality criteria are required. Validity examines whether the measurement actually captures what it is intended to measure. For example, when a survey asks about “intention to use” but actually refers to actual behavior. Reliability examines whether repeating the measurement yields similar results; the SUS score is considered stable over time. Objectivity examines whether the result is independent of the researcher; standardized questionnaires perform better in this regard than free interpretation.

How large does your sample size really need to be?

The sample size should be determined by the research question, never by habit, budget, or some arbitrary rule of thumb. As a general guideline, 30 is recommended for a cell comparison between two groups, 100 is the minimum for a target group, and 400 is needed if the results are to be truly robust.

How the sample is assembled is a second decision. In a random sample, every person has an equal chance of being included is the gold standard, but often difficult to implement in practice. Quota sampling selects participants based on specific characteristics such as age or gender and ensures a representative composition. With convenience sampling, you select whoever is currently available because it’s quick, but you should exercise caution when making any generalizations. Snowball sampling allows participants to recommend others and is suitable for hard-to-reach target groups.

The recruitment method itself also introduces its own bias, which you should be aware of:

No recruitment method is perfect. The most helpful approach is to be aware of your own biases and openly acknowledge them in the report rather than concealing them.

Combining data sources is also worthwhile: Surveys provide insights into attitudes, satisfaction, NPS, or SUS; web analytics show behavior, click paths, and bounce rates; A/B tests provide conversion and click-through rates; and support data provide tickets, FAQ views, and chatbot logs. Triangulating multiple sources gives you a more reliable picture than relying on a single source.

Mean or Median: Which Tells You More About Your Users?

Descriptive statistics transform raw data into understandable metrics and visualizations. Instead of noting “4.2, 3.8, 4.5, 4.1, 3.9,” you say “an average of 4.1 out of 5 stars.” Three questions are central here: What is the typical value, how widely do the data points vary, and what does the distribution look like?

The mean is the sum of all values divided by their number. For task times of 45, 52, 48, 51, and 49 seconds, this results in an average of 49 seconds; however, the mean is sensitive to outliers. The median is the value exactly in the middle and is significantly more robust against skewed distributions. For salaries of 40,000, 45,000, 50,000, 55,000, and 200,000 euros, the median is 50,000 euros, not the 78,000 euros that the mean would indicate. Finally, the mode is the most frequent value; for ratings of 4, 5, 4, 3, 4, 5, and 4 stars, this is 4, making it ideal for categorical data.

The range (maximum minus minimum, in the case of task times ranging from 30 to 120 seconds, that’s 90 seconds) or the standard deviation reveals how widely the values are spread. Two teams demonstrate why this matters:

Same mean, completely different reality. Team A performs consistently, while Team B fluctuates greatly, even though the average appears identical. Without a measure of dispersion, this remains invisible.

The shape of the distribution also determines which metric makes sense in the first place. In a normal distribution, the mean, median, and mode are practically the same. As is the case with response times or IQ scores. Right-skewed distributions, such as those commonly found in task times, income, or page views, pull the mean upward while the median remains stable. Bimodal distributions with two peaks, such as those involving experts and beginners in a sample, render the mean almost useless: it falls in the valley between the two groups and represents no one. The rule behind this remains simple: look at the distribution first, then do the math.

The Most Common Mistakes in Descriptive Statistics

The most common mistake is reporting only the mean. The median, measure of dispersion, and a visualization almost always tell us more. The second most common: percentages without a basis. “80% are satisfied” sounds good, but it means nothing if no one knows how many people that figure is based on. “80% satisfied (n=150),” on the other hand, can be contextualized and compared later. A third classic mistake is the manipulated Y-axis: if it doesn’t start at zero, small differences appear huge. And fourth, perhaps the most costly mistake: correlation is presented as causation, when in reality an A/B test would be the only way to clarify this.

When is a correlation meaningful, and when is it not?

Correlation measures the strength and direction of the relationship between two variables on a scale from -1 to +1. Zero means no relationship, +1 means a perfect positive relationship, and -1 means a perfect negative relationship. In UX practice, values such as r = 0.72 between the SUS score and repurchase intent, r = -0.65 between the number of errors and satisfaction, or r = 0.12 between age and NPS indicate virtually no relationship. As a rule of thumb, a value below 0.3 is considered weak, between 0.3 and 0.5 moderate, and above 0.5 strong.

What correlation does not show is cause and effect. Ice cream sales correlate with the number of sunburns, but they do not cause them; on hot days, both simply increase. Anyone who wants to derive an action from a correlation needs an A/B test, not observational data.

What does a p-value of 0.03 really mean?

Descriptive statistics describe what happened in your sample. Inferential statistics go a step further and ask whether this also applies to everyone else. An example: Version A of your checkout converts at 4.2%, Version B at 4.8%, a 14% increase. But if you flip a coin 100 times, it won’t land exactly 50 times on heads. So perhaps the difference between A and B is just random variation. That’s exactly what inferential statistics calculates: the probability that an observed difference arose solely by chance.

This probability is the p-value. It answers the question: How likely would this result be if there were actually no difference at all? A p-value of 0.03 means that a difference of this magnitude would occur in only 3% of all cases if both versions were actually the same. A threshold of p < 0.05 has become established as the significance level. It is important to note that a p-value above 0.05 does not mean that there is no difference; it simply means that the difference could not be detected with this sample.

The t-test is typically used to compare the means of two groups for metric data such as times or scores. If different individuals in each group test the old design against the new one, this is called an independent t-test, as in a classic A/B test. If the same individuals test both versions one after the other in a before-and-after comparison, it is a paired t-test. A real-world example: Group A, using the old design, takes an average of 68 seconds, while Group B, using the new design, takes 52 seconds; with p = 0.008, the difference of 16 seconds is statistically significant.

Is every significant result also relevant?

No, and that’s where many reports are misleading.

With very large sample sizes, even tiny, practically insignificant differences become statistically significant. With smaller sample sizes, however, a result can be both significant and genuinely noticeable. Significance indicates whether a difference is likely to be real. Whether it’s worth pursuing is a question that must be answered in practice, not by the p-value.

What do confidence intervals tell you that a p-value doesn’t?

A confidence interval is the range within which the true value lies with 95% probability. “SUS score 72, 95% CI 68 to 76” means: The actual score for the entire population is very likely to fall between 68 and 76. To put this in context: According to an analysis by Jeff Sauro based on over 10,000 responses and hundreds of products, the average SUS score is 68 (Sauro, MeasuringU, 2018); a score of 75 already corresponds to the 73rd percentile, while a score of 52 corresponds to only the 15th percentile. The example score of 72 from the confidence interval is thus slightly above the average of all systems ever measured.

Confidence intervals also show how uncertain a measurement is. A wide interval indicates a high degree of uncertainty and usually means that more data should be collected. If the confidence intervals of two groups do not overlap, the difference is significant without the need for a separate calculation. In practice, confidence intervals are often more meaningful than a single p-value because they show a range rather than just a yes or no.

How can multivariate analyses help you when multiple factors are at play simultaneously?

UX is rarely influenced by just one factor. “Older users take longer” is a simple statement, but it overlooks how much each factor actually contributes. Four methods can help here, summarized concisely.

Regression shows which factors influence a result and to what extent. With an NPS score as the target variable, load time could be included in the model at minus 0.4 points per second, support quality at plus 2.1 points per rating point, and the number of features at plus 0.3 per feature. An R-squared value of 0.65 means that these factors explain 65% of the variation in the NPS; the remainder depends on other, unaccounted-for variables. For prioritization, this indicates where optimization is most worthwhile.

Factor analysis reduces many individual questions to a few underlying factors. For example, ten UX questions can be condensed into three factors such as efficiency, aesthetics, and trust. The SUS itself is a well-known example of this: its ten questions load onto two factors, usability and learnability.

Cluster analysis automatically divides users into similar groups based on behavior or responses such as power users, casual users, and struggling users. This aids in creating data-driven personas, targeted optimization, onboarding, and the allocation of support resources.

Finally, the MaxDiff analysis allows users to select the most important and least important elements from a small selection over several rounds. This generates priorities without the biases inherent in traditional rating scales, such as midpoint bias or politeness responses. The output consists of utility scores for each element for example, “Navigation” with a high priority of +1.8 and “Export” with a low priority of -1.4. This is useful for feature roadmaps or for realistically weighting product attributes.

How to Improve Your Next Quantitative Analysis

Numbers are only as good as their interpretation. If you look at the distribution first and only then calculate a metric, you’ll automatically avoid half of the common mistakes. If you clearly distinguish between significance and relevance, you won’t sell a 0.02% increase in clicks as a success to anyone. And if you’re aware of your own bias and openly acknowledge it, you’ll make your results more credible, not weaker.

Specifically for your next study: Never report percentages without the sample size (n), never present a p-value without a confidence interval, and for every new metric, briefly ask what distribution actually lies behind it. This takes a few extra minutes per report but saves you from discussions that would otherwise take much longer.

If your team wants to dive deeper into this: In the workshop “Overview of Quantitative Methods for UX Professionals,” we’ll walk through exactly these methods together using your own dataset.

You can find more on the topic of research quality in “UX Research Quality: Why Good Intentions Aren’t Enough.”

And when it comes to the next step, defending these numbers to management, the article “UX Research as Risk Management: Why We Finally Need to Change Our Language” will help.

Frequently Asked Questions About Quantitative UX Methods

How many participants do I need for a quantitative UX study?

That depends on the question, not on a fixed rule. To compare two groups, 30 people per group is often sufficient; for a single target group, 100 is considered the minimum; and if you really need robust, generalizable results, you should aim for about 400.

Mean or median—which should I use?

First, look at the distribution. For symmetrical data without outliers, the mean is sufficient. For skewed distributions such as salaries or loading times with a few extreme outliers the median provides a much more reliable indication of the typical value.

What does a p-value of 0.03 mean?

It means that an observed difference would occur in only 3% of all cases if there were actually no difference at all. Below the standard threshold of 0.05, the result is considered statistically significant meaning it’s likely genuine and not just a coincidence.

Is a statistically significant result always relevant?

No. With very large samples, even tiny, practically meaningless differences become statistically significant. For example, 0.02% more clicks among 100,000 users. Whether a result is worthwhile is determined by practical application, not by the p-value alone.

What is considered a good SUS score?

The average across thousands of measured systems is 68; anything above that is above average (Sauro, MeasuringU, 2018). A score of 75 corresponds to the 73rd percentile, while a score of 52 corresponds to only the 15th percentile. Without this benchmark, it is difficult to interpret a single SUS score.