UX Research Quality: Why Good Intentions Aren’t Enough

Imagine this: Your team has invested three weeks in a user study. A 50-page report, neatly formatted, complete with quotes and journey maps. The Product Owner nods politely. The findings disappear into a Confluence folder. The next product decision is still made on a hunch.

Sound familiar?

The demand for UX research has been rising for years. At the same time, trust in the results is declining. This is no coincidence, but a quality issue. And it affects the entire industry.

We’ve already touched on this topic in a LinkedIn article. Today, we’re diving deeper.

In this article, you’ll learn where typical quality pitfalls lurk in the research process, what it really costs the company, and what principles distinguish robust UX research quality from expensive occupational therapy. As a UX consultant, I’ve been helping companies with exactly these issues since 1999. Most problems don’t stem from incompetence. They stem from habit.

📌 Key takeaways

• UX research quality often fails before the first question is even asked: at the briefing stage.

• Small, homogeneous samples create a false sense of accuracy that puts million-dollar decisions on thin ice.

• If the method doesn’t fit the research question, even flawless execution won’t yield usable results.

• Reports that no one reads are a waste of budget. Not a communication problem.

• Took-it-and-ran research is more expensive than no research at all because it creates a false sense of security.

• Reliable UX research quality is based on triangulation, method fit, and transparency.

• Quality assurance is the prerequisite for research to be taken seriously.

  
  

Why does UX research so often fail to deliver usable results?

Because the errors don’t occur during analysis, but much earlier. In my work as a UX consultant, I see four points in the process where UX research quality systematically breaks down. Most of the time, no one notices.

The briefing problem: Research without a goal

“We want to know how users feel about our product.” That’s not a research brief. That’s a wish.

An example that is representative of many projects (fictional, but typical of what I regularly experience): A marketing team commissions a “user study” for the new website. Three weeks later, a 50-page report is delivered. The key takeaway? “Users prefer intuitive navigation.” Money burned, time wasted, zero insights gained.

It’s not that the execution lacks diligence. What’s missing are research questions. Hypotheses. Defined success metrics. Without a clear goal, research only delivers something useful by chance. “By chance” is not a standard we should accept as an industry.

Any study without a precise research question produces data that no one can translate into decisions. This undermines not only the current project but also trust in all future ones.

The sample trap: Who do we survey, and who do we never survey?

Jakob Nielsen’s rule “Five users are enough” is probably the most cited and most frequently misunderstood principle in our industry. Nielsen was referring to iterative usability testing to identify interface problems—not to user research or market validation.

Nevertheless, it’s turned into a free pass for tiny, homogeneous samples.

A fintech startup (fictional example) tests its new investment app with five tech-savvy millennials from Berlin. Based on this “research,” the app is launched for the entire German-speaking market. It flops completely with older target groups. Surprised? Hardly.

Instead of representative samples, teams often resort to whatever is available: their own employees, existing customer pools, newsletter subscribers. Each of these groups is systematically biased. The employee bias is particularly insidious. A software company tests its CRM software with its own sales staff (fictional example). These people use the product daily, know all the workarounds, and rate features more positively than external users ever would. Neutral evaluation? Out of the question.

Product decisions based on this are not “roughly correct.” They are systematically wrong—in a direction you don’t know.

Methodological Mismatch: When the Tool Doesn’t Fit the Question

Usability tests and qualitative interviews dominate everyday research. Not because they’re always the right choice, but because they’re the only things many researchers know how to do.

An e-commerce company wants to understand why users abandon their carts at checkout (fictional example). The appropriate approach: a quantitative analysis of drop-off points, combined with qualitative interviews. Instead, only a usability test with five people is conducted. The results remain superficial. The real problem remains hidden.

When was the last time you conducted a contextual inquiry? Conducted a diary study over several weeks? Used card sorting for information architecture? Systematically applied quantitative methods like A/B testing or conjoint analysis (a method for evaluating product attributes through systematic comparison)?

These aren’t rhetorical questions. Methodological impoverishment leads us to answer only the questions our limited methods can address—not the questions that drive the business. A usability test does not provide an answer to purchase motivation. Yet the results are treated as if they do.

The consequence: bad decisions that look like data-driven decisions.

The reporting problem: insights that reach no one

Good research that no one reads is worthless research.

Reports are written, presented. And forgotten. Insights end up on Confluence pages that no one opens, or in presentations that are irrelevant after the meeting. Add to this the HiPPO effect, the “Highest Paid Person’s Opinion” (decision-making by the highest-ranking person rather than by data). A team invests weeks in careful research and delivers well-founded recommendations. The manager decides otherwise because they have “a hunch.”

Even more problematic: research as a fig leaf. The decision has already been made, but “research” is conducted anyway to appear data-driven. Results are cherry-picked; inconvenient insights are ignored. The timing is manipulated. Research is scheduled so late that only confirmatory results are “helpful.”

When this happens systematically, the research team’s motivation declines. The quality of the next study suffers. Decision-makers feel vindicated: “Research doesn’t bring anything new anyway.” A downward spiral.

What does poor UX research quality really cost?

More than most companies realize.

The obvious costs: wasted budget on studies with no actionable results, lost time for teams producing data instead of decision-making foundations, opportunity costs from product decisions built on shaky ground.

The real costs are structural. Poor research quality creates a downward spiral. Weak methodology yields weak insights. Weak insights lead to product decisions that don’t pan out. This undermines trust in UX research. Less trust means less budget and less time for the next study. Quality continues to decline.

The question for decision-makers isn’t “Can we afford research?” But rather: “Did the last study actually change a decision?” If the honest answer is “No,” the problem rarely lies with the research method itself. It lies in the quality of the actual implementation.

“Alibi research”—that is, research conducted simply because it’s expected—is more expensive than no research at all. It burns through the budget, ties up resources, and creates a false sense of security that is riskier than honest uncertainty.

What principles make UX research quality measurable?

Reliable UX research quality is no accident. It’s based on principles that any team can implement.

Clear research questions before every study. Before you start, define precisely what you want to know. Formulate testable hypotheses. Determine which results would trigger a decision in which direction. Sounds obvious. Happens surprisingly rarely.

Representative samples instead of available samples. The most convenient target group is almost never the right one. Invest in recruitment that reflects your actual user base. Don’t forget the people who don’t use your product. That’s often where the most valuable insights lie (survivorship bias: the distortion that arises when only “survivors” are surveyed).

Methodological fit. Every research question requires a method that fits the question. Not every question can be answered with a usability test. A research team should be just as capable of conducting quantitative analyses as it is of conducting qualitative interviews.

Triangulation as standard. A single method is never enough for reliable conclusions. Validate quantitative data with qualitative insights. Confirm individual opinions through broader surveys. Only the combination of different data sources yields a robust picture.

Ensure objectivity. No self-tests. No tests with employees for external products. No research done as a favor where the result is already predetermined. External perspectives are invaluable, even if they are more expensive.

Transparent documentation. Document not only results, but also methods, limitations, and decision-making processes. This creates traceability and makes quality visible—even for stakeholders who aren’t immersed in day-to-day research.

What’s coming: Systematic checklists for research quality

Knowing the principles is the first step. Consistently implementing them in day-to-day project work is the second—and often the more difficult one.

At Uintent, we’re currently working on a set of systematic checklists that cover the entire research process: from risk assessment to study design and execution, all the way through to reporting. It’s not a magic bullet, but a tool that helps avoid the most common quality pitfalls before they happen.

Click here for our latest downloadable resources.

One final question

UX research is at a crossroads where we must decide. Do we accept the creeping erosion of quality and become purveyors of pseudo-insights? Or do we invest in meticulous craftsmanship, even if that’s more inconvenient?

The answer doesn’t lie in better tools or faster methods. It lies in the willingness to ask the right questions. Of our users, of our clients, and of ourselves.

An industry that doesn’t take its own standards seriously can’t expect others to take it seriously.

The question isn’t whether we can afford quality. The question is whether we can afford a loss of quality.

What’s your most common quality issue in your day-to-day research work? I’m curious. Share it in the comments.

FAQ:

As a decision-maker, how do I know if our UX research is robust? 

Ask about the research questions, the sample composition, and the study’s limitations. If your research team doesn’t have clear answers to these, that’s a red flag. Good research is transparent about its own limitations.

Are five test subjects enough for a usability test?

For iterative interface tests to identify usability issues: yes, that can be sufficient. For user research, market validation, or strategic decisions: no. The correct sample size depends on the research question.

What is the difference between token research and real research? 

Token research confirms decisions that have already been made. Genuine research is open-ended and can yield surprises. The litmus test: Would your team be willing to change the product strategy based on the results? If not, it’s token research.

How do I convince stakeholders to invest more in research quality? Not with abstract ROI formulas, but with a concrete question: “Did the last study change a decision? If not: why not?” This question opens a conversation about quality without coming across as accusatory.

Does every study have to perfectly meet all quality principles? 

No. But you should consciously decide which compromises you’re willing to make and document them. The difference between poor and pragmatic research lies in transparency about one’s own limitations.

💌 Want more? Then keep reading—in our newsletter. 

Comes' out four times a year. Sticks with you longer. https://www.uintent.com/de/newsletter