How Yupp Uses Feedback to Fairly Evaluate AI Models – And What UX Professionals Can Learn From It

The most important points in a nutshell:

• Yupp compares AI responses via crowd voting

• Users evaluate quality, speed and clarity

• Evaluations are statistically analysed as pair comparisons

• The VIBE score shows which model performs better in everyday use

• Bias is actively controlled through blind tests

• Segmentation shows: model selection depends on the context of use

• Practical model for UX testing methods

Introduction: What if feedback is the product?

Are you familiar with this? You ask ChatGPT, Claude or Gemini the same question – and get three completely different answers. Sometimes one is brilliant, sometimes totally off the mark. But who actually decides which one is ‘better’? And according to what criteria?

This is where Yupp.ai comes in. A platform that makes precisely such comparisons its principle. It shows how users can contribute to the evaluation of AI models by simply providing feedback. And what does that have to do with UX? A lot. Because many of the methods Yupp uses are familiar from our practice – only on a much larger scale.

I have been working as a UX consultant on global projects for many years. What fascinates me about Yupp is that the platform cleverly combines UX methodology and AI evaluation. And it's an excellent source of inspiration for your own testing processes.

How exactly does Yupp work?

Yupp is not a classic AI platform, but a ‘meta’ system: you enter a question and receive answers from several AI models. Your task: decide which answer you like better – and why.

The key point is that these evaluations are not simply incorporated into a star rating. Instead, Yupp uses the Bradley-Terry model – a pair comparison method that creates a consistent ranking from many individual decisions. The result: the VIBE score (‘Value Informed Benchmark Evaluation’) shows which model is the most convincing in a direct comparison.

What criteria are used for evaluation?

Yupp does not only evaluate based on ‘likability’. Several dimensions play a role:

• Answer quality: How clear, helpful and relevant is the answer?

• Answer speed: How quickly does the model respond?

• Cost: What does an answer cost, e.g. when using an API?

• Confidence: Does the model make clear statements or does it remain vague?

These values are analysed together with user feedback – depending on weighting and target group.

Practical example:

In an experiment with factual vs. creative prompts, Claude and GPT-4 performed differently: Claude was better at reasoning, GPT-4 was better at storytelling. However, the evaluation was not based solely on the length of the response or the facts, but on user perception.

What about bias? Can the evaluations be trusted?

Good question. Yupp actively tests for bias. For example, through blind tests: the model names are hidden so that users do not know whether the response comes from GPT-4 or Claude.

This reduces what is known as brand bias. At the same time, systematic differences between user groups are taken into account (e.g. beginners vs. AI power users).

UX parallel:

Blinding is also an important tool in usability research to avoid perception bias. Yupp applies this principle to AI evaluation – in a scalable and data-driven way.

Why segmentation is so important

Not every question is the same. That's why Yupp also analyses the context of the queries:

• Is it a factual question or a creative one?

• Is the questioner technically savvy or more of a layman?

This results in segment scores that show which model performs particularly well in which use cases. For us UX professionals, this is a clear lesson: blanket values are of little use. What matters is performance in the context of use.

Example:

A model may be very good on average – but fail when it comes to accessible applications or sensitive health issues. Yupp makes such differences visible.

What happens to the feedback?

This is where it gets exciting: feedback is not an add-on at Yupp – it is the product. The platform sells anonymised evaluation data to AI providers, who use it to improve their models. In return, users receive Yupp credits that can be cashed out (max. £50/month).

This means that users become real data suppliers – fairly remunerated and transparent. This is also an interesting idea for the UX industry: what if user feedback were not only collected, but also used strategically and monetarily?

FAQ: What UX teams want to know about Yupp

1. Do I need programming skills to use Yupp?

 No. The platform is very low-threshold. Enter your question, compare answers, done.

2. How many models are compared?

Usually two to four per request. Mostly GPT-4, Claude, Gemini and Grok are included.

3. Can I also give feedback anonymously?

Yes. You only need an account, but your ratings are not stored in a personalised manner.

4. Is there an API for my own tests?

Not officially yet. However, Yupp plans to offer evaluation-as-a-service for companies.

5. How does this benefit me as a UX team?

 Yupp is a source of inspiration: for evaluation logic, bias checks, segment analyses and feedback systems – all topics that UX teams deal with on a daily basis.

Conclusion: What we as a UX community can learn from Yupp

Yupp shows that user-centred feedback is possible on a large scale – without losing depth. The platform uses methods we know from UX practice and brings them to a scalable, evaluable level.

UX teams would do well to take a look at Yupp in order to:

• reflect on their own testing processes,

• get new ideas for AI evaluations,

• and make better decisions when choosing models.

Want to systematically test your own prompts? Or understand how other models perform? Then take a look at Yupp.ai. 

💌 Not enough? Then read on – in our newsletter. It comes four times a year. Sticks in your mind longer. To subscribe: https://www.uintent.com/newsletter