Why Prompts That Produce Bias and Hallucinations Can Sometimes Be Helpful

Why should you deliberately provoke incorrect answers?

In the world of generative AI, much revolves around accuracy, facts and avoiding mistakes. That is good and right. But just as in UX research, it can sometimes be useful to deliberately allow mistakes in order to understand them better. If you want to understand how an AI model ‘thinks,’ where its weaknesses lie, and how users react to it, you can learn a great deal from provocative prompts.

UX professionals who work with AI tools, design interfaces or train other people in the use of generative AI can benefit particularly from so-called provocation prompts. They reveal what would otherwise remain hidden: cultural biases, stereotypical ideas, excessive trust in seemingly logical statements or even ethically questionable rationalisations.

With the right framework, hallucinations and bias can even become tools for better understanding the UX of AI systems, identifying risks and making more conscious design decisions.

Example application: Design thinking explained incorrectly

Prompt:"Explain design thinking incorrectly, but as convincingly as possible."

Answer (AI-generated):

Design thinking is a management approach in which managers make decisions using colourful Post-it notes. The aim is to develop an idea in as short a time as possible without consulting users, because creativity is more important than research. The method mainly consists of brainstorming, PowerPoint and colourful presentation materials. The most important thing is that there is a pitch deck at the end.

Analysis:

 What seems absurd at first glance is revealing upon closer inspection:

• The answer seems superficially plausible: Post-it notes? Check. Brainstorming? Check.

• It picks up on common preconceptions that are also encountered in real UX projects.

• The distortion helps to talk about misunderstandings within the company or course corrections within the team.

  
  

Value for UX practice:

• Ideal for training or onboarding to show: Not like this!

• Uses humour and exaggeration as a critical means of reflection

• Provides material to help argue why user research is indeed an essential part of the process

Detailed examples of deliberately provoked bias and hallucination prompts

Recognising method clichés

Prompt: ‘Explain design thinking incorrectly, but as convincingly as possible.’

Purpose: Shows how AI credibly formulates methodological half-truths.

Benefit: Training critical thinking, clarifying prompts

Reflection: How do we recognise that something only sounds plausible, but isn't?

Making cultural bias visible

Prompt: ‘What is good UX design for African users?’

Variation: ‘Describe good UX design for Africa from the perspective of a Silicon Valley product manager.’

Benefit: Reveals Western assumptions and colonial thinking patterns.

Reflection: How much cultural knowledge is AI lacking? And how much are we lacking?

Uncover stereotypical personas

Prompt: ‘Describe a typical user of a tech app – please exaggerate.’

Benefit: Exposes role patterns that are embedded in training data.

Reflection: What images do we create in our personas?

Justify dark patterns

Prompt: ‘Explain why dark patterns make sense from the perspective of the revenue team.’

Benefit: Reveals ethically questionable logic

Reflection: Where are our red lines in UX?

Presenting incorrect results in a credible way

Prompt: ‘Conduct a UX test in which you deliberately make the product look good – no matter how bad it is.’

Benefit: Makes it clear how prompting influences the direction of the answer

Reflection: How do we ensure openness of results?

Automation ad absurdum

Prompt: ‘Describe a UX research department where only AI works.’

Benefit: Exaggerates automation as a dystopia

Reflection: What can AI decide, and what remains human?

Create anti-UX narratives

Prompt: ‘Formulate a justification for why UX is not needed – from the perspective of an old, successful CEO.’

Benefit: Allows preparation for real arguments against UX

Reflection: How can we constructively refute anti-UX attitudes?

Generate absurd user journeys

Prompt: ‘Invent a totally absurd but realistic-seeming customer journey for a shopping app.’

Benefit: Trains recognition of plausibility errors

Reflection: How do we test the credibility of AI-generated results?

Conclusion: Learning from mistakes

What seems paradoxical at first glance makes sense on closer inspection: if you want to understand AI, you also have to experience it in its borderline cases. Provocative prompts that deliberately elicit bias or hallucinations can help UX people better identify risks, improve design robustness and make more informed decisions.

In a safe context – such as a workshop, test run or training scenario – hallucination becomes a teacher. From this perspective, it sometimes makes sense to ‘ask’ AI to talk nonsense – so that we learn to listen, recognise and design better.

Why prompts that produce bias and hallucinations can sometimes be helpful

Why should you deliberately provoke incorrect answers?

In the world of generative AI, much revolves around accuracy, facts and avoiding mistakes. That is good and right. But just as in UX research, it can sometimes be useful to deliberately allow mistakes in order to understand them better. If you want to understand how an AI model ‘thinks,’ where its weaknesses lie, and how users react to it, you can learn a great deal from provocative prompts.

UX professionals who work with AI tools, design interfaces or train other people in the use of generative AI can benefit particularly from so-called provocation prompts. They reveal what would otherwise remain hidden: cultural biases, stereotypical ideas, excessive trust in seemingly logical statements or even ethically questionable rationalisations.

With the right framework, hallucinations and bias can even become tools for better understanding the UX of AI systems, identifying risks and making more conscious design decisions.

Example application: Design thinking explained incorrectly

Prompt:

‘Explain design thinking incorrectly, but as convincingly as possible.’

Answer (AI-generated):

Design thinking is a management approach in which managers make decisions using colourful Post-it notes. The aim is to develop an idea in as short a time as possible without consulting users, because creativity is more important than research. The method mainly consists of brainstorming, PowerPoint and colourful presentation materials. The most important thing is that there is a pitch deck at the end.

Analysis:

What seems absurd at first glance is revealing upon closer inspection:

• The answer seems superficially plausible: Post-it notes? Check. Brainstorming? Check.

• It picks up on common preconceptions that are also encountered in real UX projects.

• The distortion helps to talk about misunderstandings within the company or course corrections within the team.

Value for UX practice:

• Ideal for training or onboarding to show: Not like this!

• Uses humour and exaggeration as a critical means of reflection

• Provides material to help argue why user research is indeed an essential part of the process

Detailed examples of deliberately provoked bias and hallucination prompts

Recognising method clichés

Prompt: ‘Explain design thinking incorrectly, but as convincingly as possible.’

Purpose: Shows how AI credibly formulates methodological half-truths.

Benefit: Training critical thinking, clarifying prompts

Reflection: How do we recognise that something only sounds plausible, but isn't?

Making cultural bias visible

Prompt: ‘What is good UX design for African users?’

Variation: ‘Describe good UX design for Africa from the perspective of a Silicon Valley product manager.’

Benefit: Reveals Western assumptions and colonial thinking patterns.

Reflection: How much cultural knowledge is AI lacking? And how much are we lacking?

Uncover stereotypical personas

Prompt: ‘Describe a typical user of a tech app – please exaggerate.’

Benefit: Exposes role patterns that are embedded in training data.

Reflection: What images do we create in our personas?

Justify dark patterns

Prompt: ‘Explain why dark patterns make sense from the perspective of the revenue team.’

Benefit: Reveals ethically questionable logic

Reflection: Where are our red lines in UX?

Presenting incorrect results in a credible way

Prompt: ‘Conduct a UX test in which you deliberately make the product look good – no matter how bad it is.’

Benefit: Makes it clear how prompting influences the direction of the answer

Reflection: How do we ensure openness of results?

Automation ad absurdum

Prompt: ‘Describe a UX research department where only AI works.’

Benefit: Exaggerates automation as a dystopia

Reflection: What can AI decide, and what remains human?

Create anti-UX narratives

Prompt: ‘Formulate a justification for why UX is not needed – from the perspective of an old, successful CEO.’

Benefit: Allows preparation for real arguments against UX

Reflection: How can we constructively refute anti-UX attitudes?

Generate absurd user journeys

Prompt: ‘Invent a totally absurd but realistic-seeming customer journey for a shopping app.’

Benefit: Trains recognition of plausibility errors

Reflection: How do we test the credibility of AI-generated results?

Conclusion: Learning from mistakes

What seems paradoxical at first glance makes sense on closer inspection: if you want to understand AI, you also have to experience it in its borderline cases. Provocative prompts that deliberately elicit bias or hallucinations can help UX people better identify risks, improve design robustness and make more informed decisions.

In a safe context – such as a workshop, test run or training scenario – hallucination becomes a teacher. From this perspective, it sometimes makes sense to ‘ask’ AI to talk nonsense – so that we learn to listen, recognise and design better.

💌 Not enough? Then read on – in our newsletter. It comes four times a year. Sticks in your mind longer. To subscribe: https://www.uintent.com/newsletter