- expert review
- target setting
Utilizing expert evaluation to generate reliable metrics
The challenge for this specific project was to provide fact-based feedback that would be akin in quality to feedback that could be gained from participants in a large-scale study. We decided to utilize the car HMI knowledge gathered by three of our most experienced experts in the automotive field in a way that would eliminate any personal bias.
In collaboration with our client’s team, our study manager defined the use cases that would be reviewed by each export.
After, the study manager reviewed the tested vehicles’ HMI and documented the possible paths to task completion for each use case. The goal of this exercise was to later enable the weighing of usability ratings by the percentage chance that a user would actually encounter it (e.g. if an issue was only present in one of three possible paths to task completion, the severity of the issue would be weighted accordingly). In combination with our three-point scale for issue severity, this allowed each issue to be graded with a specific score representative for the expected impact of any issue.
While the study manager set up the methodology in cooperation with our client, we gained access to the cars that the client was interested in evaluating. We regularly partner with several specialized car providers who deliver both make and model, and also vehicles with specific features. One of our long-time partners delivered the cars within a very short period.
Our automotive HMI experts evaluated each of the cars. They noted any usability issues and identified best practice solutions. In addition, they gathered metrics such as time on task and steps required for task completion. The evaluation phase was completed within two days.
The study manager (who was not part of the evaluation team), combined the three individual scores from the reviewers into single scores. The team then conducted a group review to sort out any discrepancies identified.
We ultimately provided a quality score for each of the relevant use cases and several worst and best practice examples.
Because of our results, the client set a target for their own development process. They planned to use the same methodology to directly compare HMI performance between cars. As our rating process was well documented, our client evaluated their HMI as part of the development process without their prototype having to leave the facility.
The worst and best practices identified for their interface designers what pitfalls to avoid and what implementation to emulate.