While the study manager set up the methodology in cooperation with our client, we gained access to the cars that the client was interested in evaluating. We regularly partner with several specialized car providers who deliver both make and model, and also vehicles with specific features. One of our long-time partners delivered the cars within a very short period.
Our automotive HMI experts evaluated each of the cars. They noted any usability issues and identified best practice solutions. In addition, they gathered metrics such as time on task and steps required for task completion. The evaluation phase was completed within two days.
The study manager (who was not part of the evaluation team), combined the three individual scores from the reviewers into single scores. The team then conducted a group review to sort out any discrepancies identified.
We ultimately provided a quality score for each of the relevant use cases and several worst and best practice examples.