Summative evaluation of an angiography system

THE CHALLENGE

A medical systems manufacturer was developing a new angiography system, which consisted of a hardware and software update for the previous version. The study intended to demonstrate that critical use tasks for the system could be performed safely and effectively by the intended users in the intended-use environment. Additionally, the study intended to demonstrate that user manual warnings could be fully understood by the system’s intended users. The study results were to be submitted to the FDA.

THE APPROACH

A summative evaluation was carried out with intended users in a market research facility, on an angiography system. Use tasks were assessed by a simulated use methodology, and warnings in the instruction manual were assessed via a comprehension assessment.

Defining and finding the intended users for the study

As the FDA requests that a validation study should be performed on USA soil with USA residents, the study took place in the USA, as did several previous formative studies for this system. These previous studies already set definitions for the intended users. However, they had taken place in a different state. The nomenclature and even the roles of the different specialties involved in the use of the angiography system, differ from state to state. Therefore, during recruitment, we had to perform an analysis of the specialties, to best reflect the initial intended user population. The correct intended users for the study were discussed and confirmed with several stakeholders.

Evaluation of critical tasks and user manual warnings

The study aimed at confirming that critical use tasks could be performed in a safe and effective way by intended users. One of the tasks involved moving the system’s arm with a joystick that included a safety mechanism – in order for the system’s arm to move, the joystick first had to be pushed down before moving it. Because of the bulk and weight of a fully operating angiography system, and the fact that a real system arm would have to be fixed to the ground or ceiling in a hospital environment, it was not possible to replicate a full angiography system for the study. Therefore, the test system could not include an arm. For the study, the evaluation of the correct use of the joystick has been performed by moderator observation and the movement of the system’s arm has been simulated by providing live feedback to the participant. This method proved to be very effective in assessing whether the critical task had been performed correctly and allowed to collect participant feedback when it could not be performed correctly.

Another goal of the study was to assess whether some of the user manual warning messages were comprehensible. A comprehension assessment methodology was used to assess the correct understanding of the messages: Participants were given some time to read the warning messages and were then asked to explain what the message meant in their own words. The moderator was responsible for the assessment of the understanding of the warning message. Any part of the message that seemed unclear was probed on by the moderator, until it was obvious whether the message had been fully understood or not. Additionally, the moderator aimed to understand which part(s) of the messages were confusing, and the reason(s) for the confusion. If any part of the message was not understood, participants were asked how they would reword it to make it understandable.

Results reporting in FDA format

Results of the study were described in a report following the FDA format. The results of the evaluation of the use tasks were summarized and categorized, including detailed root causes and residual risk analysis. The results of the comprehension of the warnings were also summarized and categorized, with detailed feedback on which parts of the messages potentially could prevent a full understanding of the message, with a recommendation on wording. The report was submitted to the FDA and could be processed effectively.