• medical system manufacturer
  • summative

Summative evaluation of an angiography system

Validating critical use tasks and user manual warnings of a new angiography system.
Nov
03
2019

THE CHALLENGE

A medical systems manufacturer was developing a new angiography system, which consisted of a hardware and software update for the previous version. The study intended to demonstrate that critical use tasks for the system could be performed safely and effectively by the intended users in the intended-use environment. Additionally, the study intended to demonstrate that user manual warnings could be fully understood by the system’s intended users. The study results were to be submitted to the FDA.

THE APPROACH

A summative evaluation was carried out with intended users in a market research facility, on an angiography system. Use tasks were assessed by a simulated use methodology, and warnings in the instruction manual were assessed via a comprehension assessment.

THE OUTCOME

All use tasks were performed, results were categorized, and detailed root causes and residual risk analysis were provided. The comprehension of the warnings was also assessed, and results were categorized. The report was used for submission to the FDA.

Defining and finding the intended users for the study

As the FDA requests that a validation study should be performed on USA soil with USA residents, the study took place in the USA, as did several previous formative studies for this system. These previous studies already set definitions for the intended users. However, they had taken place in a different state. The nomenclature and even the roles of the different specialties involved in the use of the angiography system, differ from state to state. Therefore, during recruitment, we had to perform an analysis of the specialties, to best reflect the initial intended user population. The correct intended users for the study were discussed and confirmed with several stakeholders.

Evaluation of critical tasks and user manual warnings

The study aimed at confirming that critical use tasks could be performed in a safe and effective way by intended users. One of the tasks involved moving the system’s arm with a joystick that included a safety mechanism – in order for the system’s arm to move, the joystick first had to be pushed down before moving it. Because of the bulk and weight of a fully operating angiography system, and the fact that a real system arm would have to be fixed to the ground or ceiling in a hospital environment, it was not possible to replicate a full angiography system for the study. Therefore, the test system could not include an arm. For the study, the evaluation of the correct use of the joystick has been performed by moderator observation and the movement of the system’s arm has been simulated by providing live feedback to the participant. This method proved to be very effective in assessing whether the critical task had been performed correctly and allowed to collect participant feedback when it could not be performed correctly.

Another goal of the study was to assess whether some of the user manual warning messages were comprehensible. A comprehension assessment methodology was used to assess the correct understanding of the messages: Participants were given some time to read the warning messages and were then asked to explain what the message meant in their own words. The moderator was responsible for the assessment of the understanding of the warning message. Any part of the message that seemed unclear was probed on by the moderator, until it was obvious whether the message had been fully understood or not. Additionally, the moderator aimed to understand which part(s) of the messages were confusing, and the reason(s) for the confusion. If any part of the message was not understood, participants were asked how they would reword it to make it understandable.

Results reporting in FDA format

Results of the study were described in a report following the FDA format. The results of the evaluation of the use tasks were summarized and categorized, including detailed root causes and residual risk analysis. The results of the comprehension of the warnings were also summarized and categorized, with detailed feedback on which parts of the messages potentially could prevent a full understanding of the message, with a recommendation on wording. The report was submitted to the FDA and could be processed effectively.

THE PROJECT LEAD

Jonathan Singh

Jonathan has more than 11 years of experience in the health industry. His main focus at uintent is Human Factors work, running human factors studies (formative and summative) for all types of medical devices. He has an in-depth knowledge of industry standards and regulations (e.g. FDA guideline, AAMI HE75, IEC 62366 and EU MDR) and Usability processes. He also has extensive experience in digital biomarkers (sensor-based endpoints) and companion health apps. Jonathan sits in our Swiss office in Basel.

Conglaturation!!! You've found our secret.
Our site provides a much better user experience when used in portrait mode, so we advise you switch to portrait to enjoy it.
If you want to continue using in landscape mode at your own risk (for your enjoyment), you can use this button :-)

Use site in landscape mode