MEDICAL SYSTEM MANUFACTURER, SUMMATIVE

Summative evaluation of an angiography system

Validating critical use tasks and user manual warnings of a new angiography system.

THE CHALLENGE

A medical systems manufacturer was developing a new angiography system, which consisted of a hardware and software update for the previous version. The study intended to demonstrate that critical use tasks for the system could be performed safely and effectively by the intended users in the intended-use environment. Additionally, the study intended to demonstrate that user manual warnings could be fully understood by the system’s intended users. The study results were to be submitted to the FDA.

THE APPROACH

A summative evaluation was carried out with intended users in a market research facility, on an angiography system. Use tasks were assessed by a simulated use methodology, and warnings in the instruction manual were assessed via a comprehension assessment.

THE OUTCOME

All use tasks were performed, results were categorized, and detailed root causes and residual risk analysis were provided. The comprehension of the warnings was also assessed, and results were categorized. The report was used for submission to the FDA.

Defining and finding the intended users for the study

As the FDA requests that a validation study should be performed on USA soil with USA residents, the study took place in the USA, as did several previous formative studies for this system. These previous studies already set definitions for the intended users. However, they had taken place in a different state. The nomenclature and even the roles of the different specialties involved in the use of the angiography system, differ from state to state. Therefore, during recruitment, we had to perform an analysis of the specialties, to best reflect the initial intended user population. The correct intended users for the study were discussed and confirmed with several stakeholders.

 

Evaluation of critical tasks and user manual warnings

The study aimed at confirming that critical use tasks could be performed in a safe and effective way by intended users. One of the tasks involved moving the system’s arm with a joystick that included a safety mechanism – in order for the system’s arm to move, the joystick first had to be pushed down before moving it. Because of the bulk and weight of a fully operating angiography system, and the fact that a real system arm would have to be fixed to the ground or ceiling in a hospital environment, it was not possible to replicate a full angiography system for the study. Therefore, the test system could not include an arm. For the study, the evaluation of the correct use of the joystick has been performed by moderator observation and the movement of the system’s arm has been simulated by providing live feedback to the participant. This method proved to be very effective in assessing whether the critical task had been performed correctly and allowed to collect participant feedback when it could not be performed correctly.

 

Another goal of the study was to assess whether some of the user manual warning messages were comprehensible. A comprehension assessment methodology was used to assess the correct understanding of the messages: Participants were given some time to read the warning messages and were then asked to explain what the message meant in their own words. The moderator was responsible for the assessment of the understanding of the warning message. Any part of the message that seemed unclear was probed on by the moderator, until it was obvious whether the message had been fully understood or not. Additionally, the moderator aimed to understand which part(s) of the messages were confusing, and the reason(s) for the confusion. If any part of the message was not understood, participants were asked how they would reword it to make it understandable.

 

Results reporting in FDA format

Results of the study were described in a report following the FDA format. The results of the evaluation of the use tasks were summarized and categorized, including detailed root causes and residual risk analysis. The results of the comprehension of the warnings were also summarized and categorized, with detailed feedback on which parts of the messages potentially could prevent a full understanding of the message, with a recommendation on wording. The report was submitted to the FDA and could be processed effectively.

Author

Jonathan Singh

Jonathan has more than 11 years of experience in the health industry. His main focus at uintent is Human Factors work, running human factors studies (formative and summative) for all types of medical devices. He has an in-depth knowledge of industry standards and regulations (e.g. FDA guideline, AAMI HE75, IEC 62366 and EU MDR) and Usability processes. He also has extensive experience in digital biomarkers (sensor-based endpoints) and companion health apps. Jonathan sits in our Swiss office in Basel.

Go back

Privacy settings

We use cookies on our website. Some of them are essential, while others help us to improve this website and your experience.
In this overview you can select and deselect individual cookies from a category or entire categories. You will also find more information about the cookies available.
Group Analyse
Name Google Analytics
Technical name _gat,_gtag_UA_120284862_1
Provider Google LLC
Expire in days 730
Privacy policy https://policies.google.com/privacy
Use Google cookie for website analysis. Generates anonymous statistical data on how the visitor uses the website.
Allowed
Group Essenziell
Name Contao CSRF Token
Technical name csrf_contao_csrf_token
Provider
Expire in days 0
Privacy policy
Use Serves to protect the website from falsification of cross-location inquiries. After closing the browser, the cookie is deleted again
Allowed
Group Essenziell
Name Contao HTTPS CSRF Token
Technical name csrf_https-contao_csrf_token
Provider
Expire in days 0
Privacy policy
Use Serves to protect the encrypted website (HTTPS) from falsification of cross-location inquiries. After closing the browser, the cookie is deleted again
Allowed
Group Essenziell
Name PHP SESSION ID
Technical name PHPSESSID
Provider
Expire in days 0
Privacy policy
Use PHP cookie (programming language), PHP data identifier. Contains only a reference to the current session. No information is stored in the user's browser and this cookie can only be used by the current website. This cookie is mainly used in forms to increase usability. Data entered in forms are e.g. B. stored briefly if there is an input error by the user and the user receives an error message. Otherwise all data would have to be re-entered.
Allowed
Group Essenziell
Name FE USER AUTH
Technical name FE_USER_AUTH
Provider
Expire in days 0
Privacy policy
Use Saves information of a visitor as soon as he logs into the frontend.
Allowed