A Brief Explanation of Effective Methods in Medical Device User Testing

by Gilero September 17, 2024

By David Orenstein, Senior Human Factors Engineer (Gilero)

Company XYZ has been the market leader in diagnostics, but recently they have begun to lose market share to low-cost competitors. In order to take back the market, XYZ is going to revamp some existing technology with novel features. The design team has come up with a few concepts but are unsure how best to evaluate what is and is not worthwhile to their users. When finding a place to start they reference to FDA guidance Applying Human Factors and Usability Engineering to Medical Devices but are unsure how best to put the methods described into practice.

Here at Gilero we have encountered this dilemma with many of our customers, and from my experience there are a few stops along this train before you make it to human factors testing (i.e. formative and summative evaluations). First stop is doing your homework, here’s how we can help:

Early Concept (“Market”) Research

I suspect what you may be thinking: “How are human factors engineers going to help? I already have a marketing team.” While your marketing team is driven to find out what features your intended users are interested in, and how incorporating those features will impact pricing and sales; my role will be to dig deeper to understand how the intended user(s) will use these features (or adapt to them if this is a design change to an existing product), and whether they will be able to do so safely and effectively. Evaluating your concepts allows us to help you determine:

• How will the concept impact the user’s overall workflow?
• If a design will be used in the manner you intended it to.
• What (if any) foreseeable safety risks there might be to the users or patients.
  • Evaluating early prototypes with users can help you identify the perceptual, cognitive, and physical challenges that your users might face when interacting with your medical device. 

We do this through a mixture of simulated use tasks; and in-depth interviews to obtain meaningful qualitative data you can learn from.

Formative Evaluations

Heuristics Analysis

Many of our customers don’t tend to consider any inputs into their risk analysis beyond the Hazards Analysis and Task Analysis. But there is another wonderful tool at our disposal called the Heuristics Analysis. A common misconception (and likely why it is overlooked) is that this tool is for evaluating software UIs only. However, heuristic principles are not limited to devices which bear a GUI but may be applied to a variety of interfaces from syringes to laparoscopic tools. They can even be applied to your device’s labeling.

For those who are unfamiliar, this is an approach to help your design team understand if they are on or off-target with respect to known best practices for design, so that it can be used safely and effectively by its intended users. It is intended to help translate qualitative data into a quantitative-like analysis the design team can understand. Heuristic analyses are often done early in your design process but can occur at any time you are considering significant changes to a design. If you are not familiar; Usability Engineers or Subject Matter Experts (SMEs) evaluate a design in its current format by applying selected design heuristics and their professional judgement. Examples of heuristic categories you may apply to device interfaces may include:

• Minimalist – Lacks extraneous information which may unnecessarily impede a user’s workflow.
• Informative Feedback – Provides prompt and informative feedback as actions are being performed.
• Accommodation – Fits the way each user group works and thinks.
• Clear Closure – Clearly notifies a user when a given task is complete.
• Predictability – The user interface does not surprise the user; and allows the user to accurately predict what will happen next.

These five items are a sample of design heuristics you may reference, and of course not an exhaustive list. The heuristics chosen for your product will depend on what is being evaluated and their applicability. Once the appropriate design heuristics are agreed upon, our team will use a systematic approach to score the interface with respect to said design heuristic in the context of a given task the intended user will perform. Heuristics which score higher in each task are flagged for further risk analysis later.

Next Stop – Formative Usability Testing 

Lets discuss the type of testing you are likely more familiar with – Formative and Summative (also known as Human Factors Validation) testing. To start we always recommend regardless of where you are in your product life cycle to perform a formative study run by professional Human Factors/Usability Engineers such as the experts at Gilero. A formative study is a wonderful opportunity to learn where your design stands with respect to safety, efficacy, and usability. It can also help you get valuable feedback on the usability of product labeling. 

There are many ways to approach formative studies and we can help you identify which will be the most advantageous for what you want to achieve. A few examples of methodologies we commonly use include:

• Simulated use – Participants are directed to perform tasks with your device under realistic simulated use conditions. This aids us our understanding of how the device will be used in the real world.
• Directed use – Participants are asked to simulate use of the device specifically as the written directions state to. While this is not applicable to real-world use, it effectively does provide us insight into the strengths and weaknesses of your device’s instructions and labeling.
• Think-aloud – Encourage participants to talk through their thoughts and actions as they occur.
  • This may be performed in combination with other methodologies.
  • Although this detracts from real-world use, it is greatly beneficial in helping us understand the user’s thought process in the moment.
• Pre-summative testing
  • Simulate the conditions of an actual summative study to gauge where we are.
  • This can often be done as a Pilot study prior to the Summative Study with a smaller sample size.

Regardless of how we run the formative, we will always provide best effort to simulate the stresses of the intended use environment while balancing how much risk to the participant can be justified.

All Aboard for Summative (Human Factors Validation) Testing

It is probably fair to say that most readers of this blog article have had some experience with a summative study, If so, you know they are more or less the same. You recruit representative users to perform simulated-use and knowledge tasks (think pop quiz) in a realistic environment using production-equivalent materials; and then observe their performance.

With summatives, there are some important insights to bear in mind. Common worries and misconceptions I have observed include:

• I am worried I will fail my summative study. 

You do not pass or fail a summative study in the same binary manner as you might pass of fail design verification testing. Please remember this is qualitative data, and study participants are not scored on whether they passed or failed a task. They are scored on if they performed the task successfully, expressed difficulty or performed a use error. There are also close calls, but their definition is as clear as mud. Generally speaking, a close call is when a study participant performs a use error but recovers without prompt from the investigator/moderator.

During the study we will provide you feedback if we see any concerning patterns or themes that may be difficult to rationalize to a third-party reviewer. Similar to chain pulling on a train we can pump the brakes on a study at any time to regroup. If we need to pause and regroup, we write it up as another formative, learn from the experience, and adjust. Depending on the issue we may be able to adjust rapidly and maintain the project timeline. The key thing to remember is if you make any significant changes prior to restarting; you will begin with a fresh, new data set.

• Oh no! The study participant performed a use error.

Use errors are not the end of the world. It is normal to observe participants make mistakes as there is no expectation of perfection. What we are looking for are not use errors in general but use errors which have the potential to lead to serious harm (if evaluating a pure medical device) or harm (if evaluating a combination product). Prior to the summative though we will have drafted and finalized the use-related risk analysis (URRA) so we will already know what observations we can “disregard” in our root cause analysis, and what we cannot. Well…for the most part; there is always the possibility that we will witness something we did not anticipate.  

About the Author

Gilero Gilero specializes in the design, development, and contract manufacturing of medical, drug delivery, and diagnostic devices, bringing innovative healthcare solutions to market. The team combines engineering expertise with a deep understanding of regulatory requirements.