Usability testing is closely linked to the IVD product’s stage, its audience, and its usage context. It is a subset of market research focused on product functionality, and its timing is tied to development and launch decisions. The two basic kinds of usability test designs include “formative testing” and “summative testing.” Each is relevant for its own role in the product development process.

Unlike most market research, usability testing focuses on what participants do with the prototype more than what they say about it. If a user describes a device as “easy to use” but is observed making several attempts to complete a test, both data points—comment and behavior—are important in the evaluation of the prototype. Additionally, the opportunity to observe the test subjects’ interactions with the prototype is very valuable: it’s a chance to see how the user experiences could be improved, and to bring in the development team’s expertise to anticipate the right solution long before a real-world user can experience the issue.

Formative usability testing allows for the earliest incorporation of human factors into medical device design and production. User experience can have a tremendous impact on a product’s performance. Identifying usability issues early and often during the development process may reveal opportunities to intercept potential user errors before they occur.

Designing an effective formative usability test requires clarity on major unanswered questions about the product’s use (e.g., is specimen preparation or loading a concern, or does result generation seem more problematic?). If there are two different ways to address a specific challenge, usability testing can allow you to observe the relative success of—and user comfort with—both options.

As with any scientific evaluation, begin by generating a hypothesis to be tested, such as “sample loading can be completed in eight seconds or less” or “result generated will be read or recorded accurately by the user 10 times out of 10.” A specific hypothesis is helpful regardless of whether one or two prototypes are under evaluation, and it can be particularly insightful in a side-by-side comparison (A/B testing).

Summative usability testing is the equivalent of quantitative research focused on product functionality. Summative usability testing is a consolidated assessment of the final product’s effectiveness, efficiency, and satisfaction. It evaluates whether the product’s final version meets all of the product requirements, and it is undertaken after formative usability testing has identified issues and addressed them.

The product’s central usability requirements link to specific tasks carried out with the product, and they may be evaluated within multiple use scenarios and/or involving more than one type of user. A variety of use characteristics is typically assessed, including error rates, time to complete tasks, and overall customer satisfaction. In contrast with formative usability testing, which centers on discovering specific design problems, summative usability testing involves an overall product evaluation against specific, defined performance expectations.

When should you perform each test?
For formative usability testing, start as soon as you have a prototype that allows someone to carry out the product functions in question. For a new product or product line, the initial design choices set the direction for the rest of the development process. To set the right priorities at the start, begin with formative usability testing. The insights generated will identify the optimal way to translate your invention into a product that will successfully address the customer’s needs.

Summative testing occurs at the end of your product development process. As the name implies, it confirms the value of the sum of your efforts. You’ve gone through multiple cycles of creating a prototype, testing its functions with users, improving its usability and then repeating the formative test with another small group of users. The formative testing process is complete when users show consistent success in using your product: previously identified issues resolved, no new issues uncovered. Now you have a final product for summative testing.

How often is formative testing done?

Formative testing is best done iteratively, saving time and money by focusing development work on the most promising approaches. Yet formative usability testing need not be expensive: in the early stages, the number of users involved in a session is often no more than five. This is qualitative research intended to identify issues and proceed through several cycles of finding and fixing the gaps in design that limit product success.

On the other hand, summative testing is quantitative research. The product used in this stage is in its final form, with known issues resolved; the number of users increases to at least fifteen for statistically significant results. These are the data points that will be shared with investors, sales reps, and customers to demonstrate that the final product matches performance expectations and to assure them of a satisfactory experience in the hands of typical users.

How is it done?

The key steps in the process for both formative and summative testing include:
• Identify the audience of users and find participants
• Plan the test

o Set goals
o Describe desired participant interaction with prototype
o Write the introduction and directions for participants
o Detail the tasks to be carried out by participants
o Carry out a pilot test ahead of time to test the clarity and completeness of your written materials

• Plan the documentation of observations
• Carry out testing, observe, and listen
• Prepare documentation; present results to stakeholders
• Identify takeaways for the development team; likely goals for the next iteration of testing

In formative testing, this process can begin as soon as you have a functional prototype. The summative process can begin once you have a final product.
Formative testing also tends to be more informal and interactive than summative to maximize the potential to get design feedback from users. In summative testing, in which the goal is statistical robustness, testing sessions are more uniform and formal.

“How can I conduct successful usability testing?”
You can conduct your own usability testing, and many product developers do. Product developers generally conduct informal usability testing with their prototypes (or simulations thereof) to identify potential issues. However, there’s no substitute for formal usability testing: it’s essential to generating objective assessments that represent the experience of real-world users—and that involves external participants.

To that end, most companies engage with professional firms that provide specialized experience and facilities to achieve the best results. These companies offer:

• Usability Testing Expertise: A full-time team engaged in usability testing that has conducted many such tests across various products and dozens of user types: knowledge of what a successful usability testing protocol looks like, how to develop it, and how to execute it.

• Joint Design of Test Protocols: Partnering with its clients on the full specifications of both formative and summative usability test protocols. Once a client has outlined its goals for the study, the provider works with them on procedural organization; user requirements including customer type and level of experience; and the setup of prototypes in the lab to match appropriate use-case scenarios.

• In-House Recruiting of Test Subjects: Personally recruiting and vetting each test participant, matching them to very specific user requirements. Attention is paid to the experience of the process to attract high-quality users genuinely interested in trying out the product and providing feedback (i.e., avoiding those merely seeking compensation for their time).

• Product Development & Clinical Research Experience: Having real experience developing products of its own. Some companies have onsite product development facilities as well as a clinical research and regulatory approval organization that work under the same roof as their usability testing team. This experience allows the firm to comprehensively anticipate concerns and challenges before they occur.

• Full-Scale Usability Testing Lab: An effective usability testing lab should feature a dedicated observation facility for observers and clients to watch and listen in real time via one-way mirrors and camera monitoring systems.

Ultimately, successful product development requires usability testing experiments. These range from preliminary, hypothesis-driven formative usability tests that sharpen the focus of the design and key product function issues, to final summative usability testing of the market version against hardened performance expectations.

Early engagement with usability testing and the iterative “find and fix” cycle can reduce development time and expense while improving the overall market fit based on high-quality evidence of end-user acceptance. When leveraged correctly, usability testing can help ensure your design is on the right path to great user satisfaction and successful market uptake.