Software in Medical Devices, by MD101 Consulting

Before applying this without critical thinking, please take note that what is described below may not be enough for cases where use errors can have severe consequences, e.g. devices intended to be sold to end-users directly. In such cases, requesting the services of specialists in human factors engineering is probably the best solution.

Usability engineering plan

Write what you do, do what you write. The story begins with a plan, as usual in the quality world. The usability engineering plan shall describe the process and provisions put in place. For standalone software, this process lives in parallel to the software design process. The usability engineering plan can be a section of the software development plan, or a separated document. The usability engineering plan describes the following topics:

• Input data review,
• Definition of use specification,
• Link with risk management,
• User interface specification,
• Formative evaluation protocol,
• Formative evaluation report,
• Design review(s),
• Summative evaluation protocol,
• Summative evaluation report,
• Usability validation.

Note: you can use the structure and content below in this article to write your own usability engineering plan (if you can afford not to pay for usability engineering specialists :-)).

Usability input data

Usability input data is a subset of design input data. They are gathered before or at the beginning of the design and development project. Depending on the context of the project, they can contain:

• Statements of work,
• User requirements collected by sales personnel, product managers …,
• Data from previous projects,
• Feedback from users on previous versions of medical devices,
• Documentation on similar medical devices,
• Specific standards, like IEC 60601-1-8 Home use of electromedical devices,
• Regulatory requirements, like IFU or labeling.

Usability input data are reviewed along with other design input data. So, you should include these data in your design input data review.

Preparing the use specification

The use specification is a high-level statement, which contains information necessary to identify:

• the user groups which are going to be subject of the usability engineering process,
• the use environment in which the device is going to be used,
• the medical indications which are needed to be explored further.

The use specification shall include the:

• Intended medical indication;
• Intended patient population;
• Intended part of the body or type of tissue applied to or interacted with;
• Intended user profiles;
• Use environment; and
• Operating principle.

Preparing the use specification can make use of various methods, for example:

• Contextual enquiries in the user's workplace,
• Interview and survey techniques,
• Expert reviews,
• Advisory panel reviews.

Usually, the use specification is prepared with expert reviews. This method is the simplest to implement (once again if you can afford not to use other methods :-))

The use specification is recorded in the usability management file.

Analysis

The usability engineering process is performed in parallel to the ISO 14971 risk management process. Below is a diagram showing the links between the risk management process and the usability engineering process. This diagram is non-exhaustive and for clarification purposes only. I didn’t represent the software development process on this diagram. This is not the purpose of this article to show the relationships between software development and risk management. Please, have a look at this post on ISO 14971 if you’re keen at refreshing your memory on software development.

Identifying characteristics for safety

This step sounds clearly like risk management. It consists in identifying:

• The primary operating functions in the device,
• The use scenarios,
• The possible use errors.

In a first approach, you can answer the questions in ISO 14971 annex C to identify characteristics to safety. If the man software interaction is prone to be a source of critical hazardous situations, more advanced methods may be required.

These data (primary operating functions, use scenarios and possible user errors) are recorded in the usability management file.

For software, the primary operating functions and use scenarios can be modelled with use-case diagrams and descriptions. Note that UML requires that a use-case diagram contains a text description of the use-cases.

Identifying hazardous phenomena and hazardous situations

This step consists in identifying the hazardous phenomena and hazardous situations (ditto). This is classical risk assessment. They are identified with data coming from:

• The use specification,
• Data from comparable devices or previous generations of the device,
• User errors identified in the previous step.

These elements are documented in the risk management file. They can be placed in a section specific to human factors.

Examples of user-related hazardous phenomena and situations:

• A warning is displayed, the user doesn’t see it,
• A value is out of bounds, the user doesn’t see it,
• The GUI is ill-formed, the user doesn’t understand it.

Identifying and describing hazard-related use scenarios

This step is once again risk analysis: the hazardous phenomena, the sequence of events, and the hazards, resulting of the human factors are identified. These elements are documented in the risk management file accordingly.

Selecting hazards-related scenarios for summative evaluation

It is not required to submit all hazard-related scenarios to the summative evaluation. It is possible to select a subset of these scenarios based on objective criteria.

Usually, the criteria is Select hazard-related scenarios where the severity is higher than a given threshold e.g.: Severity ≥ Moderate. You shall write your own criteria in the usability plan.

Said the other way round, it’s not worth including scenarios with low risks in the summative evaluation.

Identifying mitigation actions and documenting the user interface specification

The risks related to the use scenarios are then evaluated according the risk management plan (severity, frequency, and possibly detectability if you included that parameter in you risk management plan), and mitigation actions are identified, by following the risk management process.

Identification of mitigation actions can be done either before or during the formative evaluations. It is necessary to confirm the validity of the mitigation actions during the formative evaluations.

The mitigation actions are documented in the user interface specification, in order of priority (see 6.2 of ISO 14971):

• Changes in user-interface design, including warnings like message boxes,
• Training to users,
• Information in the accompanying documents: IFU and labeling.

For software, the user interface specification can be included in the software requirement specification.

Note that warnings in the graphical user-interface can be seen as design change, and not information to the user. But this shall be verified in the summative evaluation. The warning message shall be relevant enough, placed at the right step in the workflow and change the user’s mindset to avoid an hazardous situation.

Design and Formative Evaluation

The formative evaluation is performed during the design phase. You can have one or more formative evaluations. The sequence of formative evaluations in the design project depends on the software being designed. There is no canonical sequence of formative evaluations. At least one formative evaluation is required, though this could be a bit too short. Two formative evaluations sound like a good fit. The points not identified or discussed in the first evaluation can be treated in the second evaluation.

To be relevant with the design and development project, the formative evaluations should be placed before the last design review. Thus, the design and the user interface are “frozen” after the design review. This doesn’t mean that the formative evaluation happens during the design review.

The formative evaluation can be done with or without the contribution of end-users. The methods of evaluation depend on the context: questionnaires, interviews, presentations of mock-ups, observation of use of prototypes.

For software, the commonly adopted solution is the presentation of mock-ups or prototypes, with end-user proxies (like product managers, biomedical engineers) and end-users who can “play” with the mockups. It is also a good option to let the end-user proxies review the mock-ups to “debug” them, before presenting them to real end-users. So that the presentation doesn’t deviate (too much) with wacky requests from end-users.

Summative evaluation

The summative evaluation is performed at the end of the design phase. It can be done after the verification, or during the validation of the device or, if relevant or possible, during clinical assays. It aims at bringing evidence that the risks related to human factors are mitigated. The position of the summative evaluation depends on the context of your project.

The summative evaluation shall be done with a population of end-users statistically significant for the evaluation. E.g. at least 5 users of each profile defined in the use specification (see FDA guidance documents on Human Factors Engineering). The summative evaluation shall be done for every scenario selected according to criteria defined above (e.g. severity ≥ moderate). The methods of evaluation are left to your choice, depending on the context e.g.: use in simulated environment, use in the target environment.

If the data collected during the summative evaluation don’t allow to conclude on the proper effectiveness of the mitigation actions, or if new risks are identified, you shall either redo the usability engineering process iteratively, or bring rationale on the acceptability of the residual risks individually and on the overall residual risk acceptability. The rationale can be sought in the risk/benefit ratio on the use of your device.

For software, the solution commonly adopted is free tests performed by selected end-users on a beta version or a release candidate version. The summative evaluation can end with the analysis of a questionnaire filled by the selected end-users.

The user interface of the device is deemed validated when the conclusion of the summative evaluation is positive. You’re done, good job!

Application to agile methods

The steps described above can be disseminated in the increments of an agile development process.

Use specification, primary operating functions

Use specification and primary operating functions are usually defined in the initialization/inception phase of the project. This phase sets the ground of the software functions and architecture. Likewise, the use specification and primary operating functions are defined during this phase.

Depending on how much you know about the software being developed, the initialization can also be the right time to write the use scenarios. If you already know, say, 80% of the user requirements, you can write the use scenarios and make the risk assessment on these scenarios at the beginning of the project. If you don’t know much on your future software, the use scenarios have to be defined/updated during the iterations.

Iterations and usability engineering

The next steps of the usability engineering process are performed during iterations, as shown in the following diagram and explained in the next subsections.

Use scenarios and hazards mitigation

The “objects” you manipulate during the iterations are epics and user-stories. They can be seen as use scenarios or small chunks of use scenarios, depending on their size. Thus, you can use them to identify hazards related to user-errors, identify mitigation actions, and update the user-interface specification accordingly.

Depending on the items present in the backlog (eg a brand-new use scenario), it is also possible that you have to update the use specification and the list of primary operating functions, during an iteration.

Formative evaluations

Agile methods usually define “personas”, which represent the user-profiles, and are used by the software development team to understand the behavior of the users.

You may base your formative evaluation on the use of these personas. With the role of end-user proxy for the team, the product owner is responsible for the formative evaluation. He/she does the formative evaluation of the user-stories. He/she may invite another person external to the team (or to the company) to participate to the formative evaluation.

You can do the formative evaluation during the demonstration of the software at the end of the iteration. Depending on the results of the formative evaluation, new items related to the user-interface may be added to the backlog and implemented in a further iteration.

Summative evaluation

The summative evaluation is placed after the verification phase of the agile software development process. It is performed as described in the “Summative Evaluation” section above.

Incremental summative evaluation may be performed with intermediate releases. I don’t recommend that method. The user-interface is subject to changes in a further intermediate release, invalidating the conclusions of an incremental summative evaluation.

A way to see things is to say that summative evaluation isn’t something agile.

Conclusion

I hope you have a better understanding on how to implement IEC 62366-1:2015 in you software development process. Remember that I'm in software above all, human factors engineering isn't my background.
Congratulations and hate comments are welcome!

Edit: Templates
You will also find in the templates repository page, two templates useful to generate records of your usability engineering process:

• Usability Engineering File,
• Usability Summative Evaluation Plan and Report.

I share this template with the conditions of CC-BY-NC-ND license.

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 France License.