Software in Medical Devices, by MD101 Consulting

Verification before validation

This is very basic but I wanted to repeat it: the end of verification happens before the end of validation.
If we place the two activities in the software development cycle, verification happens after coding and software integration activities. Hence there's nothing to verify or validate as long as there's no software or documentation.
However, the position of verification activities depends deeply on the software development cycle.

Position in the waterfall software development

In the classical waterfall software development cycle, the verification activities are placed after coding activities. Thus the verification encompasses:

• Unit tests,
• Integration tests,
• Alpha 1 tests,
• Alpha n tests (if any) ...
• Beta 1 tests,
• Beta n tests (if any) ...

The naming of the tests phases is given here as an example. You may name your test phases as you wish.
Just remember that there's no tests phase in real clinical conditions in this example.

The diagram below summarizes that. Verification activities are highlighted with a shaded blue background. Note that the shaded blue:

• covers a part of coding and integration. This is where unit and integration tests take place,
• doesn't cover the end of the cycle (where validation happens).
  

Position in agile software development

Agile development is a continuous development cycle. Every activity of software development happens during each iteration.
If an iteration implements new functions, these functions have to be verified. Even if an iteration is only made of bug fixes, these fixes have to be verified. So, verification happens during each iteration.
The diagram below highlights with a shaded blue background the verification activities in an iteration.


But iterations are only the core of agile software development in regulated environment. As I already explained in the series of articles How to develop medical device software with agile methods, iterations shall be followed by a phase of consolidation.
The consolidation aims at verifying that what was developed earlier is a consistent, risk free and bug free software.
So the consolidation phase also contains verification activities.
Whereas iterations contain incremental verification, consolidation contains a second pass of verification of software.

The diagram below summarizes that. In shaded blue, the phases where verification happens.

IQ, OQ and PQ

Software is not present only in the medical devices but also in the production plants of medical or pharmaceutic products.
When software is used in an equipment to control the production or manufacturing of products, it has to be verified and validated.
The canonical model of verification of such equipment (software or not) is made of three phases:

1. Installation Qualification,
2. Operational Qualification,
3. Performance Qualification.

Note that the word qualification is used instead of verification. I won't argue on the differences between those two words.
This model is a very robust one and we could get some ideas of it to verify medical device software.

Installation qualification

The Installation Qualification verifies that the equipment was installed according to installation drawings and specifications.
For software, this means that your verification tests shall include the install procedures. One or more tests shall exists and verify the installation of software.

Operational Qualification

The Operational Qualification includes tests cases of start-up, operational use, maintenance, safety functions, alarms of an equipment.
For software this means that your verification tests shall include tests of all these functions and states. Your verification tests shall also include the inspection of software documentation delivered to the end-user.
Software verification is not limited to the main functions. All functions including the less used shall be tested one by one. Same for software documentation that shall be inspected to verify it contains information required to use your software in safe conditions.

Performance Qualification

The Performance Qualification aims at verifying that the user requirements and safety requirements are fulfilled.
For software this means that your verification tests shall contain scenarios to test user requirements and safety requirements. These scenarios shall be close to real use, compared to the tests of functions one by one made in the previous phase.

Software verification categories

To sum-up what I said above, the software tests cases shall contain verification of:

• Software installation,
• Software documentation delivered to end user,
• Functions of software one by one,
• Scenarios of use of software.

On top of that, we can add the verification made by:

• Unit or automated tests and,
• Integration tests.

Unit and integration tests should be planned before other tests mentioned above. That's logical, when you integrate software, you can't test the installation procedure or the functions from A to Z.

Software embedded or standalone

Verification activities are different if software is embedded or standalone.

Standalone software

For standalone software, all tests described above follow their "own" planning. There are limited constraints to the planning of tests. Most of time, only test data and/or system in interface should be missing or - worse - end users who do the test scenarios.
Test data, systems in interface and end-users are definitely the critical resources of tests.

Embedded software

For embedded software, the constraints on tests are usually more important. The planning of software tests depends on the availability of hardware.
The verification is usually split in two phases:

• Tests in simulated hardware environment,
• Tests with real hardware.

Both phases may contain the tests sub-phases described above: installation, documentation, functions one by one, and so on. This can make the planning of verification tests quite long and complex.
About critical resources, embedded software have those of standalone software and add the hardware, which the software runs on.

Who verifies it and where?

Who?

People who test software aren't the same in every phase of verification.

Different people at different stages

During the early stages of verification, tests are done by engineers:

• Unit tests by software developers,
• Integration tests by integrators (or software testers),
• Deep testing of functions one by one by software testers (or integrators)

During the late stages of verification, tests are by people with knowledge of operational use of software:

• Test of scenarios by product champions (or software testers with operational knowledge or selected end-users).

Operational not clinical

If you re-read the paragraph before the diagram just above, you'll see that I wrote "operational use", neither "clinical use" nor "medical use".
This distinction is very important. Even if it is of high value to have someone with clinical knowledge who tests software during verification phases, there is absolutely neither clinical nor medical use of software during verification.

It is possible to have a doctor who tests your software with scenarios, or even free tests during verification.
With a standalone software, you may have the feeling that the conditions are the same (basically a doctor using software on a PC), whether in verification or in validation.
This is where I think resides the confusion between verification and validation, especially for standalone software.
But I'm going to far. I should leave matter for my next article!

Where?

Software passes through various locations and platforms before being installed in the machine/PC of the end-user.
The platforms on where software is installed depends on the test phase and whether software is standalone or not.
For standalone software:

• Unit tests on software development platform,
• Software Integration tests on software development platform or software integration platform,
• Deep testing of functions one by one on software integration platform or on test platform with real interfaces,
• Test of scenarios on test platform with real interfaces.

For embedded software:

• Unit tests on software development platform,
• Software Integration tests on software development platform or software integration platform,
• Hardware + Software Integration tests on hardware integration platform,
• Deep testing of functions one by one on hardware/software integration platform or on test platform with real equipment,
• Test of scenarios on test platform with real equipment.

Again, even if tests are realized on the real equipment/PC with an end-user with medical knowledge, they remain verification tests.
All of this is not a secret, there's a lot about testing in literature. I just wanted to stress out what verification is.


As we've seen here, the verification tests are exclusively technical. I never talked about intended use or clinical assessment and how they intersect/match with verification tests. This is the purpose of my next article about software validation.