How to differenciate Bugs, Software Risks and Software Failures - Part 1
A bug can lead to a software failure.
Having bugs is a risk.
Having a software failure is a risk.
A software failure is not necessarily a bug!
Do you follow me?
If not, let me give you some more explanations.
Let's begin by the easiest concept: bugs.
With the prominence of digital appliances in our everyday life, everyone knows what a bug is:
A bug, it's when it doesn't work.
More than one concept behind bugs
This simple concept encompasses several other concepts that software engineers have defined. It was necessary to do so because the way you solve the problems raised by each concept is different.
IEC 62304 introduces a process named Problems Resolution. The purpose of this process does fit the general concept of bug. It aims at analysing and solving any bug, any issue that happens with a software in a medical device.
Thus you should trigger the Problems Resolution process when you have to manage every concept described below.
Defects or anomalies
While bug is a term used in everyday life, engineers - especially testers - prefer using the term: defect. And they prefer adding this precision of context: It's when it doesn't work in conditions where it's supposed to work. You did everything you were told, and it doesn't work! That's because:
- There is an error in the program or,
- Something was missed,
- Something was forgotten.
Example (if by chance you needed one):
A pocket calculator, supposed to work with 10-digits numbers, that gives this result: 2+2=5 --> Bug
For sure there is an error in the program, or something was missed/forgotten.
IEC 62304 prefers using the term Anomaly. It is defined at the beginning of the document and is used many times in the standard. That means that IEC 62304 focuses a big part of the efforts to reduce the amount of anomalies that remains when a software is delivered to end-users.
Characterization of defect
Defects can be so numerous, diverse and annoying that software engineers have defined a lot of attributes to characterize and categorize them. A bit like entomologists do with real bugs.
A defect has a title and an explanation plus a bunch of criteria. The most used criteria are:
- : how it affects the use of software,
- : how often it appears,
- : on which version of software,
Other "administrative" criteria are also useful:
- : is it open (still there), closed (fixed), pending (being fixed),
- : when it was found,
- : who found it,
- : who's in charge to fix it.
Some companies like to add a looong list of criteria in their databases to characterize defects. This is not very useful, as most of engineers only use a small set of them.
Severity, the most important criterion
Most of times, the severity is defined by values like:
- : software stops or can't be used,
- : software doesn't give expected results, but there is a workaround,
- : software gives a result with a non-significant discrepancy compared to expected result.
Some engineers like to add the value, for very minor things. When a word is misspelled in the user interface, but the whole thing remains understandable, then that defect can be set as cosmetic.
Severity is the most important criterion for software engineers because this is the one which is usually taken to sort defects by priority. The critical defects are fixed at first. The cosmetic ones are fixed when time remains.
This way of sorting defects can lead to epic situations when beta testers and engineers review and choose the defects that have to be fixed at first.
If you are an end-user that already participate to this kind of meeting, you already had to get along with a stubborn engineer that didn't want to set a high priority to a damned annoying bug!
If you are a software engineer, you already participate to this kind of meeting, and you already had to get along with a stubborn end-user that didn't understand anything to the way your software works and wanted to break something that works pretty well!
If you are a project manager, you know how long this meeting is going to be! :-)
But I digress.
In short, defects are deviations of software from the expected result. They are usually put in three categories: Critical, Major and Minor.
The diagram below, that I'm going to complete later on, represents the defects.
Software failure and defect are different concepts for software engineers. The general public may tend to confuse these concepts. When it ain't work, it ain't work.
When I gave the engineer-friendly definition of defect, I added the context: in conditions where it's supposed to work.
This is the context that can be used to differenciate defects and software failures:
A software failure is when it doesn't work, but it was not supposed to work in such condition.
If my 10-digits pocket calculator can't compute:
- 2+2=5 --> fails
It's a defect.
If my 10-digits pocket calculator can't compute:
- 2000000000000000000000000000 + 2000000000000000000000000000 = unexpected result --> fails
It's not a defect but a software failure. Input data are outside the expected range of value.
If my PACS viewer can't read 100% DICOM compliant data, it's a defect.
If my PACS viewer can't read corrupted data, it's a software failure.
You can see in this case that, this software failure doesn't mean that the software is guilty. Data were corrupted through the network, when burning the CDROM ... not by the PACS viewer.
The battery of my thermometer is empty and it gives a wrong value. No, my patient is not in hypothermy!
In this case, the software failure comes from a hardware failure. It may (shall!) have been expected, with something like the flashing led of the battery state, telling me to charge it.
I entered a wrong value and I got a wrong result.
Probably the type of failure that is the most difficult to avoid.
Defect vs Failure
But defects and failures are not so distinct concepts. There is a strong relationship between them.
When there is a defect that makes my software totally or partially unusable, a can say that my software fails! So, a defect can lead to a failure?
The answer is yes. But not for all defects:
- The defects that make the software totally or partially unusable, i.e. Critical and Major defects, lead to a software failure.
- The defects that don't make the software unusable, like misspelled graphical interface, don't lead to a software failure. At worst, they can make the software a little bit annoying for the user.
In short, a critical or major defect can generate a software failure.
Minor or cosmetic defects don't.
Software failures can be generated by other types of root causes, like hardware failures or user errors.
OK. But a minor defect that annoys a user, it sounds like a risk.
Talking about risks, are there software related risks that are neither defects, nor failures?
I'll explain this in the next article. It will deal with the risks and their relationship with defects and failures.