Modelling system quality

I see many people struggle with quality requirements, including experienced business analysts and requirements engineers. Worse still, sometimes people just ignore quality requirements because they don’t know how to specify them properly. While quality requirements may be a bit harder to specify than functional requirements, they are often crucial to the success of a product. To make matters worse, many of these quality requirements impact the architecture of the whole system; they can’t just be added on at the last minute. I’d say that makes it well worth investing some time to properly specify what quality levels are required. Here are some of my tips.

What are we talking about?

First let me define what I mean with “quality requirements”. My favorite definition is based on Tom Gilb’s excellent book Competitive Engineering:

A quality requirement expresses ‘how well ’ a system will perform.

This is a simple and useful definition, provided you take a broad interpretation of “perform”. Perform in this context does not mean “performance” in the sense of throughput or response time. Instead it refers to any aspect you care about: how well a system performs with respect to usability, security, availability, performance, learnability, etc.

Quality requirements are sometimes called non-functional requirements, though in most cases non-functional requirements include other types of requirements too, such as constraints.

Getting started

When you start out on a new product or a major change, it is important to find out which quality aspects are most important. Look for unique selling points as well as dissatisfiers (“Must-be quality” in Kano terminology). The companies core values or market proposition may also provide clues on important quality aspects. Select the top 3 – 5 quality aspects, and focus on those.

Why focus on just a few? It is usually hard for a system to do well on many different quality aspects (fast response time, high security, high availability, many transactions per second, many concurrent users etc.), so you really need to find out which are most important and get them right. Given a few important quality aspects, engineers can often design a suitable system.

Also, the system will still have a basic performance in the quality aspects you don’t specify: if you omit user friendliness requirements you don’t get zero user friendliness, but it may not be the easiest system to use.

Selecting quality aspects

If your stakeholders can’t think of any quality aspects, there are many ways in which you can help them. I often ask “What targets do you have?” or similar questions to get people started. Alternatively you can use models such as ISO 9126 or ISO 25010 to get started. Bear in mind that these models contain quite abstract terms; you will need to be more specific and make sure what your stakeholders mean. Abstract terms like “reliability” and “maintainability” can mean very different things to different stakeholders. Also, these models are aimed at software engineering and systems engineering disciplines; other types of products may require very different quality aspects. For example, clothing may need to be comfortable and food may have to be nutritious or tasty.

Specifying the required quality levels

Once you have selected a few quality aspects, you can drill down to what it really means. Work with key stakeholders to define the “gist” or “essence” of what they mean. For example, starting with “maintainability” you may find that in essence what the stakeholders want is the ability to release new versions of the software without disrupting operations. This could then be your quality requirement.

The next step is to determine how to validate this requirement. Key considerations here are:

  • What level is required? In the above example you need to define what level of disruption is acceptable. Perhaps you also need to define types of releases or frequency of releases. The required level could be e.g .: “Less than 5 minutes disruption per release.”. It could also be specified per month, or be dependent on the type of release, to name a few possible alternatives.
  • How and how often are you going to measure the quality level? The example level above implies measuring when the system is already operational. This is often the best match with what stakeholders specify. However, if the system doesn’t meet the requirements, it is a bit late to find out at this stage – and probably costly to fix too. For crucial quality requirements I suggest trying to find early stage predictors which can be measured. In our example of releasing without disrupting operations, one possible design-time predictor could be the number of dependencies between system components. The higher the number, the more likely operations will be disrupted because components which are being used are more likely to need to be restarted. The difficult part here is finding appropriate thresholds: at what number of dependencies should we start to worry? Of course, if you release often (as with Agile approaches), you have more chance of measuring both the predictor and the actual disruption. This makes it more feasible to work out the correlation between these two measures and to then set target levels appropriately.

What is the right quality level?

If you end up in situation where you have no idea what the appropriate quality level should be, try one or more of the following:

  • Measure how well an existing system performs. Then determine how well the new system must do in relation to the reference system.
  • Specify a range by using multiple levels, for example a minimum level below which the product cannot be released and a desired level above which no more effort should go on improving this particular quality aspect. Note that for some quality aspects bigger numbers mean better performance (e.g. availability and throughput), while for others it is the reverse (e.g. response times, down-time).
  • Specify different levels for different parts of the system. For example, the response time for retrieving customer details may be quite different from the response time for printing customer details.
  • Examine the (estimated) cost of a few different quality levels. There are often “natural” thresholds to a systems performance: thresholds caused by limitations of the architecture for example. You may find that increasing availability from, say, 99.0% to 99.2% doubles the development cost. This may be enough to convince stakeholders to settle for a lower availability.
Cost thresholds to quality

Cost thresholds at quality levels 5 and 10

Just do it

I hope these tips will encourage you to give quality requirements sufficient attention. If you are still concerned that it is too difficult, remember that just finding out which quality aspects are most important may help the team to design the system in such a way that it can perform well in those areas. That alone may be the difference between success and failure.