File on Four on safety-critical software

[David Farber <farber () central cis upenn edu>]

Date: Sun, 24 Oct 93 21:13:07 GMT
From: Pete Mellor <pm () csr city ac uk>
Subject: File on Four on safety-critical software


The BBC Radio 4 programme "File on Four" last Tuesday was devoted to the
subject of safety-critical software.


It discussed the causes of the "software crisis", described a number of the
famous recent disasters, and then considered what can be done about it.


The problems that arise with software are due to "complexity", "novelty",
and "discontinuity".


Prof. Cliff Jones of Manchester characterised the complexity of software in
terms of the number of branch points it may contain, and hence the number of
possible paths through it. The combinatorial explosion of possible paths
makes exhaustive testing impossible in all but the simplest programs. It may
be difficult to achieve with 50 Lines of code and 10 branch points. With
10,000 LOC and the same density of branch points, the testing time would
exceed the time elapsed since the big bang. As he pointed out, the Sizewell B
Primary Protection System contains 100,000 LOC.


Prof. Bev Littlewood of CSR criticised the tendency of manufacturers to take
advantage of the presence of software to make systems perform more and more
new functions, so that the soft-centred systems are many times more complex
than those they replace, and are also attempting completely novel things,
whose consequences may be poorly understood.


Cliff Jones returned to describe the discontinuous nature of digital systems,
whereby changing one bit in the internal state or input may cause a vast
change in the output. This is in contrast with the "traditional" engineering
approach which relies on the fact that, in systems which are governed by
the laws of physics instead of the rules of logic, a continuous variation
of input leads to a continuous variation of output.


I spoke briefly about the problems of certifying software to a certain
numerical level of reliability, with particular reference to avionics systems,
where the regulations specifically exclude software from reliability
measurement, with the result that manufacturers treat its reliability as 1
for the purposes of calculating overall system reliability.


Martyn Thomas described the guidelines for avionics software as simply
exhorting the manufacturer to "try harder" with the more critical stuff.
Dan Hawkes of the CAA pointed out that "do your best" is an approach often
used with certain hardware systems on aircraft, and said that software was
treated no differently from some hardware.


Martyn was in favour of using formal methods on safety-critical software, and
also of qualifying the people who work on it. There are no formal
qualifications required before a practitioner may tackle a safety-critical
item of software, and yet, as Cliff Jones pointed out, programming
productivity has been found to vary by up to a factor of 10 between
individuals. "Error-prone" modules are regularly found in industry, and this,
Cliff speculated, could be due to their being written by error-prone
individuals.


Various disasters were cited, including the recent Gripen crash in Stockholm,
and the London Ambulance fiasco. Both have been well covered on the list.
A couple of anecdotes of interest concerned the man who called an ambulance
for his mother, who had suffered a stroke. After several hours, he took her
to hospital in his own car. The ambulance turned up 10 hours later. The crew
of another ambulance radioed base after being sent out on a call and asked why
the undertaker had managed to get there before them!


My favourite item concerned the garage owner who specialises in tuning
software. He gets the binary from the engine management ROM by mounting it
in some harness on his PC. With experience, he is able to spot the data
tables containing the engine parameters, and he alters the values. By trial
and error, he has found how to undo the "detuning" that is often designed
into these systems, and he makes the car give you "... a bigger kick in the
back, when you give it some boot!" (or words to that effect!).


Regarding Sizewell B PPS, which was discussed at some length, a "confidential
leaked document" had found its way into the producers' hands, and was
quoted at length. Presumably, this was the same as the one that was featured
on BBC Channel 4 TV news not long ago, and is now being sold for two pounds
by Computer Weekly? I'll let you know when I receive my copy! :-)


Peter Mellor, Centre for Software Reliability, City University, Northampton
Sq., London EC1V 0HB, Tel: +44(0)71-477-8422, JANET: p.mellor () csr city ac uk