Confessions of a Quality Manager

Being the adventures of four jet-setting quality consultants who like to talk shop even more than they like good food and drink.

This is fantasy consulting. For the real thing, go to Fell Services' Quality pages.

Part I

Dietrich's nose was twitching: a sign of severe emotional stress. He was trying to do a root cause analysis and getting absolutely nowhere. It could be because we'd all met up for an impulse weekend in Athens, officially to see the Acropolis. Unfortunately, Horace, as a closet romantic, had insisted that we should spend most of the time tramping round the Greek concrete jungle in search of Byzantine churches. I'd have been happy just to look (briefly!) at the Church of the Metamorphis on Ermou, now that it's been redesigned as part of a pedestrian walkway, but Horace had insisted on looking in far too many little courtyards and poking his nose round far too many nondescript buildings. When he'd muttered about the juxtaposition of the profound and the profane, none of us had interpreted that as averting our eyes from the grey apartment buildings ("polikatikios" corrected Horace, grumpily, I think his feet hurt as well) and concentrating instead on the numerous examples of Art Nouveau wrought iron, all of which he'd meticulously photographed.

A small mutiny by everyone bar Horace had meant that we could take a break at the Acropolis, which, after all, was the reason we'd all converged on Athens in the first place. I wanted to see the plaque of the two Greek boys who tore down the Nazi flag in 1941, though Dietrich, who feels uneasy about his country's past, preferred to look at the Caryatid statues, copies only, four of them being in the Acropolis museum and the fifth in exile, thanks to Lord Elgin's acquisition and determination that it could only be kept safe in the confines of the British Museum.

In the interests of completeness, the names of these two guys were Maonolis Glezos and Apostolis Santas, though I'm ashamed to say I concentrated more on the spectacular view -- the buildings of Athens stretching into the distance, the green of the Olympic stadium by a hill of pines and the temple of Olympian Zeus. It's easy to believe in the Gods, faced with a view like that, though somewhat more difficult to imagine how King Aegeus could have thrown himself into the sea in grief that his son Theseus had been killed by the Minotaur. For one thing, Theseus was still alive (another example of assuming the worst) and for another, the sea is miles away. But I'm the one making assumptions now -- things have changed since Aegeus' time, the Eridanos river which once flowed through the city is now underground (covered by the Romans), I have no reason to assume that geographical features should remain the same for all eternity.

There was universal relief when we got back to the hotel and collapsed in the suite, ready for retsina, stuffed vine leaves and octopus tentacles. Everyone kicked off their shoes with varying degrees of relief, and Dietrich immediately started worrying about his root cause analysis. Looking up at our politely non-judgemental faces, he started stammering in embarrassment. His nephew was doing a school project, thought that his uncle knew everything, it was all practice and didn't we all do continuing professional development anyway?

If we'd liked Dietrich a little less and if we hadn't all been aware of occasions in the past when we'd done each other favours, we'd have explained (forcibly!) that weekends together were rare at the best of times and that, thanks to Horace, we'd walked what felt like hundreds of kilometres. We only had one more day of freedom which Horace had already suggested could be spent walking round Kerameikos, the ancient cemetery in downtown Athens. Horace, I think, was due for a disappointment -- Julio and I, and even Dietrich, had other plans for tomorrow.

If anyone could give a definition of root cause analysis, it would be Julio. He was very good at folding his hands piously and reciting definitions in a counter-tenor monotone. "Root cause analysis is the systematic process of gathering and ordering all relevant data about counter-quality within an organisation, then identifying the internal causes which have generated or allowed the problem, then analysing the comparative benefits and cost effectiveness of all available prevention options" he intoned, exactly on cue. Personally, I would have defined it a bit more succinctly, as the practice of determining the root causes of problems, root causes being the fundamental causes that, if removed or corrected, would preclude the occurrence (or reoccurrence) of the problem.

Root cause analysis, of course, is one of the tools of lean methodology. It's fundamental to the continuous improvement philosophy, since treating the symptom will not remove the cause and it's one of the fundamental tenets of the Toyota Production System. "What isn't?" asked Horace, who was not a fan of the TPS. (I'm sure his feet hurt -- for him to say something like that, he must have had at least two painful blisters).

It's so easy, when faced with a problem, to go for the obvious solution, especially since the symptom of the problem may be relevant to customer dissatisfaction. Supposing a customer complains that a package is late arriving or went missing completely. The instinctive thing to do is to concentrate on that customer -- make sure the next package arrives well in time or send a replacement immediately. But although this may please the customer for now and retain the customer's custom (a bulky phrase, but I can't think of a more elegant way of putting it), it won't stop the problem reoccurring at some time in the future. Worse, the extra time spent on the customer may mean that the next few customers don't get their allocated period of time or even, because you're rushing, a simple mistake may occur, leading to more customer dissatisfaction.

One classic example involves a pool of oil on the factory floor. The Plant Manager notices it, while on the way to the daily logistics meeting, and asks the foreman to see it gets mopped up. The next day, doing the same thing, the Plant Manager sees what looks like exactly the same pool of oil in the same place. There then follows a predictable scenario of blame and resentment. And more oil.

If the Plant Manager had been at all interested in root cause analysis, a logical question could have been why the oil was there at all. The foreman could have replied that it was due to a leaking gasket in the pipe above. Further investigation might have revealed that the supplier's gaskets had a bad record of leakage but that they had been selected as a supplier of choice because of the lowness of their original tender. And should the Plant Manager then ask why economy was used as the deciding criterion for selection of equipment, he might have received the answer that the directive from above (I mean, ultimately from him) had been that procurement, who did the ordering, had to be "extremely cost conscious".

The moral of this long-winded fable is that the Plant Manager was responsible for the oil pool himself, and the words "stone" and "glass houses" somehow come to mind.

The advantages of root cause analysis resemble those of other quality initiatives. The incremental approach of the process can lead to the identification of barriers and causes of problems which will ultimately lead to permanent solutions. Using root cause analysis can establish repeatable step by step processes, which are, of course, a good thing, said Horace, a true auditor. This logical approach means that a repeatable step by step process can be mapped and can then be documented and repeated as necessary.

On the other hand, although root cause analysis can be applied to any problem, it?s not always the preferred improvement technique. For a start, data is needed for any meaningful analysis, so that the mechanics of the situation are understood and obtaining this can be very time intensive. The technique is also better with processes, rather than individual occurrences, but is normally used for fairly major problems, with the caveat that the solution may actually be more expensive than a short term fix. One problem is normally found in a network of related problems and may have an interlocking system of causal roots -- and a situation can very easily arise when one factor could be at the root of two problems, but that each problem requires contradictory things for improvement. There was also the problem, as Dietrich had found, that repeating occurrences of the problem are separated by time (so people may not realise that they are reoccurring) or the situation may apply to different people (so there is no awareness that the problem is reoccurring).

posted by Dovya R @ 12:28 PM :

Saturday, December 14, 2002

Part II

So much for the philosophy of root cause analysis. It suddenly occurred to us that Dietrich hadn't mentioned what he doing root cause analysis of. There was a big groan when he sheepishly admitted that his nephew's school work referred to the Kursk.

Getting a root cause analysis framework for the situation isn't too complicated -- we needed to know what happened, when and what area or service was impacted. SSGN Kursk was designed by Rubin Central Design bureau and launched in 1994 with a complement of 107 men. The submarine contained ten watertight compartments (emergency hatches in the first and ninth compartments), 24 cruise missiles, six torpedo tubes and cost 226 million Russian roubles. It set out from port on August 10 2000 with 118 men on board, to take part in scheduled war games with the rest of the Russian Northern Fleet. On 12 August 2000, the Kursk was scheduled to find a formation of ships and strike at its main target with a salvo of torpedoes. The Commander reported that the first task had been carried out, but the submarine did not make contact to report the completion of the second part of the task. There were two explosions at 11.29 and 11.31 (Moscow time). This information was confirmed by the Norwegian seismic array service, the three non-Russian submarines near the war games area, plus the Russian submarine and cruiser which were actually in the area.

The Kursk had suffered a huge explosion and sank in about two minutes. It was later realised that as at 13.00 on that date, 23 men had assembled in the ninth compartment. The escape hatch of that compartment had been damaged in the explosion and no one was able to escape. However, the men estimated that they could survive about ten days before rescue arrived. At 19.00 the same day, three submariners started to charge the regeneration unit to produce more oxygen to freshen the stale air. It was dark -- there was no electricity -- and water was leaking into the compartment. One of them dropped the regeneration plates into the water and there was an explosion which consumed all the oxygen and gave off huge amounts of carbon monoxide. As this explosion was unexpected, the submariners were not wearing the breathing gear they had by them. About five people were killed instantly by this explosion. The rest inhaled the carbon monoxide and fell unconscious into the water where they drowned.

Following intense pressure from the Russian media and the crew members' families, the part of the Kursk containing the ninth compartment was raised a year later by a Dutch consortium, Mammoet-Smit.

If we'd been keeping to the root cause analysis schedule, we should have been considering why the accident happened, looking at each step and how it contributed to the event, assessing the human, organisational, equipment and environmental factors, both controllable and uncontrollable, together with the human resource issues. Unfortunately, this is an area of few facts and many opinions, not to mention contradictory reports. Publicly available information was secondary and each author had his hidden agenda.

The knee-jerk reaction was to assume that the Kursk collided with another ship or submarine. There have been eleven documented collisions between Russian and NATO submarines between 1967 and 2000, eight of which occurred in that area of the Barents Sea. Between 1946 and 2001, there have been at least 115 incidents involving collisions between submarines and civilian ships and, since I'm spouting other people's statistics, six nuclear submarines, including the Kursk, have been lost at sea.

The idea that the Kursk was sunk by friendly fire was immediately rejected unanimously by the Russian experts. None of the missiles or torpedoes used in the war games contained real ammunition, on grounds of safety, lack of necessity and, especially, cost. The area was off limits to civilian ships and a subsequent examination of the submarine hull showed that no surface ship could inflict that sort of damage.

Whether the Kursk collided with another submarine is more difficult to assess. The accident occurred in very murky water with a strong cross-current. There were three non-Russian submarines near the area: the USS Memphis, USS Toledo, and HMS Splendid, as well as two intelligence ships: the USS Loyal and the Norwegian Marjata, all shadowing the Russian exercises. The Memphis appears to have been closest to the Kursk; immediately after the explosions, it withdrew from the area and subsequently headed towards Bergen for a courtesy visit or to replenish supplies or for repairs (sources vary) at the suspiciously low speed of between 5-9 knots per hour. A pair of Russian naval reconnaissance aircraft tracked the progress of the USS Memphis and very nearly caused an incident when they approached Norwegian airspace too closely without warning. The Russian government asked officially for a visual inspection of the Memphis' hull (also the Toledo's hull); this request was declined by the Pentagon and the Memphis shortly departed Bergen, possibly for Plymouth. I can't find a record of anyone wanting to see the Splendid's hull, but then the British government denied that there was a British submarine in the area anyway. (See what I mean about contradictory sources -- it's difficult to know what's true).

The case for suggesting a collision between two submarines is plausible. It could have happened by underwater ramming while the Russian submarine was moving up to the surface and the hypothetical other submarine was submerging. Something resembling a conning tower fairwater of the type mounted on US or UK submarines was discovered about 50 metres away from the Kursk, though there was no indication of whether it was recent or old wreckage. It was reported that a green and white rescue buoy was seen by the nuclear cruiser Peter the Great, but that later disappeared (the Russian navy use only red and white rescue buoys, but the UK, US and Norwegian navies use green and white ones). Initial investigation also allegedly reported a metallic anomaly near the wreck of the Kursk. Considering that the area had been the Northern Fleet's combat training range for many years, it is reasonable to assume that the investigators would be aware of any established wrecks. It was suggested by the Northern Fleet's commander, Admiral V Popov, that the anomaly had been making international SOS signals, produced by an automatic mechanism. He claimed that Russian submarines do not have this type of equipment.

There are several other theories, some of which run perilously close to the fantastic. One which seems quite reasonable refers to the torpedoes in the Kursk. The submarine was due to fire a torpedo on August 12. Although there were two torpedo designers on board, additional to normal crew, the torpedo to be used was the same sort which the Russian Navy has been using for about twenty years. However, the Kursk torpedo to be fired did have a new upgraded accumulator battery. UK torpedo designer Maurice Stradling has postulated that there might have been a leak of hydrogen peroxide, an oxidising agent in the torpedo, which caused the first explosion. He suggests that the second explosion could have been caused by a fireball detonation of the stored torpedoes. It's not totally unprecedented: a similar explosion apparently wrecked HMS Sidon at Devonport in 1955, killing 13 men. All this was aired in a BBC "Horizon" programme, which drew on "secret government documents".

Under normal circumstances, we'd have gone on to draw up an implementation plan and probably some fault trees. We might even have written out each cause on a separate piece of card (or electronic card) and linked them all with arrows of varying thickness depending on the importance and relevance of the links. We'd probably have fooled around re-enacting a submarine doing the killer whale jump. But Julio, who was bored, found an unusually sensational Russian press report which mentioned several previous documented leaks in the hydrogen peroxide and fuel tanks of the torpedo and that some official maintenance documents had been falsified and bore dummy signatures.

We all suddenly sat up, realising the quality implications of this. Speculating about a submarine accident when all we had to go on were memories of a childhood viewing of John Mills' film "Morning Departure" was one thing; realising that there had been deliberate quality fraud was another -- if it was true. We didn't know whether we could believe a report from a Russian newspaper which appeared to have been translated into English by a Frenchman. We didn't even know if the newspaper was an attention-seeking sensationalist tabloid or a serious broadsheet. All we knew was that our game of root cause analysis had gone sour, that Dietrich had got way more material than his nephew could possibly need for his school project and that we'd been shocked back into the ugly reality of the hurried, inadequately met deadlines of the real world. It was a sad ending to a pleasant evening: and all we had to look forward to tomorrow was Horace herding us downtown away from sumptuous restaurants and bars and into a picturesque cemetery.

posted by Dovya R @ 12:26 PM :