A starting point for a discussion on marrying Agile methods and CMMI.Decisions without Data
Posted by agilecmmi on Apr 22, 2010 in Data, contingency, continuity, decisions, planning, standards | 5 comments
As many people know, for the six days ending Tuesday this week, the UK along with much of Europe has been in a virtual travel “lock-down” when it comes to commercial turbine-engine air travel.
The instigation of this situation has been the eruption of the Eyjafjallajökull volcano in Iceland last week whose plume of ash and debris was carried by the wind and jet stream straight over to Europe.
We humans are no match for one-two-punch of geology and meteorology, but how we respond to events such as these is entirely within our control. It appears that the collective wisdom in Europe had no contingency planning for what to do in this sort of situation. As a result, the air-space lock-down went on for days — many argue, now, it should not have lasted more than 48 hours, and, should never have resulted in a nearly system-wide blanket closure of air-space in most of Europe under any circumstances.
But even the lack of a plan for what to do isn’t the underlying cause, but merely a symptom of the root cause:
No defined standards, and, decision-making despite a lack of data.
As early as Saturday, 17 April, airlines were conducting their own flight tests with actual (but empty) passenger aircraft, and were returning information regarding the flight conditions in several places in Europe. By Sunday, at least four major airlines had conducted their own flight tests and were beginning to compare data and report the same thing: There are many places where it is not safe to fly, but there are *more* places where it is safe to fly and we should work out a way to exploit these places.
What sort of data was prompting aviation and meteorological officials in Europe to keep the sky closed?
Weather RADAR data and satellite imagery.
Weather RADAR data and satellite imagery showed an ash cloud spreading over Europe. On the face of things, this would prompt most rational thinkers to do what Europe did: progressively shut-down the airspace as the cloud made its way across Europe. (Ash-laden air doesn’t make for good compressible, combustible materials in air-breathing engines, not to mention the damage it would cause in the works.) However, since when did “on the face of things” ever really prove to be enough information?
There were two problems with using weather RADAR and satellite imagery both as a basis of determining the impact of the ash, and, as a source of data for making decisions:
- It doesn’t give you an accurate sense of proportion or density, and
- It can mislead you into seeing a more serious situation than exists.
What does weather RADAR look for?
Water.How deep into an ash cloud can weather RADAR see?
Not far past the outer boundary.What does ash look like on a weather RADAR?
A solid block of lead.How much ash density does it take for weather RADAR to freak-out and “see” a massive block of solid ash?
Not much at all.
OK, so now we’ve established that using weather RADAR alone isn’t sufficient from which to be making decisions, let’s move our discussion to a simpler, but more pervasive gap in Europe’s air-traffic planning:
They had no established standards for how much ash in the air is enough ash to cause them to shut-down commercial aviation and bring businesses, commerce, and economies around the globe to a serious, sputtering stall, (not to neglect the stranding of hundreds of thousands of people all over the planet, including myself), and putting many plans, deals, and families into a tail-spin.
Even when European agencies did send aircraft to the air to test it, they didn’t know whether the data they brought back was telling them things were safe or unsafe. They assumed “any ash is bad”. It wasn’t until the airlines got together with engine and airframe manufacturers to look at the data collected by the airlines themselves and use the governments’ meteorological data to come up with “safe” standards for air-ash-density, that the collective governments (the last of whom were in the UK and Ireland) decided to lift the air ban in a dramatic change-of-position late on Tuesday evening (European time).
Let me summarize:
- No standards.
- Data collected but not meaningful.
- Empirical data collected by equipment not intended for how it was being used and interpreted without true insight (literally).
- Decisions being made anyway.
What I’ve skipped in this post is all discussion of contingency planning, continuity planning, and challenges with communicating across dozens of countries, laws, and decision-making structures. Much of which were all gummed up throughout this mess for lack of thinking things through before the ash hit the turbo-fans.
The focus, here, however is on one crucial point, forget planning, because none of it would have mattered:
Europe was making decisions without data.
Are you?
Hillel
My professional passion is to build high performance organizations out of companies motivated to be lean, agile, and achieve world-class results. My best clients are companies who have the courage, leadership, insight, foresight and discipline to be the best places to work, the best value to their customers and the best performing for their shareholders. I take a tough love approach and, frankly, have little patience for executives who *want* these things but expect to achieve them without putting in any effort or making any changes.
5 Responses to “Decisions without Data”
This site.
The opinions expressed here are the authors' and contributors' and do not express a position on the subject from the Software Engineering Institute (SEI) or any company or SEI Partner affiliated with the SEI.
Your Host
Hillel Glazer
Principal & CEO Entinex, Inc.
Email me.-
I agree that decisions were made without data. However there is one omission in your statements about ash and radar – ash can be invisible to radar if it exists in a dry air mass and is in a low enough density. Even at that low density, it can cause catastrophic damage to turbine engines. This is what happened to British Airways Flight 9 in 1982 (http://en.wikipedia.org/wiki/British_Airways_Flight_9 ).
Sometimes extreme decisions must be made without adequate data in order to maintain public safety. In this case, what would the headlines be saying if Eurocontrol had NOT shut down the airspace totally, and one or more aircraft had crashed? How would the public have reacted?
So, was Eurocontrol overly cautious? Perhaps.
Have airlines cut corners on maintenance to save costs at the expense of safety in the past? Do they command their pilots to fly through known turbulence in order to save on fuel costs? Were they putting immense pressure on Eurocontrol to get back into the air in order to keep revenues flowing? Yup.
Given those data, I believe that despite the disruption to the general public the decision was the most prudent under the circumstances.
Dave Rooney
Westboro Systems
Hi Dave, Parts of your reply actually support my assertions about RADAR, as well as standards for flying through “ash clouds”.
(Also, for what it’s worth, the BA 9 example isn’t an example of “very low ash density”.)
In very low ash density, and in dry air, using RADAR to determine the scope of ash mass is just misguided.
But, we know that Eurocontrol was entirely unguided, so to be misguided about how to interpret weather RADAR isn’t a surprise.
There was a massive high pressure system over Europe and most of the eastern Atlantic on the days before, during, and after the volcano’s ejections of ash and debris. Yes, high pressure is generally dry air, but to be so dry as to render the data useless is something that didn’t seem to come into the picture — if it was even the case that it was so dry, I don’t know, I haven’t looked-up the weather data.
Also, for ash to not even show up on RADAR means the density must have been very low. Ash molecules are not smaller than water molecules, and for the density to be so low that they don’t even appear as water means the impact would have been unnoticeable.
Nonetheless, I’m not arguing that Eurocontrol *should not* have shut down the airspace. I’m sure there were several hours, if not days where there was significant ash all over the region. What I’m arguing is that it was likely shut down far longer than necessary, and, their strategy was mostly to wait and see what the volcano and the weather were going to do to the cloud rather than to work on coming up with standards and measures against them. The airlines and manufacturers took the lead on that.
It’s not that simple. Doppler weather radar is specifically tuned to ’see’ water in the form of precipitation. Actually, it’s tuned to see the movement of that water.
Ash isn’t water.
A quick Google search yielded this: http://cat.inist.fr/?aModele=afficheN&cpsidt=18291511 . They’re discussing a prototype algorithm for detecting volcanic ash with standard weather radar, and the paper is from 2006.
Again, human safety trumps economics. I’m betting that no one on board BA 9 or KLM 867 would argue with Eurocontrol’s decision.
Hoo boy. It seems that whenever something inconvenient happens on a relatively large scale, armchair experts start popping up like mushrooms over rain. First of all, Europe was not making decisions entirely without data. Granted, as Frank Furedi said ( http://www.spiked-online.com/index.php/site/article/8607/ ), there was not enough data for an accurate risk assesment. However, claiming that the decision was based on no data at all is misleading.
For example, KLM Flight 867 clearly proved that volcanic ash can and does pose an air safety risk. Since it is currently unclear at what density the ash particles become a serious hazard, it’s prudent (or “worst-case thinking”, if you prefer) to err on the side of caution and preservation of human life.
As for the decision making process, let’s stop pretending once and for all that management and decision making is the same across the board, context-free and easily canned. I don’t think knowledge and experience in CMMi, on its own, makes anyone qualified to criticize the way the Eyjafjallajökull volcano eruption was handled. It doesn’t mean you shouldn’t voice your opinion — it’s a free Internet and you do have some interesting points — but it does mean that it should be taken with more than a grain of salt. As Steve Yegge said, “Have you ever legalized marijuana?” steve-yegge.blogspot.com/2009/04/have-you-ever-legalized-marijuana.html
Actually, Vojislav, not only has my opinion been validated by the new European process for dealing with the volcano, but I’m not just a CMMI person. (Which you would know if you actually read more about me before posting a personal attack.)
About the data matter: not enough data, not interpreting the data correctly, using the data incorrectly, drawing the wrong conclusions from the data, gathering the wrong data and making rules based on any of these miscalculations are all symptoms of a related issue. Data is not a simple matter and over-simplifying data is dangerous. Making mistakes with data is only better that not having data in that when you misuse data you look only slightly smarter than when you didn’t have data at all, but in the end, having the wrong data has the same net result as having “no data” — you make bad decisions.
Whether in deciding how to deal with the volcanic ash in the airspace or dealing with other matters, decisions without data (whether it’s “no” data or the wrong data, etc.) is a bad idea. And, the new rules for dealing with the ash (or in the future) validate the point that they now have much better data, modeling, methods, and decision structures than before. Actually, they didn’t have *any* before — which was really the point of my post. They were making decisions without knowing on what to base their decisions. In other words, they were reacting (which is more reflexive and involuntary) instead of responding (which is more deliberate and measured).
And, just for your information, I happen to be an Aerospace Engineer and a pilot. So, before you open your mouth about what I am or am not qualified to speak on, go get your data.
Thanks for proving my point.