What to Measure?

Of all the process areas, one ML2 PA stands out that causes much contortion, consternation, and non-value added (when performed poorly): Measurement and Analysis (MA).

Why?

Simple. People don't realize how easy it can be, and if it proves not so easy, the real questions are then: why are you using CMMI for process improvement? why, in fact, do you want to improve your processes?

These questions become very uncomfortable for organizations whose only real reason for using CMMI is for their rating.

I'm a stickler for insisting that organizations actually want long-term value-added improvements to result from their use of CMMI. That's code for not just the rating.

All the PAs in CMMI are to improve the development of products and services. That means that there is a beneficial contribution of the goals of every PA towards the improvement of processes that affect development of products and services. By inclusion, then, even Measurement and Analysis (MA) should be about things that affect your projects and products. In fact, the more proactive the measures and the analysis, the better. But, I'd settle for measures and analysis that simply indicate something of value about the organization's process efforts... unfortunately, that seems to be the elusive piece.

Let's start with a quick recap of what's in MA:

Goal 1: Know what you want to measure, why you want to measure it, how you'll collect, store and report the measures, how you will analyze what you collect, and what information and decisions these measures and analyses support.

Goal 2: According to Goal 1, go get the measures, analyze, store and report them and make appropriate decisions.

What's so hard about these?

It seems people are scared witless over numbers. It all starts in Goal 1. Many folks, it seems, in their worry over finding any measures start using their imaginations instead of following the simple formula provided by the PA.

The formula goes something like this: "We need to know   (information need)   because we want to improve   (process activity)   to see improvements in   (information objective)   (or to the measurable extent      )."

In other words, start FIRST with the information your organization needs to fulfill information and process objectives. After identifying these needs and objectives, everything else should fall out nice and easy.

Here's another tip: Ask yourselves the following question: You're spending money, resources, time, etc., on processes, and process improvement, right?  What would you like to know about where your money is going?  What would you like to know about what you're getting for it?

In fact, in every PA there's a practice called GP2.8, whose short name is Monitor and Control the Process.  In many ways, this practice can break the measurement quandry and solve any organization's MA needs.  If they can't think of anything else to measure about their organization (utilization and bill-ability not being sufficient -- for reasons to be discussed later), GP2.8 should make it easy.

Think of GP2.8 as asking the following question in each process area: What is going on this process area that when we measure it tells us something of value about our activities in this process area?

If you can't answer that last question, then you must ask yourselves whether or not you're performing the goals of the PA in a way that truly adds value.  Continued trouble with this question indicates that your process improvement effort may not be properly scoped... or worse... may have the wrong goals.

The truth of the matter is that CMMI is very useful for many things, but not everyone can easily identify a value-added need to improve in every process area of a particular maturity level.  Until a value-added improvement objective is identified for each process area, GP2.8 will prove challenging.

Of course, there's more to GP2.8 than strictly measures of the PA activities, and more to MA than strictly PA measures.  GP2.8 can (and should) also look at the project's plan for performing PA activities and compare the actual activities to those in the plan.  And, MA can (and should) also be about other [non-PA] measures important to the organization.  Though, until you can identify improvement objectives more substantial than "achieve MLx", both MA and GP2.8 -- as well as much of CMMI -- will prove lacking in value and outcomes.

So, why aren't utilization and bill-ability valid MA measures?  Because if you were to try to go "fix" problems in utilization and/or bill-ability, which of your organization's processes would you go fix?  Utilization and bill-ability are like a gauge at the very end of a large, complex machine.  If all you see is that one gauge in the green, and you have no insight into how well any other part of the machine is doing, what do you do when suddenly the gauge is yellow or red, let alone how can you prevent the one gauge from going yellow or red?

While there are several PAs whose practices support maintaining effective utilization and bill-ability, it takes a level of sophistication and detailed understanding of process interactions to know how to connect utilization and/or bill-ability to these practices.  More generally, there are many things an organization may measure and/or want to measure that CMMI does *not* easily support.  E.g., marketing effectiveness, billing/payment efficiency, employee satisfaction, and others.

MA should be seen as being part of the overall improvement activities of the organization's processes, not just a process about measuring for measurements' sakes.  Measuring for measurements' sakes will definitely limit any organization's ability to mature beyond ML2, and, is most definitely not anything even remotely associated with being lean or agile.