Chaos, Agile, and Asking the Right Questions

In my last two articles (here and here) I talked about IT’s ongoing struggles to meet business needs. Over the past several decades, many studies have explored these concerns—performed by different groups, asking different questions, for different purposes—but with remarkably similar results.

The Bad News

Taken as a whole, there are two notable results from this large collection of studies:

(1) Overall, between 50% and 80% of all IT projects have measurable delivery issues.

(2) Perhaps more surprising, the results have remained basically unchanged over several decades. During this time, almost every facet of the IT industry has changed, and many of these changes were intended to solve exactly the problems that these studies were measuring—with little apparent effect.

Such results would be hard to believe (and easy to argue with) in any given study. When taken together, though, across such a spectrum, over so many years, it’s just hard to ignore the fact that there’s something amiss in software land.

Probably the best known of these studies (also the most continuous, and possibly the most contentious) is the CHAOS report, updated annually by the Standish Group. Here’s a refactored view of a small slice of their data from 2015, as reported by InfoQ:

Figure 1. Waterfall methodology: project results by project size.

Figure 2. Agile methodology: project results by project size.

Leaving aside certain questions about the survey methodology and specific results, let’s just take a 30,000’ view of what’s going on with these six cases.

Observation 1

The worst case scenario—waterfall methodology for large projects—is really awful.

No wonder, in one study from 2010*, 75% of respondents felt that their projects were “doomed right from the start.”

Observation 2

Agile is better, and significantly so.

Why is agile better? Three contributors some to mind:

As a methodology, Agile is designed to be responsive to ill-defined and changing business requirements. It’s about failing fast, and recovering fast. Clients get interim results sooner, which shortens feedback delays.
Agile drives better client engagement earlier in the process. An engaged client is better aware of what’s going on, and why, resulting in more realistic expectations and more forgiveness when things go awry.
Agile tends to shun commitments to a specific outcome at a specific time for a specific cost—especially commitments that are well into the future. Fewer commitments leave less room for disappointment.

It would seem, then, that much of the impact of Agile methodologies is largely a result of better managing business expectations!

Clearly, Agile improves success in projects. But ultimately, the business needs success in systems—systems delivered when and as needed, that can be readily adapted when and as business needs change over time.

These are not the same thing. Projects are largely an organizational funding construct, while systems are things that projects deliver.

So, at some level, studies like this are measuring the wrong things. Or, at least they’re not measuring the things that businesses ultimately need the most.

This is, in my view, the core flaw in such studies: they measure IT performance against project expectations rather than the resulting systems’ ability to meet business needs over a complete systems lifecycle (which is not the same as a software development lifecycle).

Studies of this kind tell us little about whether and how well Agile facilitates true business agility.

Observation 3

Given the caveats above, project success is still important. As a business investment in IT, a project is something we always want to succeed (even if the definition of success doesn’t match that in CHAOS).

Toward this end, the data shows that while Agile is better, it’s not enough better.

Even in the best case scenario—agile methodology for small projects—40% of projects exhibit difficulties (ie, way too many). This is less awful, but maybe not quite good.

Perhaps development methodology is less critical to these outcomes than some people would have us believe. This should not come as a surprise. Expecting better results from a development methodology will pretty much always disappoint if the problem is not a development problem.

What makes us think, perhaps reflexively, that all IT projects are development projects, for which either agile or waterfall methodologies are appropriate? Maybe the 40% of projects with challenged or failed delivery weren’t development projects.

Observation 4

Project size is a better predictor of success than development methodology. Smaller project size by far outweighs development methodology in improving project success metrics.

If my theory is correct, that the root cause of most IT project failures is actually business and/or system complexity, this is exactly what we’d expect to see! All else being equal, smaller projects are almost always less complex.

The Good News

As IT leaders, these observations lead us to a couple of easy decisions:

Move toward agile when you can. But remember that it’s not a panacea, so don’t oversell it to your business partners (much less to yourself).
Since smaller projects are better, more or less always try to only do small projects. However, given that most of us live in the real world, where large projects are the norm, this often comes down to how well we convert large, risky projects into smaller, less risky projects. Separation of concerns comes to mind here. Note that enterprise-scale architects come in handy for this sort of thing.

[Yes, I’m an architecture consultant, and this is a wanton act of (mostly) shameless self-promotion.]

And to one slightly harder decision, or at least harder to accomplish:

Above all, learn to ask the right questions. If your business must be agile (and which ones don’t?), dedicate focused attention on how well your systems will adapt over their complete solution lifecycle. This is less obvious, certainly to business leadership, so demands periodic attention from IT leaders.

This last point is critical. I’ll explore this thought further in future posts.

References
* Doomed from the Start? Why a Majority of Business and IT Teams Anticipate Their Software Development Projects Will Fail, Winter 2010/2011 Industry Survey. Geneca LLC, 2011.