Why projects fail, is why (we think) they succeed!

When I started my career as a Project Manager, I too was convinced that following a plan was a mandatory requirement for project success. As I tried to manage my first projects, my emphasis was on making sure that the plan was known, understood and then followed by everyone involved.

When I started my career as a Project Manager, I too was convinced that following a plan was a mandatory requirement for project success

I wrote down all the work packages needed, and discussed with the teams involved when those work packages could be worked on, and completed. I checked that all the dependencies were clear, and that we did not have delays in the critical path (the linear path through a project plan that has no buffer).

I made sure that everyone knew what to do, to the point that I even started using daily meetings before I heard of Scrum which would, later on, institutionalize that practice in many software organizations.

As my projects succeeded, I was more and more convinced that the Great Plan was the cause for their success.

As my projects succeeded, I was more and more convinced that the Great Plan was the cause for their success. The better the plan the more likely the project would succeed - I thought. And I was good at planning!

Boy, was I wrong!

It was only later - after several successful, and some failed projects - that I realized that The Plan had little effect on the success of the projects. I could only reach this conclusion through experience. Some of the projects I ran were "rushed", which made it impossible to create a Great Plan, but had to be managed "by the seat of the pants". Many of them were successful nonetheless.

In other cases, I did create a plan that I was happy with. Then I had to change it. And then change it again, and again, and again - to the point that I did little else but change The Plan.

Confusing the chicken with the egg, which came first?

The example above is one where I had confused the final cause (chicken) with the original cause (egg).

When something works well, we will often retrospectively analyze the events that led to success, and create a story/narrative about why that particular approach succeeded. We will assign a "final cause" to the success: in my example I assigned the "final cause" of project success to having a Great Plan, and the events that created the Great Plan.

This is normal, and it is so prevalent in humans that there is a name for it: retrospective coherence. Retrospective coherence is what we create when we evaluate events after-the-fact and create a logical path that leads from the initial state to the final state via logical causality, that can easily be explained to others. These causality relationships are what lead us to create lists of "Best Practices".

Best Practice lists are the result of Retrospective Coherence, and because of that many are useless

When the solution becomes the problem

However, this phenomena of Retrospective Coherence is not necessarily a good thing. In my initial example about Project Management I was convinced that the Great Plan and the related activities were the reason for success because that is what I could "make sense" of when I looked back in time. But as I gained experience I was forced to recognize that my "Best Practice" did not, in fact, help me in other projects. This realization, in turn led me to question the real reasons for success in my previous projects.

After many years of research and reflection I came to realize that many projects are successful purely by random reasons. For example: someone did an heroic effort to come to work during the week-end and recover the Visual SourceSafe database that had been corrupted once again and for the 1000th time!

But there are many other reasons why projects succeed by pure random chance. Here are some:

• In one project we had a few great testers that were not willing to wait to the end of the project to test the product. What they found changed the requirements and made the project a success
• Some projects were started so that we could deliver the "perfect feature set" to our customers. But as time went by and the deadlines were closing in, some managers - sometimes even me - understood that delivering on time was more important, and therefore changed the project significantly by reducing scope.
• Some developers were single handedly able to both, increase product functionality, while reducing the code base by 30%. This feat increased quality massively and made a delivery on time even possible.
• In one project we tried to use Agile. As a result, we started practicing timeboxed iterations and eventually ended up releasing so often that we could never be late

These are only a few of the reasons why projects succeed despite having a Great Plan, rather than because of it.

The Original Cause

The reasons for project success that I listed above are only a few that can be called "original cause" for project success. Original causes are those that actually start a chain of events that lead to success (or failure), but are too detailed or far into the past to be remembered while doing a retrospectively coherent analysis of project success (after-the-fact).

The kicker

But the kicker is this: when we get caught in "Final Cause" assignment through the retrospective coherence lenses or our logical mind, we lose a massive opportunity to actually learn something. By removing the role of "luck" or "randomness" from our success scorecard we miss the opportunity to study the system that we are part of (the teams, the organization, the market). We miss the opportunity to understand how we can influence this system and therefore increase our chances of success in the future.

Many people in the project management community still think - today - that having a Great Plan is a "Best Practice", and that you cannot succeed without one. I would be the first to agree that having a plan will increase your chances of success, but I will also claim that the Great Plan alone (including following that Great Plan) can never deliver success without those random events that you will never recognize because you are blind to the effects of chance in your own success.

In our lives we must, always, strive to separate Original Cause (what actually caused success) from Final Cause (why we think success happened by analysing it after-the-fact).

In a later post I will discuss how to increase the chances of project success by - on purpose - inserting randomness and chance into the project. Stay tuned...

Image credit: John Hammink, follow him on twitter

The three main trends in long agile adoption processes

I was having a discussion with fellow agile adopters yesterday and a bunch of interesting patterns emerged from the discussion:

• We are getting to agile/process improvement fatigue
• The management of software development is changing, but agile adoption processes do not have a mandate to change the management approach
• We don't have a process to learn at the organizational level

Here are some ideas I brought with me from yesterday.

We are getting to agile/process improvement fatigue

This is certainly a familiar sentiment among those of us that have been involved in agile adoptions for several years. One trend/common sentiment I felt yesterday is that agile and lean are seen as just another trend that will pass. Many are trying to look for "better" approaches in meta-approaches (like Lean over agile, or Toyota-driven ideas over Lean, etc.)
This meta-approach is seen as solving the problems with the previous approach.

There's certainly some truth to that. One cannot really improve without going deeper in the process of understanding, and that naturally leads to more abstract approaches. Something the Alistair Cockburn readers will recognize as passing from the Shu-level to the Ha-level.

I don't see this as a problem, I see that as a natural learning process. Agile in general and Scrum in particular have been around for long enough for people to recgnize it's shortcomings and to look for "better" ways. It's the learning process. The problem is when people don't really look for the theory behind the patterns they are trying to apply and just change patterns (say from Scrum to Kanban) without understanding why.

As long as we fail to recognize the theory behind what we are applying, we are doomed to make the same mistakes, over and over again.

The management of software development is changing, but agile adoption processes do not have a mandate to change the management approach


This was one trend that I raised in the discussion. Here's my argument summarized.
Agile is the result of a fundamentally different approach to management (analytical management vs. complexity-aware management). However, the management that is driving/leading the adoption of Agile software development is versed and informed by a completely different way of thinking. They live and bread a different paradigm of management. Until that changes it is impossible to make agile adoptions sustainable.

Symptoms of this problem are quite common and likely familiar to you. Managers set targets for agile adoption, but don't change the way they manage the organization (a requirement to meet those targets). Managers say they want agile adoption but at the same time try to micro-manage the teams, don't respect their need for self-organization and set all kinds of policies that go against teams taking responsibility over their work. And the list goes on.

Main point: there's a basic conflict between the analytical management mind-set and agile adoption. Until we solve that the best we can do (and it's a lot already!) is to get our teams to be agile, but not the structure that directly affects and limits their performance.

We don't have a process to learn at the organizational level

The point that follows from the previous is that in order to be able to get out of the analytical management paradigm we need to build a way for the organization to learn, to experiment and then adapt.

The lack of this method of learning, a meta-management method if you will, is what dooms us to be stuck in the previous paradigm. There is plenty of information around that can help managers learn and improve their work the same way we ask teams to do. But that learning is not happening in the vast majority of companies adopting agile. Until that happens agile is firmly stuck in the realm of the team developing software. It's like buying F1 tyres for a family car and expecting it to win a rally race.

Conclusion

What encourages me is to see that fellow colleagues in the agile journey also understand these issues and are (some of them at least) directly addressing these.

I hope they succeed and share their knowledge with the rest of us...

Photo credit: Denis Collette @ flickr