Chaordic projects and the empirical quest

The scientific process relies heavily on empirical observations. We must, in the software development industry, also rely on empirical observations of successful software development projects and follow the learnings from those projects.

With software development projects the hard part is that most of what happens is not quantifiable and if it were - by decomposing its parts - we would loose of an important view: how all the parts interact together is often the reason why projects succeed or fail.

Successful software development projects are, in my empirical experience, Chaordic. Chaordic means, according to Dee Hock, the author of the word:

"1. The behavior of any self-governing organism, organization or system which harmoniously blends characteristics of order and chaos (...) 3. Characteristic of the fundamental organizing principles of evolution and nature."

In my experience, although I have been a project manager in many projects, the ones that I consider successful were in large measure not controlled by me or any other single person but rather a community of enrolled/engaged people.

Here's an example of the organization that emerged in one of the largest projects I've managed.

The team was divided into 2 large teams that developed architecturally/naturally different components (main teams). Each of these teams had a "project manager" or a person that was responsible to follow the development of each main team. For each component there was also a clear division of responsibilities in smaller groups (sub-component teams).

The project managers leading the 2 main teams met regularly with a larger set of stake holders, not just project related but also related to other important activities/functions in the company, this allowed the "management team" as it was called to build a holistic view of the project and how it fitted the company strategically (high level long term goals) and tactically (shorter term goals). This allowed the group to make decisions based on many parameters not only project-related ones.

As the main and sub-component teams interacted I could not - and did not try - to control how they worked together and interacted. The main teams worked largely on their own with a few synchronization points defined by all of the project participants in a kick-off meeting early in the project timeline.

The sub-component teams also worked independently and synchronized their work independently from the synchronization points that I mentioned earlier, this meant that the sub-component teams pretty much coordinated their own work in a self-organizing way.

What enabled the self-organization of sub-component teams was that each sub-component team member knew other sub-component team members quite well and almost all were co-located in one floor. Communication happened naturally and meetings were called only when needed without intervention from the main team project managers.

The items I want to call your attention to are:

1. The project was divided into "natural" divisions (those main teams had been working independently before);
   
2. The sub-component teams largely self-organized (they knew each other very well and the pace of new entries was slow enough to allow the new people to "absorb" the local culture and adapt quickly)
   

These characteristics have been present in other successful projects that I've managed or been involved with. The attentive reader will notice that the "natural", "self-organizing" and "organic growth" terms refer heavily to the "natural" or Chaordic organization I'm referring to in this post.

The Chaordic Age

In the "Birth of a Chaordic Age" Dee Hock describes through his own empirical observations and subsequent studies what he sees as a better way to look at how organizations (business or otherwise) work and should be organized. In this book, references to Nature and "natural" organizations are abundant. The author tries to convey the idea that organizations should also be "naturally" organized as opposed to "oppressed" into an unnatural structure through command-and-control approaches (more details in the book).

Natural team organization is important, and I still get surprised by how often I see "natural team organization" emerge in successful projects.