Hierarchies remove scaling properties in Agile Software projects

There is a lot of interest in scaling Agile Software Development. And that is a good thing. Software projects of all sizes benefit from what we have learned over the years about Agile Software Development.

Many frameworks have been developed to help us implement Agile at scale. We have: SAFe, DAD, Large-scale Scrum, etc. I am also aware of other models for scaled Agile development in specific industries, and those efforts go beyond what the frameworks above discuss or tackle.

However, scaling as a problem is neither a software nor an Agile topic. Humanity has been scaling its activities for millennia, and very successfully at that. The Pyramids in Egypt, the Panama Canal in central America, the immense railways all over the world, the Airbus A380, etc.

All of these scaling efforts share some commonalities with software and among each other, but they are also very different. I'd like to focus on one particular aspect of scaling that has a huge impact on software development: communication.

The key to scaling software development

We've all heard countless accounts of projects gone wrong because of lack (inadequate, or just plain bad) communication. And typically, these problems grow with the size of the team. Communication is a major challenge in scaling any human endeavor, and especially one - like software - that so heavily depends on successful communication patterns.

In my own work in scaling software development I've focused on communication networks. In fact, I believe that scaling software development is first an exercise in understanding communication networks. Without understanding the existing and necessary communication networks in large projects we will not be able to help those project adapt. In many projects, a different approach is used: hierarchical management with strict (and non-adaptable) communication paths. This approach effectively reduces the adaptability and resilience in software projects.

Scaling software development is first and foremost an exercise in understanding communication networks.

Even if hierarchies can successfully scale projects where communication needs are known in advance (like building a railway network for example), hierarchies are very ineffective at handling adaptive communication needs. Hierarchies slow communication down to a manageable speed (manageable for those at the top), and reduce the amount of information transferred upwards (managers filter what is important - according to their own view).

In a software project those properties of hierarchy-bound communication networks restrict valuable information from reaching stakeholders. As a consequence one can say that hierarchies remove scaling properties from software development. Hierarchical communication networks restrict information reach without concern for those who would benefit from that information because the goal is to "streamline" communication so that it adheres to the hierarchy.

In software development, one must constantly map, develop and re-invent the communication networks to allow for the right information to reach the relevant stakeholders at all times. Hence, the role of project management in scaled agile projects is to curate communication networks: map, intervene, document, and experiment with communication networks by involving the stakeholders.

Scaling agile software development is - in its essential form - a work of developing and evolving communication networks.

A special thank you note to Esko Kilpi and Clay Shirky for the inspiration for this post through their writings on organizational patterns and value networks in organizations.

Picture credit: John Hammink, follow him on twitter

How injecting randomness into your project can help it succeed

Success and failure differ, very often, by very little. Take nature as an example. A small change in our DNA (a few mutated genes) can have catastrophic consequences. On the other hand, without these mutations humans would never have come into existence.

Humans and other species in the planet evolved because of chance (although not totally chaotic) changes in their DNA. As small change after small change happened, the surviving individuals were able to spread the viable, and ultimately the most adequate changes to the next generation - and the process repeats still in our own bodies today. If Nature has taught us something about improvement and evolution is that you must have a little bit of luck involved!

The DNA of a project

In software projects - my domain - I see some similarities to this natural model that are worth exploring. Projects also have a DNA which includes the structure and the communication connections between individuals. Communication being one of the most quoted causes for failure in software projects, and therefore one of its critical success factors.

The More I Practice the Luckier I Get

Once a project gets started, a particular structure is put in place (governance, management, teams, etc.). Very often that structure remains in place, and static until the end of the project. But is this a smart choice?

Another aspect of the project that rarely changes is the network of connections between the individuals. This network is what carries information from individual to individual and eventually "selects" the information that is acted upon by the project team. If a piece of information reaches an individual in the project, and that individual does not consider it relevant, that information will not spread further. In contrast, when a piece of information is perceived as important by an individual, that information will be actively spread by that individual.

The Illusion of Control

The structure that is set for a project strongly influences the communication links that can emerge between individuals. Who talks to whom? What meetings exist where people interact regularly? etc. But the reverse is also true. The existing communication network within an organization is a strong influence on what governance structure is put in place, how teams are formed, etc. These co-dependent processes create a positive (or negative) spiral of events for the project, but because neither (structure, or communication network) is often changed, that spiral is unstoppable. If it is positive, it will help the project succeed. If this co-dependent processes create, instead a negative spiral, that will relentlessly remove the chances of success for that project.

This co-dependent relation between two key processes (structure and communication network) is why we can increase the chances of success in our projects by simply causing random perturbations in the project. Randomness helps us explore different patterns for structure and communication networks.

As we carefully design and inject small - safe-to-fail - changes into the project, we can observe how it reacts and adjusts. Later, we can retrospectively amplify or remove/dampen the changes based on the outcome we see. If something works, keep it and ask how can you make it grow. If something creates a net negative outcome, dampen or remove it completely.

The cycle goes like this:

• Define and implement small, safe-to-fail changes in the structure or communication network for the project
• These changes lead to emergent behavior as the individuals and teams adapt to the changes
• A new project configuration emerges which drives new project results
• Finally, we evaluate the results of these changes and decide which components we will try to amplify or dampen

What Do You Mean Random?

There are many ways in which we can, randomly, explore better configurations for our projects. Below I list only a few to illustrate the concept:

• At the start of a project, let the teams chose their own composition. This establishes new connections between individuals in the project as some will choose to work with new team members. However, these new connections will not eliminate the previous connections between individuals. The net effect should be that your project now has a more connected communication network (more individuals with strong connections to each other). In practice this works just like when we form new neural connections in our brain: more connections leads to different thinking and acting patterns
• Organize common project events where people interact with each other outside the day-to-day routine. Planning events can be organized following a structured approach, but including also some unstructured time (à lá unconference, using e.g. Open Space) where people can interact based on their interests. This will also help form new connections between individuals in the project and spread information that would otherwise be locked down and not accessible
• Have regular project "coffee breaks" that happen at the same time and same place. In these events people can connect with other individuals and ask questions from the project management team or the most connected individuals in the project. This unstructured communication increases the chance that some piece of information will "jump the hierarchical borders" and reach people in decision-making positions, or people with influence that can later act on that information.

Conclusion

These practices are designed to inject randomness (unplanned situations or connections) into the project. The goal is not to create Chaos (a scary word for many project teams), but to generate new pathways for the information to flow within the project team, as well as novel project structures that are more adapted to the current challenges the teams face.

Using these (or other) practices will increase your project's chances for success. They don't eliminate the need for the project to be managed, or to have structure. They do increase the chance of success for the project by exploring new organizations and structures through a process of small (safe-to-fail) changes that can lead to unplanned, but ultimately superior performance.

These were just a few examples. How would you inject randomness into your project? Have you done that in the past? Please share your experiences in the comments for the benefit of others.

Photo credit: John Hammink, follow him on twitter

Communication is really hard! No, seriously!

So, we knew already that communication is hard. Nothing new there, but here's a few insights into why, the so called Wiio's laws.

Remember the discussion about why we should no rely on documentation to share information? How about this law:

1. Communication usually fails, except by accident

Sobering observation! Now, wouldn't you wish you had talked to that developer yesterday instead of sending him that e-mail? You would have avoided a lot of problems today, wouldn't you?

The Green Shift Anti-Pattern

This is a pattern that effectively prevents any team from succeeding, and in many cases just causes a whole organization to behave in a dysfunctional way.

Check it out. Has it happened to ? No, honestly!

Blogged with the Flock Browser