Fractals, the solution to all of your scaling problems. Including Scaling Agile

It is no secret that I love planning. I'm not coming out of the closet now, that's been true forever! And at some point in my life I was even "cool" with that.

Additionally, I want you to know (although you will not yet understand why) that I still love planning. That's me :) Now pay attention, I'm about to shoot myself in the foot.

Loading the gun

When I was in college there was a topic that I loved, the topic was Information Theory. There's so much stuff in that area of research that I can't even begin to touch the tip of the proverbial iceberg, so I'll just say - for now - that information theory is an area of research that investigates information and how it is codified. For example: how do I compress a text file? Turns out, text can be very efficiently compressed. Perhaps the blog post you are reading can be codified in a few hundred bytes. The cool thing is why that happens: redundancy. Redundancy is an unappreciated quality of all systems.

So unappreciated in fact, that we all praise it's mirror twin: efficiency. A compressed text file is very efficient (i.e. it codifies a lot of information in a very small amount of disk space - I could probably compress this post into 140 bytes and tweet it out - that would be efficient).

If we can create so efficient representation of information why would we stick the old-skool inefficient representation (for example: language)? I'm glad you ask! The reason is that language with all its redundancy is easy to understand even if you cut parts out or mangle the letters. Try reading the following paragraph (click for a larger version):

Did you understand what that said? Of course you did! Redundancy saves the day! Yay!

Now about software and organizations

In organizations and companies, we also have redundancy - plenty of it! And just as well, because without it most companies and organizations would stop working altogether. Redundancy gives us resilience! Just like in the language example above: even with parts of the words cut out of the phrase you were able to understand it! And this is just how organizations work: through, and thanks to, redundancy. This is the reason why some "downsizing" efforts end up killing whole companies, and the often touted "efficiencies" or "synergies" leaders try to gain from mergers and acquisitions end up destroying economic value more often than they create.

The trick with redundancy is to repeat

By now you probably agree that redundancy is good - and it is. But how do you apply it to your organization and processes?

Before we go there, we have to tackle a very neat concept of mathematics. Fractals. Fractals have a property that is mind-boggling. Fractals are concepts that once explored end up generating infinite (yes, infinite!) information. In fact, a fractal line has infinite length even while fitting in a finite space! I won't bore you with the math details, but check this page on Wikipedia about the length of the British coast line - it has a neat demonstration of how a finite space can hold a line of infinite length.

This means that fractals are generative when it comes to information: they generate infinite amounts of information. And this happens to be a very useful property to have in mind when we explore how organizations work.

Making the case for infinity (and beyond)

In this post I argued that removing rules from your company's process book is actually better for your business and for your teams. The next step is to remove as many rules as you can, so that you end up with a small and simple set of generative rules - just like a fractal. Fractals are very simple equations that have in themselves an infinite number of solutions. And that's exactly what our processes should be: a small set of rules that, once in use, accommodate an infinite amount of possible behaviors - this is what I mean by "complex behavior" in the post on disciplined organizations.

Conclusion

Turns out fractals are perfect (yes, perfect - as in perfectly efficient) compression algorithms: a simple equation can be solved in an infinite number of ways, which when plotted in 2D or even 3D generate an infinite line in a finite space.

This property is extremely useful when applied to processes in your company because you cannot predict how people should behave in the future, but you can create an environment that - just like a fractal - allows every actor / person in the company to act in an infinite (and therefore practically unpredictable) number of ways and adapt to whatever the ever changing reality throws at them.

If you believe that your business environment is constantly changing, and that your organization is akin to a living organism you have to embrace the concept of fractal organizations.

Fractals work for you when they allow your blood vessels to reach every cell of your body (within a few cells distance), and when they allow your brain to store vast quantities of information even if it is small enough to fit in your head. Fractal organizations are organizations that can adapt in an infinite number of ways in response to an unpredictable environment.

If change is the only constant, how do I adapt to that?

Epilogue

Before we can understand how to apply the concept of fractal organizations and benefit from that, a very serious question must be answered: If people can behave in infinitely different ways, how do we prevent organizations from turning into chaos? That's a question we will explore soon - stay tuned! :)

Do you want to know more?

Title image credit: John Hammink, follow him on twitter.

What is Chaos? And how we found out about it...

As he looked at numbers e noticed that the oscillations of his model did not repeat.

In fact he entered the numbers again, and again, and again but his model would refuse to behave the same way twice.

During our long education we were taught that given the initial state of a system (the present parameters that define the state of the system) plus the equations that fully describe the system, you will be able to plot the future behavior of the system forever.

And we have plenty of evidence that this approach works. For one, we are able to predict when Comet Halley will next visit our corner of the galaxy. And many astronomers were able to predict that last visit thousands of years ago!

So, why was his model stubbornly behaving differently every time he punched the numbers in? After all he knew the system perfectly - he had designed it!

The best way he had to describe this "unpredictable" behavior was with the word "chaotic", a never ending sequence of never repeating patters. Nothing was the same even if the initial state was the same and he was the one defining the equations for this toy-weather model. I mean he had defined ALL the equations...

It took a few days, but he figured it out. He had entered the parameters with a precision of 1/1000, but during processing the computer executing the model would use numbers with a precision of 1/1000000. On initial consideration this did not seem a relevant difference, after all a difference of 1/1000 was equivalent to having a butterfly flap its wings in China and having that create a storm in North America.

Systems that would never repeat in behavior even if they ran for ever

Later, this and other experiments would be repeated all over the world, in many different domains but the results would be similar. All over the world scientists were discovering other systems that were "sensitive dependence to initial condition" (aka suffered from the Butterfly effect), the scientific definition for "chaos" which later became the popular term to describe systems that would never repeat in behavior even if they ran for ever. These systems exhibited infinite variety of behavior when certain conditions were met. What were those conditions? That we will explore in a later post

Photo credit: John Hammink, follow him on twitter

Kanban vs Scrum, the ultimate fight? Don't think so, here's why:...

Wow, what a week! A BIG post on Kanban by Scrum evangelist @jcoplien litterally put the blogosphere (and twittersphere on fire!).

It is good to have these family fights in the Agile family once in a while. As a life-philosopher once said: "These things gotta happen every five years or so, ten years. Helps to get rid of the bad blood".

But what are the differences between Kanban and Scrum, really? What are they?

Here's my take on the differences:

Kanban innovation

Kanban is, from it's origin a more systematic approach to measuring, visualizing and following up work in a "system" (one team, many teams, a company you name it). Thanks to the work by the Poppendiecks, David Andersson and Don Reinertsen (and others I'm sure) a pretty interesting and innovative economic framework as been put into the software process development lingo. Cost of Delay, Queues, optimize the whole, etc.
This economic framework is, in my view, the major innovation that Kanban proponents bring to the table. This was to be expected as many of the early adopters of Kanban were using Scrum before and felt the need to quantify and analyze some aspects of software development that Scrum did not tackle.

Scrum innovation

Scrum brought to the fore-front of process discussions issues that had never made it to our attention before: Self-organization, people/team dynamics, problem solving within a short cycle with feedback loops in place (Sprint), etc.
The major innovation of Scrum in my view was the introduction of the Socio-technical system of software development as Liker describes it in the Toyota way. Other similar software development processes took some of the ideas that Scrum also took, but shied away from the self-organization at the team level and blocker-removal focus of roles such as the scrum master. Those methods did not last long because the teams felt like puppets instead of adults with the possibility (and responsibility) to produce the best product they could.

Wrap-up

Both Kanban and Scrum brought significant innovations to the software development world. Those are just two types of innovations. There is more coming from the community and instead of focusing only on these two methods (which all of you should experiment with!) we should also look at what else is missing in the software development ecosystem.
My next interest area is "Complex Systems" (Complexity + Systems Thinking) and Management as a profession. I'll be exploring those subjects more and more in the future as a way to complement my use of Kanban *AND* Scrum. I suggest you do the same and find what else is missing in your environment and look for what else is around that could help you complement what you are already using. Here's a suggestion: start with @jurgenappelo's book on Management!

Happy reading, happy learning!

The three main trends in long agile adoption processes

I was having a discussion with fellow agile adopters yesterday and a bunch of interesting patterns emerged from the discussion:

• We are getting to agile/process improvement fatigue
• The management of software development is changing, but agile adoption processes do not have a mandate to change the management approach
• We don't have a process to learn at the organizational level

Here are some ideas I brought with me from yesterday.

We are getting to agile/process improvement fatigue

This is certainly a familiar sentiment among those of us that have been involved in agile adoptions for several years. One trend/common sentiment I felt yesterday is that agile and lean are seen as just another trend that will pass. Many are trying to look for "better" approaches in meta-approaches (like Lean over agile, or Toyota-driven ideas over Lean, etc.)
This meta-approach is seen as solving the problems with the previous approach.

There's certainly some truth to that. One cannot really improve without going deeper in the process of understanding, and that naturally leads to more abstract approaches. Something the Alistair Cockburn readers will recognize as passing from the Shu-level to the Ha-level.

I don't see this as a problem, I see that as a natural learning process. Agile in general and Scrum in particular have been around for long enough for people to recgnize it's shortcomings and to look for "better" ways. It's the learning process. The problem is when people don't really look for the theory behind the patterns they are trying to apply and just change patterns (say from Scrum to Kanban) without understanding why.

As long as we fail to recognize the theory behind what we are applying, we are doomed to make the same mistakes, over and over again.

The management of software development is changing, but agile adoption processes do not have a mandate to change the management approach


This was one trend that I raised in the discussion. Here's my argument summarized.
Agile is the result of a fundamentally different approach to management (analytical management vs. complexity-aware management). However, the management that is driving/leading the adoption of Agile software development is versed and informed by a completely different way of thinking. They live and bread a different paradigm of management. Until that changes it is impossible to make agile adoptions sustainable.

Symptoms of this problem are quite common and likely familiar to you. Managers set targets for agile adoption, but don't change the way they manage the organization (a requirement to meet those targets). Managers say they want agile adoption but at the same time try to micro-manage the teams, don't respect their need for self-organization and set all kinds of policies that go against teams taking responsibility over their work. And the list goes on.

Main point: there's a basic conflict between the analytical management mind-set and agile adoption. Until we solve that the best we can do (and it's a lot already!) is to get our teams to be agile, but not the structure that directly affects and limits their performance.

We don't have a process to learn at the organizational level

The point that follows from the previous is that in order to be able to get out of the analytical management paradigm we need to build a way for the organization to learn, to experiment and then adapt.

The lack of this method of learning, a meta-management method if you will, is what dooms us to be stuck in the previous paradigm. There is plenty of information around that can help managers learn and improve their work the same way we ask teams to do. But that learning is not happening in the vast majority of companies adopting agile. Until that happens agile is firmly stuck in the realm of the team developing software. It's like buying F1 tyres for a family car and expecting it to win a rally race.

Conclusion

What encourages me is to see that fellow colleagues in the agile journey also understand these issues and are (some of them at least) directly addressing these.

I hope they succeed and share their knowledge with the rest of us...

Photo credit: Denis Collette @ flickr

Don't just plan, plan to be surprised!

Walking a tightrope without a net maybe good entertainment, but it's hardly what project teams enjoy the most. 

When the deadline comes and your team is late, you know why that is. No matter how well you plan (and you must) there will always be a volcano or a flood or simpler things like changing requirements or power-outages that will delay the project.

Delays are inevitable, we plan to get rid of the obvious ones, we do creative risk management to get rid of the less obvious ones, but ultimately it is impossible to avoid problems! It is smarter to be ready to deal with problems when they do happen.

This is why I liked this post by Dave Snowden about two seemingly contradictory strategies to deal with environmental disasters (or project problems, which ever applies to you). In the post he states:

They did recognise that some failures were inevitable, so they focused on speedy detection and fast recovery. In other words they adopted a strategy of resilience rather than one of robustness.  

There's no reason why you should use only one of these strategies in your projects, in fact they are complementary. Investing in planning is about implementing a "robust" strategy. Investing in adaptation mechanisms (like iterations, re-planning, demo with retrospectives, etc.) is about implementing a "resilient" strategy.

It is insane to think that a single-minded focus on "resilience" is a good idea, but so is a single-minded focus on "robustness"! Why do so many projects prefer either over the other? 

Photo credit: wollbinho @ flickr

Setting targets actually decreases your performance. Don't believe me? See this video...

Many of the readers of this blog have probably faced this situation already in the past. When I was adopting Scrum in a local company, I was faced with the tyranny of setting targets. Targets and bonuses were part of that company's HR policies. "This is how we motivate people" as the saying goes.

Now picture this. When we started adopting Scrum we were, as expected, adopting also timeboxed software development (a key part of scrum). But the targets were that we should always be on time (+-10%).

So, get this: you get more money if you are on time (to motivate you to be on time, I guess) but your process is timeboxed! Easy money as they say...

The morale in this story is that targets are useless! If you want to improve the system (R&D for example) you need to manage the system, not set targets!

By adopting Scrum (effectively changing the system) we were able to be 100% on time! And that had nothing to do with the target setting, the system design (using Scrum) was the reason!

This is an insight that many managers don't have today: as manager you must design and manage the system. Setting arbitrary targets and not providing a method (system) is useless!

The video below of a conversation with John Seddon, is a very good explanation of how targets are useless and even counter productive. I especially loved his points:

1. there's no reliable method for setting a target
2. when you use targets or any other arbitrary measure you drive waste into your system
3. once you learn how to use measures derived from the work, and you involve the people who do the work you achieve a level of performance you would never have set as a target.

How about the mistakes we did by not doing anything?

I'm sure there are many good quotes from Russel Ackoff. Here's one that makes my day, I see this happening all the time!

"Schools and other organizations use accounting systems that only record one type of mistakes. If you do something you should not have done you pay the price, but not when you don't do anything.
Now look at the situation most of us are in: working for an organization that says that making a mistake is a bad thing. But only one kind of mistake can be caught: doing something that you should not have done. Now, if you want to maximize your security in that context what's your best strategy? Don't do anything!!"

-- Paraphrased from the video, emphasis added.

Starts at 8:55 until the end of the video.