Fractals, the solution to all of your scaling problems. Including Scaling Agile

It is no secret that I love planning. I'm not coming out of the closet now, that's been true forever! And at some point in my life I was even "cool" with that.

Additionally, I want you to know (although you will not yet understand why) that I still love planning. That's me :) Now pay attention, I'm about to shoot myself in the foot.

Loading the gun

When I was in college there was a topic that I loved, the topic was Information Theory. There's so much stuff in that area of research that I can't even begin to touch the tip of the proverbial iceberg, so I'll just say - for now - that information theory is an area of research that investigates information and how it is codified. For example: how do I compress a text file? Turns out, text can be very efficiently compressed. Perhaps the blog post you are reading can be codified in a few hundred bytes. The cool thing is why that happens: redundancy. Redundancy is an unappreciated quality of all systems.

So unappreciated in fact, that we all praise it's mirror twin: efficiency. A compressed text file is very efficient (i.e. it codifies a lot of information in a very small amount of disk space - I could probably compress this post into 140 bytes and tweet it out - that would be efficient).

If we can create so efficient representation of information why would we stick the old-skool inefficient representation (for example: language)? I'm glad you ask! The reason is that language with all its redundancy is easy to understand even if you cut parts out or mangle the letters. Try reading the following paragraph (click for a larger version):

Did you understand what that said? Of course you did! Redundancy saves the day! Yay!

Now about software and organizations

In organizations and companies, we also have redundancy - plenty of it! And just as well, because without it most companies and organizations would stop working altogether. Redundancy gives us resilience! Just like in the language example above: even with parts of the words cut out of the phrase you were able to understand it! And this is just how organizations work: through, and thanks to, redundancy. This is the reason why some "downsizing" efforts end up killing whole companies, and the often touted "efficiencies" or "synergies" leaders try to gain from mergers and acquisitions end up destroying economic value more often than they create.

The trick with redundancy is to repeat

By now you probably agree that redundancy is good - and it is. But how do you apply it to your organization and processes?

Before we go there, we have to tackle a very neat concept of mathematics. Fractals. Fractals have a property that is mind-boggling. Fractals are concepts that once explored end up generating infinite (yes, infinite!) information. In fact, a fractal line has infinite length even while fitting in a finite space! I won't bore you with the math details, but check this page on Wikipedia about the length of the British coast line - it has a neat demonstration of how a finite space can hold a line of infinite length.

This means that fractals are generative when it comes to information: they generate infinite amounts of information. And this happens to be a very useful property to have in mind when we explore how organizations work.

Making the case for infinity (and beyond)

In this post I argued that removing rules from your company's process book is actually better for your business and for your teams. The next step is to remove as many rules as you can, so that you end up with a small and simple set of generative rules - just like a fractal. Fractals are very simple equations that have in themselves an infinite number of solutions. And that's exactly what our processes should be: a small set of rules that, once in use, accommodate an infinite amount of possible behaviors - this is what I mean by "complex behavior" in the post on disciplined organizations.

Conclusion

Turns out fractals are perfect (yes, perfect - as in perfectly efficient) compression algorithms: a simple equation can be solved in an infinite number of ways, which when plotted in 2D or even 3D generate an infinite line in a finite space.

This property is extremely useful when applied to processes in your company because you cannot predict how people should behave in the future, but you can create an environment that - just like a fractal - allows every actor / person in the company to act in an infinite (and therefore practically unpredictable) number of ways and adapt to whatever the ever changing reality throws at them.

If you believe that your business environment is constantly changing, and that your organization is akin to a living organism you have to embrace the concept of fractal organizations.

Fractals work for you when they allow your blood vessels to reach every cell of your body (within a few cells distance), and when they allow your brain to store vast quantities of information even if it is small enough to fit in your head. Fractal organizations are organizations that can adapt in an infinite number of ways in response to an unpredictable environment.

If change is the only constant, how do I adapt to that?

Epilogue

Before we can understand how to apply the concept of fractal organizations and benefit from that, a very serious question must be answered: If people can behave in infinitely different ways, how do we prevent organizations from turning into chaos? That's a question we will explore soon - stay tuned! :)

Do you want to know more?

Title image credit: John Hammink, follow him on twitter.

What is Chaos? And how we found out about it...

As he looked at numbers e noticed that the oscillations of his model did not repeat.

In fact he entered the numbers again, and again, and again but his model would refuse to behave the same way twice.

During our long education we were taught that given the initial state of a system (the present parameters that define the state of the system) plus the equations that fully describe the system, you will be able to plot the future behavior of the system forever.

And we have plenty of evidence that this approach works. For one, we are able to predict when Comet Halley will next visit our corner of the galaxy. And many astronomers were able to predict that last visit thousands of years ago!

So, why was his model stubbornly behaving differently every time he punched the numbers in? After all he knew the system perfectly - he had designed it!

The best way he had to describe this "unpredictable" behavior was with the word "chaotic", a never ending sequence of never repeating patters. Nothing was the same even if the initial state was the same and he was the one defining the equations for this toy-weather model. I mean he had defined ALL the equations...

It took a few days, but he figured it out. He had entered the parameters with a precision of 1/1000, but during processing the computer executing the model would use numbers with a precision of 1/1000000. On initial consideration this did not seem a relevant difference, after all a difference of 1/1000 was equivalent to having a butterfly flap its wings in China and having that create a storm in North America.

Systems that would never repeat in behavior even if they ran for ever

Later, this and other experiments would be repeated all over the world, in many different domains but the results would be similar. All over the world scientists were discovering other systems that were "sensitive dependence to initial condition" (aka suffered from the Butterfly effect), the scientific definition for "chaos" which later became the popular term to describe systems that would never repeat in behavior even if they ran for ever. These systems exhibited infinite variety of behavior when certain conditions were met. What were those conditions? That we will explore in a later post

Photo credit: John Hammink, follow him on twitter

Dude! Where's my manager? or Why you should attend LESS2011

Many teams start their agile transition from the "developer-side". This is quite normal, developers (coders and testers) feel the pain more than others because they need to actually get the product/system finished. But by focusing on developers, aren't we missing something? Aren't we forgetting that many of the consequences we so detest come from un-informed decisions by people higher in the chain?

In my experience many agile transitions fail by not involving managers in the process. Sure, it is possible to change how your team works with minimal involvement from your manager, but at some point the team is constrained more by management decisions than actual technical practices or even understanding of the product.

I remember a story of a team that was on their path to agile adoption, but could not progress further because management had decided that certain tools should be used. The usual explanations were given: "IT can only support one tool", "we need to harmonize our tool landscape", etc. Whatever the reason for the decision what we can say is that in this case management had a real - and negative - impact on the team's capability to deliver working software.

Sure we can go rogue and use our own tool chain "hidden" from IT and our manager (and many teams have done that), but that is not a long term strategy. Sooner or later we will bump again against a set of decisions that will hinder us from progressing.

I believe we need a new model for Agile adoption. One that includes the managers, leaders, VP- C-level people in our organizations.

It is this belief that has led me to participate in a project to organize a set of conferences focused on the role of leaders and managers. Last year we organized the first of that set of conferences in Helsinki and named it LESS2010. This year we are continuing that project with LESS2011 in Stockholm.

There are many reasons for you or your manager to attend the LESS2011 conference in Stockholm. From the individual speakers (check-out our keynote line up) to the people you will meet. But If I had to name one reason it is this: Agile and Lean adoptions require our managers to understand the new mindset, and without that we are bound to fail. So, get yourself and your manager to LESS2011 and talk to other managers! Share your experiences and questions and come learn from people that have been facing long lasting agile transitions (the kind that requires management involvement)!

On rewarding and performance systems...

The ideas of "pay-for-performance", and it's twin sister "management by objectives" exist to try and increase the performance of a group and ultimately the whole company. The basic idea is: promise them a carrot and they'll work more / better. But is that true? Can a company really perform better by promising people they will get more money if they individually perform better?

It is an appealing though. Instead of working with the people, managers are advised to put the right rewards in place and let people loose. Things will magically work out, or so the theory goes.

Although there is some truth to that, the fact is that this simplistic view of reality hides, or ignores, the deeper impacts of the individual performance reward culture:

• First, by placing rewards at the centre of the performance system, the companies shift the conversation to rewards and away from work
  
• Most people feel they perform above average, but reward systems tend to grade people on "the curve". This leads the lower performers to get a higher grade than they deserve (but still lower than they think it should be) and the top performers a lower grade than they deserve because of the need to "follow the curve". This has the notorious effect of leaving everybody unhappy.
  
• Most importantly, the reward mechanisms that focus on individual performance completely miss the fact that a company exists in a complex environment. Especially non-trivial work (like software), is impossible to reduce to a set of repetitive steps. The immediate consequence of this is that, even if all the people perform at their best and get the top evaluation, the company could still totally fail to meet the level of their competitors. The performance of the system (the company) depends on the complex interactions between the different people in the company. If all try to optimize their personal performance, the system as a whole can perform a lot worse.http://www.blogger.com/img/blank.gif
  

What amazes me is that there are still people who advocate "pay-for-performance" in complex environments like software development.
The fact that we work in a complex environment all but ensures that "pay-for-performance" is at best indifferent to the overall company's performance and at worse an active value detractor for the company.

An additional bonus

Deming and Shewhart proved that in a stable system, trying to change it's performance from the outside can actually reduce the performance of the whole system. Deming called this "tampering": messing about with a system you don't understand (link to Deming and tampering).
"Pay-for-performance" and "management by objectives" are an instance of "tampering". Instead of messing about with a system we don't understand we, as managers, should focus our energy on understanding that system and changing the system through the use of small and controlled experiments.

This is not as easy as it sounds, but is the only way in which managers add value to their companies.

If you still want to have variable salary costs in your company, then use a profit sharing scheme: where everybody shares in on the success and pain, rather than individuals independently. Besides, can you really evaluate a single individual's contribution to the success of the whole company? (Especially when the company is not succeeding and that individual still claims their bonus?)

One question that is raised against the view I tried to explain is: "Fine, but how do I keep top performers? They certainly want to be rewarded by their good work, right?" My answer to this question is that people value many different things. While some people are influenced by short term gains, most people do not act consistently with that hypothesis. There's evidence that people react more consistently with (see this video):

• "meaning", i.e., they are more motivated and committed when they feel that their work has meaning; or
  
• "peer-recognition", i.e., people tend to feel more committed and motivated in an environment where their work is appreciated by their peers.
  

A better way to keep your best people is to work on creating an environment where people's work is recognized by their peers, and the meaning or nexus of that work is clear to everyone.

You, as a manager, have to focus on those things!

Photo credit: L2F1 @ flickr

LESS goes Stockholm for LESS2011, join us there in October

Last year I was involved in the organization of a unique conference. LESS2010 was a unique conference because it brought together different communities. We wanted to make sure that people with different points of view would discuss those points of view. The aim was to create a space for sharing ideas from different communities in the hope that new ideas would emerge. And emerge they did!

When the conference was done we had brought together the academic community which is involved in researching Agile adoption in the real world; the Agile community that is day-in-day-out applying Agile ideas in their own places of work; the Lean community and finally the Beyond Budgeting community that is sharing novel ways to manage and lead companies. This year we are doing the same, only better!

LESS goes Stockholm for LESS2011

You can check the details of the conference in the web-site, but here's a hint. We are working again on bringing different communities together!

The tracks that we have chosen are around the topics of Agile, Lean, Beyond Budgeting, Complexity Sciences, Systems Thinking and one topic that we deal with every day: Organization Transformation.

This year the trend is even more clear that LESS2011 is a conference for leaders in R&D organizations. Perhaps the only leadership focused conference in Europe at this point.

I do hope that more of these start happening because that's the next frontier for Agile and Lean adoption in our places of work.

So, I encourage you to submit a paper/session to the conference. Being there and talking to the the people with the experience and ideas is very important for us to continue our day-to-day work with new ideas and fresh energy!

See you in Stockholm!

You cannot transition to Agile. Stop and just embrace it!

I am writing this blog post to explore a concept. So bear with me, I'll probably ask more questions than I'll answer.

Why do most Agile fail in our companies (or government organizations for that matter)? My view is that we cannot actually transition from a command and control management paradigm to Agile / Complex management paradigm. The reasons are not fully clear to me, but I believe that it has something to do with the fact that we actually (typically) try to use a pre-determined way to make those transitions happen.

Case in point: When we try to move from Waterfall software development to Agile software development, we will typically draw a plan up for the transition with "steps" or "phases". Those "phases" or "steps" will typically be "stable points" in the evolution of our system (the company or organization). However, the Agile / Complex management paradigm assumes, at its core that software work is complex, therefore there is no predictable causality. The consequence of this is that the "steps" or "phases" in between the command and control paradigm and the Agile paradigm cannot themselves be "stable" in the sense that predictability can be recognized.

By following the argument above I'd state that: transitions fail because we try to move from a command and control paradigm to an Agile / Complex paradigm by applying command and control models. It is impossible to 'move orderly to a complex environment'.

What does it mean in practice for us? Well, for starters we cannot "plan" the transition in the same way we tried to plan our waterfall projects in the past. We can, and should have a goal or an idea of where we want to be. But after that we must embrace the new paradigm, or "Adopt the new Philosophy" as Deming put it. There are no intermediate steps between the "old command and control mindset" and the new "complex / agile mindset".

As this is an idea I'm still developing, I'll probably return to this subject and write some more, but in the meanwhile: what do you think? Does this make sense? What did you get from the above?

Photo credit: Marc Soller @ flickr

Stop fighting reality, take the red pill! A tale about managers

Today I'm on a mood for a rant. Be warned! :)

I constantly get gob-smacked (to borrow a term from a friend) about the lack of simple understanding of reality.

Let me explain. One particularly common symptom in management is that they tend to "believe" (that's the word) that they can affect what happens in a large/complex software development team. They believe that if they ask for more, pressure the teams that the teams will actually do what they ask them. That's almost never the case, and when it is you don't want to live with the result of that (quality problems, missed cases, etc.).

The more they push the teams to work on more things, the less they get. I call this the "reality is a bitch" problem. No matter what you think will happen you will always be surprised and there's nothing you can do about it... until you realize that you are not managing teams, you are managing the system in which they work!

Here's an example. A set of teams go away to plan the next iteration, they come back with their plans. Plans they believe in (i.e. a sprint backlog that is prioritized and estimated). During the Sprint all kinds of surprises happen (reality...) and then they deliver anywhere between 0% and usually 80-90% of what they had planned (it's less often that they deliver everything they planned and that's ok because the Sprint backlog was prioritized anyway).

After this manager scream bloody murder and pushes the teams to be more accurate! To plan more. Guess what happens? Yes, they do indeed plan more, (and in some cases, rare as they may be) plan better, but ultimately deliver between 0% and 80-90% of what they planned.

And here's where the manager's face hits the proverbial wall. The typical manager will try to find someone to blame (likely not himself) and increase the pressure by, perhaps, shouting at some innocent bystander like the project manager or the architect and demand more accuracy in the planning. You know the drill.

The problem with this picture is that the manager does not recognize that a group of teams will always deliver a certain % of their planned work (the larger the group the more reliably so). They may oscillate between, say 0% and 50% most of the time but their accuracy is realiable. What that means is that, any person aware of systems thinking will identify that there is a system at work that prevents the teams from being more accurate. And here is the lead for the informed manager: if you find such a situation don't waste your time shouting at people around you, instead go and study the system. Ask the question: "why can't they deliver on their plans?" and then ask why again (and again, and again). Investigate the deep causes for the planning failure. It is only then that you will be able to do something about it!

Embrace reality, understand reality and then change reality! Fighting reality is a waste of your time and just annoys everybody around you.

Photo credit: clawzctr @ flickr

Don't just plan, plan to be surprised!

Walking a tightrope without a net maybe good entertainment, but it's hardly what project teams enjoy the most. 

When the deadline comes and your team is late, you know why that is. No matter how well you plan (and you must) there will always be a volcano or a flood or simpler things like changing requirements or power-outages that will delay the project.

Delays are inevitable, we plan to get rid of the obvious ones, we do creative risk management to get rid of the less obvious ones, but ultimately it is impossible to avoid problems! It is smarter to be ready to deal with problems when they do happen.

This is why I liked this post by Dave Snowden about two seemingly contradictory strategies to deal with environmental disasters (or project problems, which ever applies to you). In the post he states:

They did recognise that some failures were inevitable, so they focused on speedy detection and fast recovery. In other words they adopted a strategy of resilience rather than one of robustness.  

There's no reason why you should use only one of these strategies in your projects, in fact they are complementary. Investing in planning is about implementing a "robust" strategy. Investing in adaptation mechanisms (like iterations, re-planning, demo with retrospectives, etc.) is about implementing a "resilient" strategy.

It is insane to think that a single-minded focus on "resilience" is a good idea, but so is a single-minded focus on "robustness"! Why do so many projects prefer either over the other? 

Photo credit: wollbinho @ flickr

Tired of useless boring planning meetings? Read on, I've got a solution for you

In a meeting today I was graphically reminded of how some of the basic concepts in software development still escape many of us. Case in point, the meaning of capacity.

In many people's minds capacity still means "how many man-hours we have have available for real work". This is plain wrong.

Let's decompose this assumption to see how wrong it is.

1.  First, in this assumption is implicit that we can estimate exactly how many man-hours we have available for "real work". The theory goes like this: I have 3 people, the sprint is 2 weeks/10 days, so the effort available, and therefore capacity, is 30 man-days. This is plain wrong!!! How? let's see:
   
   
   
   1.  Not all three people will be doing the same work. So, even if you have a theoretical maximum of 30 man-days available not all people can do the same work. If, for example, 1 person would be an analyst, another a programmer and finally the third a tester, then that would leave us with effectively 10 man-days of programming, analysis and testing effort available. Quite different from 30 man-days!
      
   2. Then there's the issue that not 100% of the time available for each people can actually be used for work. For example, there are meetings about next sprint's content, then there's interruptions, time to go to the toilet... You get the picture. In fact it is impossible to predict how many each person will use for "real" work.
      
   
   
2. Then there are those pesky things we call "dependencies". Sometimes someone in the team is idle because they depend on someone else (in or out of the team) and can't complete their feature. This leads to unpredictable delays, and therefore ineffective use of the effort available for a Sprint.
   
3.  Finally (although other reasons can be found) there's the implicit assumption that even if we could know perfectly the amount of effort available we can know how much some piece work actually takes from beginning to end, in exact terms! This is implicit in how we use the effort numbers by then scheduling features against that available effort. The fact is that we (humans) are very bad at estimating something we have not done before, which is the case in software most of the time.

The main message here is: effort available (e.g. man-hours) is not the same as capacity. Capacity is the metric that tells us how many features a team or a group of teams can deliver in a Sprint, not the available effort!

Implications of this definition of capacity

There are some important implications of the above statement. If we recognize that capacity is closer to the traditional definition of Throughput then we understand that what we need to estimate is not just size of a task, plus effort available. No, it's much more complex than that! We need to estimate the impact of dependencies, errors, meetings, etc. on the utilization of the effort available.

Let me illustrate how complex this problem is. If you want to empty a tank of 10 liters attached to a pipe, you will probably want to know how much water can flow through the pipe in 1 minute (or some similar length of time) and then calculate how long it takes to completely empty the tank. Example: if 1 liter flows through the pipe in 1 minute then it will take 10 minutes to empty a 10 liter tank. Easy, no?

Well, what if you now try to guess the time to empty the same tank but instead of being given the metric that 1 liter of water can flow in the piper for each minute, you are instead given:

• Diameter of the pipe
  
• Material that the pipe is built in
  
• Viscosity of the liquid in the tank
  
•  Probability of obstacles existing in the pipe that could impede the flow of the liquid
  
• Turbulence equations that allow you to calculate flow when in the presence of an obstacle
  

Get the point? In software we are in the second situation! We are expected to calculate capacity (which is actually throughput) given a huge list of variables! How silly is that?!

For a better planning and estimating framework

The fact is that the solution for the capacity (and therefore planning) problem in software is much, much easier!

Here's a blow by blow description:

• Collect a list of features for your product (not all, just the ones you really want to work on)
  
• With the whole team, assess the features to make sure that neither of them is "huge" (i.e. the team is clueless about what it is or how to implement it). If they are too large, split them in half (literally). Try to get all features to fit into a sprint (without spending a huge effort in this step). 
  
• Spend about 3 sprints working on that backlog (it pays off to have shorter sprint!)
  
• After 3 sprints look at your velocity (number of features completed in each sprint) and calculate an average
  
• Use the average velocity to tell the Product Owner how long it will take to develop the product they want based on the number of Features available in the backlog
  
• Update the average expected velocity after each sprint

Does it sound simple? It is. But most importantly, I have done many experiments based on this simple idea, and I'm yet to find a project where this would not apply. Small or big, it worked for all projects where I've been involved in the past.

The theory behind

There's a couple of principles behind this strategy. First, it's clear that if you don't change the team, the technology or any other relevant environmental variable the team will perform at a similar level (with occasional spikes or dips) all the time. Therefore you can use historical velocity information to calculate a long term velocity in the future! 
Second, you still have to estimate, but you do that at the level of one Sprint. The reason is that even if we have the average velocity, an average does not apply to a single sprint but rather to a set of sprints. Therefore you still need to plan your sprint, to identify possible bottlenecks, coordinate work with other people, etc.

The benefits

Finally the benefits. The most important benefit here is that you don't depend on estimations based on unknown assumptions to make your long term plans. You can rely on data. Sure, sometimes the data will be wrong, but compare with the alternative: when was the last time you saw a plan hold up? Data is your friend!
Another benefit is that you don't need to twist anyone's arm to produce the metrics needed (velocity and number of features in backlog), because those metrics are calculated automatically by the simple act of using Scrum.

All in all not a bad proposition and much simpler than working with unavoidably incorrect estimates that leave everybody screaming to be saved by the bell at the end of the planning meeting!

Photo credits:
uggboy @ flickr
johnmcga @ flickr

Setting targets actually decreases your performance. Don't believe me? See this video...

Many of the readers of this blog have probably faced this situation already in the past. When I was adopting Scrum in a local company, I was faced with the tyranny of setting targets. Targets and bonuses were part of that company's HR policies. "This is how we motivate people" as the saying goes.

Now picture this. When we started adopting Scrum we were, as expected, adopting also timeboxed software development (a key part of scrum). But the targets were that we should always be on time (+-10%).

So, get this: you get more money if you are on time (to motivate you to be on time, I guess) but your process is timeboxed! Easy money as they say...

The morale in this story is that targets are useless! If you want to improve the system (R&D for example) you need to manage the system, not set targets!

By adopting Scrum (effectively changing the system) we were able to be 100% on time! And that had nothing to do with the target setting, the system design (using Scrum) was the reason!

This is an insight that many managers don't have today: as manager you must design and manage the system. Setting arbitrary targets and not providing a method (system) is useless!

The video below of a conversation with John Seddon, is a very good explanation of how targets are useless and even counter productive. I especially loved his points:

1. there's no reliable method for setting a target
2. when you use targets or any other arbitrary measure you drive waste into your system
3. once you learn how to use measures derived from the work, and you involve the people who do the work you achieve a level of performance you would never have set as a target.

How about the mistakes we did by not doing anything?

I'm sure there are many good quotes from Russel Ackoff. Here's one that makes my day, I see this happening all the time!

"Schools and other organizations use accounting systems that only record one type of mistakes. If you do something you should not have done you pay the price, but not when you don't do anything.
Now look at the situation most of us are in: working for an organization that says that making a mistake is a bad thing. But only one kind of mistake can be caught: doing something that you should not have done. Now, if you want to maximize your security in that context what's your best strategy? Don't do anything!!"

-- Paraphrased from the video, emphasis added.

Starts at 8:55 until the end of the video.