How to choose the right project? Decision making frameworks for software organizations

Frameworks to choose the best projects in organizations are a dime a dozen.

We have our NPV (net present value), we have our customized Criteria Matrix, we have Strategic alignment, we have Risk/Value scoring, and the list goes on and on.

In every organization there will a preference for one of these or similar methods to choose where to invest people’s precious time and money.

Are all these frameworks good? No, but they aren’t bad either. They all have some potential positive impact, at least when it comes to reflection. They help executive teams reflect on where they want to take their organizations, and how each potential project will help (or hinder) those objectives.

So far, so good.

“Everybody’s got a plan, until they get punched in the face” ~Tyson

Surviving wrong decisions made with perfect data

However, reality is seldom as structured and predictable as the plans make it out to be. Despite the obvious value that the frameworks above have for decision making, they can’t be perfect because they lack one crucial aspect of reality: feedback.

Models lack on critical property of reality: feedback.

As soon as we start executing a particular project, we have chosen a path and have made allocation of people’s time and money. That, in turn, sets in motion a series of other decisions: we may hire some people, we may subcontract part of the project, etc.

All of these subsequent decisions will have even further impacts as the projects go on, and they may lead to even more decisions being made. Each of these decisions will also have an impact on the outcome of the chosen projects, as well as on other sub-decisions for each project. Perhaps the simplest example being the conflicts that arise from certain tasks for different projects having to be executed by the same people (shared skills or knowledge).

And at this point we have to ask: even assuming that we had perfect data when we chose the project based on one of the frameworks above, how do we make sure that we are still working on the most important and valuable projects for our organization?

Independently from the decisions made in the past, how do we ensure we are working on the most important work today?

The feedback bytes back

This illustrates one of the most common problems with decision making frameworks: their static nature. They are about making decisions "now", not "continuously". Decision making frameworks are great at the time when you need to make a decision, but once the wheels are in motion, you will need to adapt. You will need to understand and harness the feedback of your decisions and change what is needed to make sure you are still focusing on the most valuable work for your organization.

All decision frameworks have one critical shortcoming: they are static by design.

How do we improve decision making after the fact?

First, we must understand that any work that is “in flight” (aka in progress) in IT projects has a value of zero, i.e., in IT projects no work has value until it is in use by someone, somewhere. And at that point it has both value (the benefit) and cost (how much we spend maintaining that functionality).

This dynamic means that even if you have chosen the right project to start with, you have to make sure that you can stop any project, at any time. Otherwise you will have committed to invest more time and more money (by making irreversible “big bang” decisions) into projects that may prove to be much less valuable than you expected when you started them. This phenomenon of continuing to invest beyond the project benefit/cost trade-off point is known as Sunk Cost Fallacy and is a very common problem in software organizations: because reversing a decision made using a trustworthy process is very difficult, both practically (stop project = loose all value) and due to bureaucracy (how do we prove that the decision to stop is better than the decision to start the project?)

Can we treat the Sunk Cost Fallacy syndrome?

While using the decision frameworks listed above (or others), don’t forget that the most important decision you can make is to keep your options open in a way that allows you to stop work on projects that prove less valuable than expected, and to invest more in projects that prove more valuable than expected.

In my own practice this is one of the reasons why I focus on one of the #NoEstimates rules: Always know what is the most valuable thing to work on, and work only on that.

So my suggestion is: even when you score projects and make decisions on those scores, always keep in mind that you may be wrong. So, invest in small increments into the projects you believe are valuable, but be ready to reassess and stop investing if those projects prove less valuable than other projects that will become relevant later on.

The #NoEstimates approach I use allows me to do this at three levels:

• a) Portfolio level: by reviewing constant progress in each project and assess value delivered. As well as constantly preparing to stop each project by releasing regularly to a production-like environment. Portfolio flexibility.
• b) Project level: by separating each piece of value (User Story or Feature) into an independent work package that can be delivered independently from all other project work. Scope flexibility.
• c) User Story / Feature level: by keeping User Stories and Features as small as possible (1 day for User Stories, 1-2 weeks for Features), and releasing them independently at fixed time intervals. Work item flexibility

Do you want to know more about adaptive decision frameworks? Woody Zuill and myself will be hosting a workshop in Helsinki to present our #NoEstimates ideas and to discuss decision making frameworks for software projects that build on our #NoEstimates work.

You can sign up here. But before you do, email me and get a special discount code.

If you manage software organizations and projects, there will be other interesting workshops for you in the same days. For example, the #MobProgramming workshop where Woody Zuill shows you how he has been able to help his teams significantly improve their well-being and performance. #MobProgramming may well be a breakthrough in Agile management.

Picture credit: John Hammink, follow him on twitter

Hierarchies remove scaling properties in Agile Software projects

There is a lot of interest in scaling Agile Software Development. And that is a good thing. Software projects of all sizes benefit from what we have learned over the years about Agile Software Development.

Many frameworks have been developed to help us implement Agile at scale. We have: SAFe, DAD, Large-scale Scrum, etc. I am also aware of other models for scaled Agile development in specific industries, and those efforts go beyond what the frameworks above discuss or tackle.

However, scaling as a problem is neither a software nor an Agile topic. Humanity has been scaling its activities for millennia, and very successfully at that. The Pyramids in Egypt, the Panama Canal in central America, the immense railways all over the world, the Airbus A380, etc.

All of these scaling efforts share some commonalities with software and among each other, but they are also very different. I'd like to focus on one particular aspect of scaling that has a huge impact on software development: communication.

The key to scaling software development

We've all heard countless accounts of projects gone wrong because of lack (inadequate, or just plain bad) communication. And typically, these problems grow with the size of the team. Communication is a major challenge in scaling any human endeavor, and especially one - like software - that so heavily depends on successful communication patterns.

In my own work in scaling software development I've focused on communication networks. In fact, I believe that scaling software development is first an exercise in understanding communication networks. Without understanding the existing and necessary communication networks in large projects we will not be able to help those project adapt. In many projects, a different approach is used: hierarchical management with strict (and non-adaptable) communication paths. This approach effectively reduces the adaptability and resilience in software projects.

Scaling software development is first and foremost an exercise in understanding communication networks.

Even if hierarchies can successfully scale projects where communication needs are known in advance (like building a railway network for example), hierarchies are very ineffective at handling adaptive communication needs. Hierarchies slow communication down to a manageable speed (manageable for those at the top), and reduce the amount of information transferred upwards (managers filter what is important - according to their own view).

In a software project those properties of hierarchy-bound communication networks restrict valuable information from reaching stakeholders. As a consequence one can say that hierarchies remove scaling properties from software development. Hierarchical communication networks restrict information reach without concern for those who would benefit from that information because the goal is to "streamline" communication so that it adheres to the hierarchy.

In software development, one must constantly map, develop and re-invent the communication networks to allow for the right information to reach the relevant stakeholders at all times. Hence, the role of project management in scaled agile projects is to curate communication networks: map, intervene, document, and experiment with communication networks by involving the stakeholders.

Scaling agile software development is - in its essential form - a work of developing and evolving communication networks.

A special thank you note to Esko Kilpi and Clay Shirky for the inspiration for this post through their writings on organizational patterns and value networks in organizations.

Picture credit: John Hammink, follow him on twitter

Dealing with Complexity in Software Projects - The theory that explains why Agile Project Management works

Why do projects fail?

This is a question that haunts all project managers. Good and bad, experienced and beginners, Agile or non-Agile. The reason for that is simple: we believe that if we crack the code of why projects fail, we will be able to avoid failure in our own projects. After all who does not want to be in a successful project?

We believe that if we crack the code of why projects fail, we will be able to avoid failure in our own projects.

But before we can can answer such a difficult question, we need to understand the factors that influence project failure. Many will immediately say that lack of planning or bad planning are major causes for project failure. We've all been told that more planning is the solution. And we have been told that the "right" planning is the solution. Sure, we all know that, but does that "right kind of planning" look in practice?

Enter Agile...

We know that "the right planning" is the solution, but we need to have a functional definition of what that "right planning" looks like. Agile has contributed greatly to further our understanding of software projects. For example: thanks to Agile we now can confidently state that individuals and their interactions are a key enabler for successful projects, not just "the right kind of planning". But there is more...

We are also starting to understand that there are some fundamental failures in the existing Theory of Project Management. Thanks for researchers like Koskela and Howell[1] we even have a framework to analyze what is wrong - very specifically - with traditional Project Management. Most importantly, that framework also helps us understand what needs to be different for projects to succeed in the new world of Knowledge Work.

In the article below (sign-up required) I explore the differences between the existing theory of project management and what Agile methods (such as Scrum) define as the new way to look at project management.

This paper explains some of the ideas that are part of  the "Chaos Theory in Software Projects" workshop. In that workshop we review the lessons learned from Complexity and Chaos theory and how they apply to Project Management.

The goal of the workshop is to give project managers a new idea of what is wrong in the current view of project management and how that can be changed to adapt project management to the world of Knowledge work. To know more about the workshop don't hesitate to get in touch: vasco.duarte@oikosofy.com.

[1] Koskela and Howell, The Theory of Project Management: Explanation to Novel Methods, retrieved on April 2014
Picture credit: John Hammink, follow him on twitter

How Chaos Theory will influence management and management styles in the future

Managers all over the world are faced with a critical challenge to their role. They ideas about management and their management style is being challenged. And this is even more important because many managers have reached a position of in their career where they thought they could "take it easy". Nothing could be further from the truth.

Today the role of managers in all industries is shifting. And in no industry more than the knowledge industry.

In this video I explore why this is happening and where we may be able to look for solutions. I also present a concrete set of consequences that will affect you as a manager from the trends we are witnessing in the knowledge industry.

Do you want to know more?

Ready to explore what you as a manager can learn from The Science of Chaos?

You came to the right place! :) Mystes in Finland organizes a workshop about Chaos Science applied to the challenges of managing small and large knowledge work organizations. You can visit their site to know more about the workshop and to sign up. Places are limited. In that workshop I will touch on the following topics:

• Current theoretical base for managing projects
• What is wrong with managing software projects today and why?
• What can we learn from Chaos Theory and how to apply it in real life projects?
• A model for a successful project using what we have learned from Chaos Theory

Do you have specific questions that intrigue you? Send them to us and we promise to address them during the workshop!

Management science's impossible quest: in search of predictability

Many years ago some people came up with a brilliant idea: "what if, we could turn abundant lead (Pb on the table of elements) into scarce gold?". A brilliant idea to be sure, but as we know now, impossible. What made Alchemy grow and thrive as a science, was that people at that time did not know that it was impossible. They did not have enough knowledge of chemistry to know that you cannot turn an element's atom into another element's atom. That did not stop them however! And thanks to those crazy idealists today we have Chemistry. How about that for a twist of fate?

Alchemy's history can be traced back for millennia, and that name stands as the prototypical failed science (except for poets and … idealists). Failed because it based its existence on a belief that was impossible to make reality in our world.

I find many parallels between Alchemy and Management science today. Alchemy wanted to find the magic Philosopher's Stone that could turn led into gold, Management scientists are in a similar quest: in search of the method that can make management succeed as a science. Management science's goal today is to find the magic ingredient that can give us predictability.

Basic Science

Many management-fads are based on the hypothesis that you can control the world around you, mold it in a way that can give you predictability. This (mythical) predictability is a core assumption that enables the management's idea of success: I will do X so that I can achieve Y and therefore succeed.

This predictability is, however, a form of causality. Many, perhaps including you, will not think of it that way, but predictability is only present in systems where you can determine causality. A predictable world is one where we can reliably predict what happens when we do X, i.e. a world where we can establish causality.

Newton's laws gave us predictability in the physical macroscopic world of objects, because they told us what would happen if, for example, you rolled a sphere in an inclined plane, or shot a bullet from a canon, or shot an arrow with a bow. Relativity theory gave us predictability (to a certain extent) in the microscopic world of atoms, or sub-atomic particles also because it established useful and applicable causality rules

Medicine gives us (mostly) predictability in the world of diseases and biologic systems like the human body because it has a system of studying and documenting causality: if you take Aspirin after a great Saturday night party, you will feel milder effects of the oncoming hangover.

You can see why management theorists (and practitioners) would want to achieve the same goal: predictability.

Predictability is management science's Philosopher's stone.

The Philosopher's stone

Planning is one of those attempts at achieving predictability. We plan, because we believe that planning will give us the predictability we seek. Without the goal of predictability we would not plan, there would be no point.

But planning itself rests on the assumption that we can predict (here it comes again) causality in the world where the plan is to be executed. If we did not believe that we could predict B to happen after doing A, then planning would not be possible. It would be illogical.

We plan to be able to predict, but plans only work if we can predict in the first place!

The problem with what is stated above is that it is a circular dependency: we plan to be able to predict, but the act of planing rests on the assumption that the world is predictable. You see where this is going, right?

The sad fact is that there are many other management techniques that both rest on, and try to achieve predictability. For example: pay for performance, management by objectives, or the yearly strategic plan.

Going around in circles

All these management techniques have one thing in common. They are static. They exist at a precise point in time: Strategy is defined in the Spring or Autumn and is supposed to be executed for the next 12, 18, 24 or more months. Objectives or goals upon which our evaluation is based are set several months in advance of the point of evaluation.

All these techniques require perfect, 20/20 foresight. How likely is that?

The basic problem with most perspectives on management today is that they are static analyses of a future environment. And all decisions are made because we believe we can predict the future.

We make these predictions to help our businesses be more predictable, but our predictions depend on our business being predictable. This circular dependency is why most management approaches are doomed to failure. Like Alchemy.

Looking forward

We need a new Management Science, one that does not require the existence of a predictable world. When it defines goals, it does so in a way that is dynamic, i.e. the goals change with the observation of reality. This dynamic adaptation process is what we, in the Agile community, call a feedback loop.

A future-proof management approach must start by basing it's core assumptions on the existence of feedback loops that must be studied and tested as we make decisions. And these feedback loops must be the driving factors for action in management. Not the plans!

How would such a management approach look like?

The first assumption of such a management approach must be that the world is not predictable beyond a very short time. This may be a simple statement, but the consequences of this simple statement are fundamental.

• First, management techniques must not be based on the existence of a perfect, predictable future. Management techniques must be based on the acknowledgement that we cannot predict what is the best (or worst) possible outcome of our actions. What good is it to plan to sell 100 widgets when the market is - unexpectedly - demanding us to produce 1000 widgets? or 10 widgets?
• Second, management techniques must be designed to include feedback loops that inform action. Not just as information collection loops, but as fundamental action-defining loops. For example: every single decision is temporary. Every decision must state it's assumptions and when one assumption is proven wrong, the decision must be reverted or at a minimum re-evaluated. When we execute our plans, we must be alert to surprises; without detailing our assumptions we can never be surprised. Surprises are triggers for action - no surprises, no learning, no adaptation.
• Third, an individual's performance cannot be predicted up front. This tells us that we can only evaluate an individual's performance after we have the facts of what happened. Maybe a sales person failed to meet their sales target, but that could be because they helped R&D get the first pilot customers for an entirely new business!

What would your world look like if you believed and followed the three principles above? How would it differ from your current perspective on management?

Image credit: John Hammink, follow him on twitter.

The simple recipe for disciplined organizations

One question puzzles non-Agilists more than any other question. It is the question that uncovers why Agile does not fit their view of the world. A question that makes non-Agilists feel insecure and reject Agile completely or mostly. This question is: how can less structure, and less planning deliver software more reliably, and with higher quality, and faster, and with better usability, and with better customer satisfaction?

Do you know the answer to this question? Many Agilists don't. Here's one attempt at answering that question.

Agile is different

Those of us that have been practicing Agile for a while have learned to "break the rules" - or reach the Ha or Ri levels in the Shu-Ha-Ri model. We've practiced, and practiced, and practiced. We've learned what works (some times) and what does not work (most of the time), and we have been able to hone our practice to a point where Agile just feels "natural". But what is "normal"? How would you, a Ha or Ri level practitioner of Agile define Agile?

This reminds me of a story I heard from Angel Medinilla, a dear friend and fellow Agile journeyman:

In a dojo somewhere in a southern country, the Aikido apprentice reaches out to the sensei to understand the secret of Aikido.

Aikido apprentice: Sensei, what is the secret of Aikido?
Sensei: The secret is to breath, to balance, to connect with your opponent and whole body movement.

The apprentice was frustrated, he was expecting something enlightened and concrete for him to use.
Then, after 40 years of practice the apprentice finally says to his sensei:

Apprentice: Sensei, finally I understand the secret of Aikido
Sensei: Good. So, what is it?
Apprentice: It was simple and in front of my eyes all along. The secret is to breath, to balance, to connect with your opponent and whole body movement.

The sensei in that story knew that the apprentice would eventually come to realize that Aikido was indeed "just" to breath, to balance, connecting with the opponent and whole body movement. So, how about Agile, what is Agile all about? And how does it deliver more discipline, with less rules?

Chaos is very disciplined

While I tried to make sense of Agile, I explored many possible models. One of them was Chaos Theory. As I explored the Theory of Chaos I found a very interesting video about one of the favorite topics in the Agile community: emergence. Emergence is the process by which a system (any system) organizes itself based on real rules and exhibits what we would call complex behavior. For example, in this post I explained how a colony of Ants can exhibit complex behavior by following just a few rules. That complex, adaptive behavior emerges from the rules, and the action of the individuals. The rules are real (there is a benefit in following them), and the resulting behavior is complex and adaptable but also, very very disciplined (because there is a benefit to that discipline, you follow it or you die of hunger).

Going back in time

The counter-intuitive aspect of emergence is that it goes against all accepted "truths" about organizational design. You see, emergence can not be explained by reducing our model of the system to simply modeling of the agents' independent behaviors. Emergence is a result of all the parts interacting in the system. If you remove one of the rules from the ant colony you would get something different (possibly even the break down of the whole colony).

Simple rules lead to complex behavior. Complex rules lead to stupid behavior.
--adapted from original quote by Dee Hock

Reductionism is the most common process for organization design used by humans today. We tend to start by modeling small independent parts, and when we are satisfied with each, we put them together. But there's a problem with that approach. When you try to design an organization, or a process, by first identifying all the necessary steps individually, and then you sequence them orderly you make the system stupid. Or, in other words, you create rules that remove the possibility for complex, adaptive behavior to emerge.

This process or analyzing, ordering, prescribing organizational and process design is labeled analytic, and it is based on the theory that if you break something apart (like the organization), and analyze those parts independently you will understand it sufficiently enough to know how the whole system works. The problem is that an organization, like any system, has dynamics that can only be seen when all the parts are assembled and never when they are analyzed separately.

By contrast, if we define clear and simple rules (think the rules of chess) and nothing else, we are allowing the system to - through emergence - exhibit complex, adaptive behavior. The big question comes next: where do you draw the line between simple rules that allow for complex behavior to emerge, and the analytic process? When do we cross from one to the other?

Drawing a line in the sand

This is not an easy question to answer, and it is highly context dependent. However, I have found that the moment you think of rules that apply to single agents in a system you are approaching that line (and possibly crossing it). For example: when you have specific rules for every type of agent in the system you are designing through a process of Analytic Reduction. Like when you have specific rules for Project Managers to follow; you have specific rules for Developers follow; you have specific rules for Testers to follow; etc. ad infinitum.

A heuristic that I have found useful is to remove rules whenever possible. Scrum, for example, does this by assigning no special role to any team member, but instead giving the team a set of simple rules and boundaries (timeboxed iterations, interaction with PO, Definition of Done, etc.)

This realization led me to understand the critical role that generalists play in facilitating complex adaptive behavior (but that will have to wait for a later post).

Conclusion

Complex adaptive behavior emerges in systems where the rules are simple, and the agents have a high degree of freedom to make decisions. By trying to analyze systems with an reductionist process we are removing the possibility for agents (us, the people) to make intelligent decisions based on their context, and therefore we are reducing the possibility for complex adaptive behavior to emerge. The analytical mindset kills emergence. But more importantly, when it comes to performance Emergence kicks Reductionism in the groin.

If you tell people where to go, but not how to get there, you'll be amazed at the results. - George S. Patton

Next time you are designing a process or an organization take this into account and remove as many rules as you can. That will help you achieve more! Define the goals and let the people surprise you with their results!

Photo credit: John Hammink, follow him on twitter

Why projects fail, is why (we think) they succeed!

When I started my career as a Project Manager, I too was convinced that following a plan was a mandatory requirement for project success. As I tried to manage my first projects, my emphasis was on making sure that the plan was known, understood and then followed by everyone involved.

When I started my career as a Project Manager, I too was convinced that following a plan was a mandatory requirement for project success

I wrote down all the work packages needed, and discussed with the teams involved when those work packages could be worked on, and completed. I checked that all the dependencies were clear, and that we did not have delays in the critical path (the linear path through a project plan that has no buffer).

I made sure that everyone knew what to do, to the point that I even started using daily meetings before I heard of Scrum which would, later on, institutionalize that practice in many software organizations.

As my projects succeeded, I was more and more convinced that the Great Plan was the cause for their success.

As my projects succeeded, I was more and more convinced that the Great Plan was the cause for their success. The better the plan the more likely the project would succeed - I thought. And I was good at planning!

Boy, was I wrong!

It was only later - after several successful, and some failed projects - that I realized that The Plan had little effect on the success of the projects. I could only reach this conclusion through experience. Some of the projects I ran were "rushed", which made it impossible to create a Great Plan, but had to be managed "by the seat of the pants". Many of them were successful nonetheless.

In other cases, I did create a plan that I was happy with. Then I had to change it. And then change it again, and again, and again - to the point that I did little else but change The Plan.

Confusing the chicken with the egg, which came first?

The example above is one where I had confused the final cause (chicken) with the original cause (egg).

When something works well, we will often retrospectively analyze the events that led to success, and create a story/narrative about why that particular approach succeeded. We will assign a "final cause" to the success: in my example I assigned the "final cause" of project success to having a Great Plan, and the events that created the Great Plan.

This is normal, and it is so prevalent in humans that there is a name for it: retrospective coherence. Retrospective coherence is what we create when we evaluate events after-the-fact and create a logical path that leads from the initial state to the final state via logical causality, that can easily be explained to others. These causality relationships are what lead us to create lists of "Best Practices".

Best Practice lists are the result of Retrospective Coherence, and because of that many are useless

When the solution becomes the problem

However, this phenomena of Retrospective Coherence is not necessarily a good thing. In my initial example about Project Management I was convinced that the Great Plan and the related activities were the reason for success because that is what I could "make sense" of when I looked back in time. But as I gained experience I was forced to recognize that my "Best Practice" did not, in fact, help me in other projects. This realization, in turn led me to question the real reasons for success in my previous projects.

After many years of research and reflection I came to realize that many projects are successful purely by random reasons. For example: someone did an heroic effort to come to work during the week-end and recover the Visual SourceSafe database that had been corrupted once again and for the 1000th time!

But there are many other reasons why projects succeed by pure random chance. Here are some:

• In one project we had a few great testers that were not willing to wait to the end of the project to test the product. What they found changed the requirements and made the project a success
• Some projects were started so that we could deliver the "perfect feature set" to our customers. But as time went by and the deadlines were closing in, some managers - sometimes even me - understood that delivering on time was more important, and therefore changed the project significantly by reducing scope.
• Some developers were single handedly able to both, increase product functionality, while reducing the code base by 30%. This feat increased quality massively and made a delivery on time even possible.
• In one project we tried to use Agile. As a result, we started practicing timeboxed iterations and eventually ended up releasing so often that we could never be late

These are only a few of the reasons why projects succeed despite having a Great Plan, rather than because of it.

The Original Cause

The reasons for project success that I listed above are only a few that can be called "original cause" for project success. Original causes are those that actually start a chain of events that lead to success (or failure), but are too detailed or far into the past to be remembered while doing a retrospectively coherent analysis of project success (after-the-fact).

The kicker

But the kicker is this: when we get caught in "Final Cause" assignment through the retrospective coherence lenses or our logical mind, we lose a massive opportunity to actually learn something. By removing the role of "luck" or "randomness" from our success scorecard we miss the opportunity to study the system that we are part of (the teams, the organization, the market). We miss the opportunity to understand how we can influence this system and therefore increase our chances of success in the future.

Many people in the project management community still think - today - that having a Great Plan is a "Best Practice", and that you cannot succeed without one. I would be the first to agree that having a plan will increase your chances of success, but I will also claim that the Great Plan alone (including following that Great Plan) can never deliver success without those random events that you will never recognize because you are blind to the effects of chance in your own success.

In our lives we must, always, strive to separate Original Cause (what actually caused success) from Final Cause (why we think success happened by analysing it after-the-fact).

In a later post I will discuss how to increase the chances of project success by - on purpose - inserting randomness and chance into the project. Stay tuned...

Image credit: John Hammink, follow him on twitter

What is Chaos? And how we found out about it...

As he looked at numbers e noticed that the oscillations of his model did not repeat.

In fact he entered the numbers again, and again, and again but his model would refuse to behave the same way twice.

During our long education we were taught that given the initial state of a system (the present parameters that define the state of the system) plus the equations that fully describe the system, you will be able to plot the future behavior of the system forever.

And we have plenty of evidence that this approach works. For one, we are able to predict when Comet Halley will next visit our corner of the galaxy. And many astronomers were able to predict that last visit thousands of years ago!

So, why was his model stubbornly behaving differently every time he punched the numbers in? After all he knew the system perfectly - he had designed it!

The best way he had to describe this "unpredictable" behavior was with the word "chaotic", a never ending sequence of never repeating patters. Nothing was the same even if the initial state was the same and he was the one defining the equations for this toy-weather model. I mean he had defined ALL the equations...

It took a few days, but he figured it out. He had entered the parameters with a precision of 1/1000, but during processing the computer executing the model would use numbers with a precision of 1/1000000. On initial consideration this did not seem a relevant difference, after all a difference of 1/1000 was equivalent to having a butterfly flap its wings in China and having that create a storm in North America.

Systems that would never repeat in behavior even if they ran for ever

Later, this and other experiments would be repeated all over the world, in many different domains but the results would be similar. All over the world scientists were discovering other systems that were "sensitive dependence to initial condition" (aka suffered from the Butterfly effect), the scientific definition for "chaos" which later became the popular term to describe systems that would never repeat in behavior even if they ran for ever. These systems exhibited infinite variety of behavior when certain conditions were met. What were those conditions? That we will explore in a later post

Photo credit: John Hammink, follow him on twitter