Why do we use Agile?

The question in the title of this post is an interesting one. Many people use Agile for the right reasons (because they think that it can help them with their work in many ways), but there are some people that do it for the wrong reasons:

1. Because they like some part of one Agile methodology;
   
2. Because they read somewhere that it would improve productivity;
   
3. Because they like the word "flexible", as in "flexible releases";
   
4. ...
   

Agile (the movement) is often confused with agile methodologies. They are not the same.

Agile, as defined by the Agile Manifesto is a set of values and principles. These values can be followed in any methodology, even waterfall. Yes!, waterfall is not the antonym of Agile. Waterfall is just one of many methodologies used in software development (normally by non-agile organizations/teams).

Discipline is also not an antonym of Agile like Barry Boehm and Richard Turner proposed in their inaptly titled book "Balancing Agility with Discipline" (read a review by Ron Jeffries here).

Agile requires many of the same ingredients that made many non-agile projects successful (the 16.2% the Standish group refers to in this (PDF) instance of their famous Chaos report).

So what's the difference? You may ask. The difference is that Agile also proclaims a set of values that are not part of any methodology, rather they are values that should be applied independently of the methodology used in any software project.

What I'm trying to establish is that, although agilists tend to scorn at the waterfall methodology they - in my opinion - do so not because of the methodology itself, but because of a larger reason. The Waterfall methodology has been used by organizations that espouse the true antonym of Agile (as described by the Values and Practices in the Agile Manifesto): Command-and-Control.

Now would be time for another definition: "The Command-and-Control Movement", but I'll leave that for another post.

Systems Thinking applied to software

Henrik Mårtensson has posted two good pieces on systems thinking and software development in his Kallokian blog.

Check them out here and here. It is worth it.

I especially like this paragraph at the end of the second piece:

Organize to optimize the value stream (flow organization), not to keep people busy (functional organization)! Both Lean and Theory Of Constraints have proved, both empirically and theoretically, that this is the best way to go.

It's the motivation stupid!

This is something I've come across many times in my career. I constantly see motivated individuals excel against all odds and theoretically more prepared individuals fail when they should have made it with time left to spare.

Now Jason Yip just linked to an article about that same fact: "Motivation is more important than innate ability".

Check out the details. It's worth a reading, or two ;)

Command and control method of management

From Joel
:

In other words, the military uses Command and Control because it’s the only way to get 18 year olds to charge through a minefield, not because they think it’s the best management method for every situation.

Go and read it, especially if you feel you are in a Command and Control company ;)

Performance measurements can kill your efforts to bring Agile to your organization

Dave Nicolette has written a simple but really insightful piece on why performance measurement should be done with care and at the team level if you want an Agile team to gel and succeed.

The bottom line: measure what your team produces and let them decide who is not worth to stay in the team anymore.

Ricardo Semler in his book Maverick" touched the same subject, his approach: measure the performance one level higher than the people you are supposed to measure, this way you get people to work together for the company instead of people fighting each other in fruitless political wars.

In any case, performance measurement, if done wrong, can kill any efforts to bring Agile to your organization.

I have a really hard time trying to convince colleagues at work that Big Design Up Front (BDUF) does not work, that it is waste and the more time you spend coding the more likely you are to reach a better design solution. But before we go deeper into this discussion it is worth defining what, after all, is BDUF.BDUF is characterized by a few preconceptions about software development. They are: You should not make mistakes, therefore you should spend time up front making sure you will not make mistakes later If you don't design for (put your favorite non-functional requirement here) you will not be able to put it there at the end We need to make big changes to the (whatever module) so we need to really re-think the architecture and what we need to change to be able to do those changes before starting to program. This feature/project is so big that we must really design it properly before we can start coding The list would go on, I'll stop here.The problem with explaining why BDUF does not work is that iterative design is most effective in exactly the same situations as the proponents of BDUF say it would fail. Each of the reasons above could be used to propose Iterative Design (ID): You should not make mistakes, therefore you should try to make as many mistakes as possible as early as possible and learn from them before you release the software If you don't iterate your design for (put your favorite non-functional requirement here) you will not be able to design something that works correctly We need to make big changes to the (whatever module) so we need to go through each change one at a time with a proper support of unit tests and iterate the design while making sure we don't break anything that was working properly This feature/project is so big that we must really start small and build the design over several iterations. It cannot be designed up front. So what is the difference between ID and BDUF then? It is rather one of approach. While BDUF proponents defend that you should tackle uncertainty (which is at the heart of each of the statements above) with a pre-planned (pre-designed) approach. The proponents of ID argue that you should spend only enough time to come up with a solution that would work with the current environment, and then with time, adapt or change your design based on the additional information that you obtain while executing the design (don't forget the Unit Tests! ;).So, when faced with a "big/hard/complex change" a BDUF adept would design until he would be perfectly happy with the design before proceeding to implementation. In real life however things are not that simple. Most designs fail ("most" because at some point one will work), they fail because we work with complex environments that are hardly predictable enough for us to be able to produce reliable designs that touch every possible point of the known problem space.An ID adept would use enough time to have a design solution that would work. The main difference here is that an ID would not try to make the design full-future proof (i.e. ready to tackle all possible predictable situations), but rather concentrate on making a design that would work with the restrictions and situations to be tackled at the time of design (not hypothetical future possible situations). Later, if and when the design would fail in a specific situation it would be changed or altered.Many of the BDUF proponents have an issue with this approach because "intuitively" it seems that the ID approach is "wasteful" in that it does not produce a "ready" design but rather it produces a design to which programmers will have to come back again and again "because they did not spend enough time designing it properly" (so the comment goes).But that's just it! The ID approach is counter-intuitive, but it works. It is present in many successful industries (like Toyota's famous Production System) and it works.I'm sure at some point in time there will be a proper analysis "why" ID works better than BDUF, but until then trust the empirical evidence, empirical evidence is also important, not just theory (which is what the BDUF success is based on).So, finally, why doesn't BDUF work in my opinion? In more than 10 years experience I've seen BDUF and ID fail often, but the ID people are always able to recover from their failures faster and more often than the BDUF people. I guess the reason is: it's impossible to avoid mistakes, so any process that accepts and accounts for that will be more effective in the long run. Posted by

I have a really hard time trying to convince colleagues at work that Big Design Up Front (BDUF) does not work, that it is waste and the more time you spend coding the more likely you are to reach a better design solution.

But before we go deeper into this discussion it is worth defining what, after all, is BDUF.

BDUF is characterized by a few preconceptions about software development. They are:

1. You should not make mistakes, therefore you should spend time up front making sure you will not make mistakes later
   


2. If you don't design for (put your favorite non-functional requirement here) you will not be able to put it there at the end
   


3. We need to make big changes to the (whatever module) so we need to really re-think the architecture and what we need to change to be able to do those changes before starting to program.
   


4. This feature/project is so big that we must really design it properly before we can start coding
   

The list would go on, I'll stop here.

The problem with explaining why BDUF does not work is that iterative design is most effective in exactly the same situations as the proponents of BDUF say it would fail. Each of the reasons above could be used to propose Iterative Design (ID):

1. You should not make mistakes, therefore you should try to make as many mistakes as possible as early as possible and learn from them before you release the software
   


2. If you don't iterate your design for (put your favorite non-functional requirement here) you will not be able to design something that works correctly
   


3. We need to make big changes to the (whatever module) so we need to go through each change one at a time with a proper support of unit tests and iterate the design while making sure we don't break anything that was working properly
   


4. This feature/project is so big that we must really start small and build the design over several iterations. It cannot be designed up front.
   

So what is the difference between ID and BDUF then? It is rather one of approach. While BDUF proponents defend that you should tackle uncertainty (which is at the heart of each of the statements above) with a pre-planned (pre-designed) approach. The proponents of ID argue that you should spend only enough time to come up with a solution that would work with the current environment, and then with time, adapt or change your design based on the additional information that you obtain while executing the design (don't forget the Unit Tests! ;).

So, when faced with a "big/hard/complex change" a BDUF adept would design until he would be perfectly happy with the design before proceeding to implementation. In real life however things are not that simple. Most designs fail ("most" because at some point one will work), they fail because we work with complex environments that are hardly predictable enough for us to be able to produce reliable designs that touch every possible point of the known problem space.

An ID adept would use enough time to have a design solution that would work. The main difference here is that an ID would not try to make the design full-future proof (i.e. ready to tackle all possible predictable situations), but rather concentrate on making a design that would work with the restrictions and situations to be tackled at the time of design (not hypothetical future possible situations). Later, if and when the design would fail in a specific situation it would be changed or altered.

Many of the BDUF proponents have an issue with this approach because "intuitively" it seems that the ID approach is "wasteful" in that it does not produce a "ready" design but rather it produces a design to which programmers will have to come back again and again "because they did not spend enough time designing it properly" (so the comment goes).

But that's just it! The ID approach is counter-intuitive, but it works. It is present in many successful industries (like Toyota's famous Production System) and it works.

I'm sure at some point in time there will be a proper analysis "why" ID works better than BDUF, but until then trust the empirical evidence, empirical evidence is also important, not just theory (which is what the BDUF success is based on).

So, finally, why doesn't BDUF work in my opinion? In more than 10 years experience I've seen BDUF and ID fail often, but the ID people are always able to recover from their failures faster and more often than the BDUF people. I guess the reason is: it's impossible to avoid mistakes, so any process that accepts and accounts for that will be more effective in the long run.

Agile does not give you recipes, it makes you use your brain...

In a post in the Agile project management mailing list somebody asked if it would not be time for an Agile Project Management Body of Knowledge book (APMBOK) inspired, probably on PMI's PMBOK, the Project Management Body of Knowledge.

What I think about this question is that PMBOK and Agile represent completely different approaches to project management.

The PMBOK takes a deterministic detailed approach to project management that can be used (at least parts of it) in _any_ project be it software (agile or not) or building construction.

Agile is a movement based on values and principles (not prescriptions of "best practices" like the PMBOK).

You can be an Agile project manager and look at the PMBOK for "things to consider when managing a project", however, if you are looking for a prescription on how to do Agile project management you are (IMHO) looking for the impossible.

Agile does not prescribe what you must do, just what values and what principles you should look to for guidance, the rest is up to you.

This is probably one of the factors that makes Agile adoption so hard. People think they can just do stand-ups, retrospectives, TDD and everything else will work. That's not how it goes... those are only practices, what you should focus on are Agile values and principles: www.agilemanifesto.org.