My Agile adoption story, and why it matters to you

Every person has their own different story of how they found out about Agile and how they came to experiment and ultimately adopt it in their projects.
Mine is also different than most, I started as a skeptic. I was one of those that said "it will never work". But I did come around. Here's the story of how it happened

The downfall of traditional waterfall

As many of the people that entered the software development industry during the 90's, I also had my first job in a small company. A start-up that was growing very fast and exhibited all the usual growth pains: more projects than people, lots of extremely hard-working but not experienced people, egos to match, etc.

As it happens, the typical way start-up companies deal with the growth pains is to hire a couple of experienced people that come in and "install the process" that will deliver us from all problems. And that's what happened to us. The company hired a few experienced project managers, they slashed the number of projects and created a process to structure and organize our work. We badly needed it.

However after a while we figured out that the new process was not helping us finish projects on time or with the right features or quality. It was better than chaos but not really that good. At this time I got interested in processes and organizational issues. My thinking was: "there must be a better way".

Re-discovering the wheel

What followed was something many of us in the industry have gone through and will continue to go through in the future. I started reading about other companies that had faced the problems we faced and discovered that it was not only us, and especially it was not only the software industry that was trying to apply the same old ideas and finding that they failed when applied in real projects.

I discovered Deming first, then Toyota, Six Sigma, the whole nine yards. I started understanding some of our problems: focusing on schedule and forgetting that quality is more important; separating work and creating handovers architects to designers, to developers, to testers, etc.

It was clear that what we were doing at that time (Rational Unified Process) was not a good option for us. That's when I started experimenting with different approaches.

My Path to Agile

I started by using simple RUP and focusing on managing the scope, but this not your garden variety of scope management. This was aggressive scope management, negotiated from start to finish of the project. The whole idea was to plot the progress against expected progress and remove content every time we saw a deviation.

This solution was OK for schedule-driven projects, but we still had the same basic problems of leaving risky or valuable things to later in the project, our teams were still very much silos working on partial features which lead to a lot of work partially done. This was not the droid solution we were looking for.

Agile appears on the scene

At that time a local research institute was pushing for more research on Agile Software Development projects. As luck would have it I was switching projects and the new project was selected as one of the pilots. Now, keep in mind that this was 2004, not the heyday of Agile yet, and I was a skeptic. I decided to take the project and prove conclusively that Agile was not the success story we were being told it was. Boy, was I wrong!

The Agile pilot project

We decided to do everything by the book to ensure that we really learned from the project execution and would then be able to find better ways of working based on that experience. A new team was hired, and we started slowly designing the process based on Scrum. Other teams were using XP for comparison's sake.

Then, a funny thing happened. As I read more and more about Agile, I started to find similarities between Agile and what I had read in Deming and Jeff Liker's work. There was the uncompromising focus on the customer, the respect for people, the empowered team with a Product Owner to steer the scope (which I had learned could work well), and more.

I was starting to feel that Agile was no longer a small thing to improve a dead wrong process. It was much more. It was, and is a paradigm shift the consequences of which are still dawning on me today, a good 5 years after the first project.

This is my story

I wanted to share this story with you because I know that many have had similar experiences or are going through that transformation right now. There are a lot more of us, going through the evolution and learning from our application of Agile. That collected knowledge and learning will hopefully improve our industry in the long run.

Agile Finland, a Finland based non-profit association wants to support more of these transitions and that's why we put together a program to support those of us that have started the Agile transition process.

Agile for the experienced

Myself and a group of other Agile Finland members decided that we would create a program to support those of us that have started the transition to Agile and are at a cross-roads, looking for improvements that cannot be found in the available literature. We also don't want to visit every single conference on Agile Software Development we can find. We want to learn from each other and from the top practitioners and researchers in the industry.

The Agile Coaching and Leadership Academy was created from this cooperation. But this is no longer a project by a small group of Agile practitioners. Agile Finland was able to attract a much bigger training institution to help us and vouch for the curriculum that we are putting together.

University of Helsinki joins the effort

Together with the Computer Science department at the University of Helsinki we created the basic program for the course, but now we need the most important ingredient: you, the people who want to learn and improve the performance of your organizations. We want to offer this opportunity to 15 people interested in not only listening to the best minds in the field, but also actively contributing by sharing your experiences and learning from other's experiences.

Check out the program and the details of the training program here and join us in improving our Agile knowledge.

Photo credit: jillclardy @ flickr

Patterns of Agility. How to recognize an Agile project when you see one...

Instead of fighting about "who's agile" or "who's more agile than whom", it would be useful to create a set of patterns, that once recognized would help us define if we are or have been able to successfully implement an Agile life-cycle for our project and portfolio.

These patterns or symptoms are useful for us when analyzing a project (e.g. for a retrospective) and assessing the progress of a team or a company from waterfall to Agile software development. These are necessarily only a small collection of patterns. Many more are needed to create a complete language that would help us define better what an Agile project should feel and look like.

Before getting deeper into the patters let me define at the outset what I mean by life-cycle. In my context life-cycle refers to a stage/phase in the life of a project. Example: a project that is "about to end" is in a different life-cycle phase than a project that is "about to start" or "ramping up". I define the following useful, although not exhaustive, life-cycle phases: Start, Middle, End.

In the following text I try to define what different types of projects (Waterfall, Linear Iterative and Agile/Incremental Iterative look and feel like.

Waterfall

Life-cycle phase: Start

In this life-cycle phase a waterfall project is typically in a phase where only the project management team is assigned to the project and the goal is to create a plan. The plan is typically composed of a set of requirements, an architecture target (maybe an architecture plan or just a high-level picture to be developed further), a budget and (most importantly) a project plan.

You know you are in a waterfall project when in the life-cycle phase Start you are mostly spending time in front of a presentation or word editing software and have some meetings (maybe weekly) where scope, budget and project plan are discussed. This phase typically ends with a "gate" where the project plan and Requirements + Architecture are approved.
A dead give away of a waterfall project is that, in this phase no one asks or worries about writing a single line of code. Some sophisticated waterfall projects may include a "prototype" in the life-cycle phase Start, but it will usually be a throwaway and developed by a sub-contractor who will not actually work in the project (after all the other teams are busy with the other projects).

Life-cycle phase: Middle

In this phase the project is in "full speed", which usually means that teams are working with very little coordination (after all they have a "plan" to follow). Typically components are assigned to teams and they work on those in isolation.
A dead give away that you are in a waterfall project is that your team does not integrate code with the other teams at all. Some sophisticated waterfall projects may have "sync-points" or "integration camps" where the component teams come together after a long period of development to integrate their code. These are grim affairs with much overtime and gritting of teeth.

Life-cycle phase: End

In this phase the waterfall project is "code complete" and aiming to "code freeze". There is usually a milestone/gate that was passed exactly at the start of this life-cycle phase called "code complete" or "feature complete". It is at that point that the teams of testers (typically many and in some far away country) are assigned to "test" the product. Obviously what they will be doing is a rather superficial sweeping of the floors to check that everything works. It is only after that that the project can pass the "code freeze" milestone/gate and the real testing can start.
You know you are in a waterfall project when the project runs out of time before it has quality enough to be released. The number of bugs being found is still very high, but probably at about the same level of the bugs being fixed. This is when "management" comes in and pushes all the teams to fix as many bugs as they can (pulling all-nighters or all-week-enders if needed). Much motivational language is used but the count of bugs being fixed stays about the same. The project will eventually ship something, but the actual quality level can only be vaguely understood. Business takes precedence in the decision making.

Iterative, aka Linear Iterative

Life-cycle phase: Start

In an iterative project the Start phase will typically be called Inception and will, like the waterfall projects focus mostly on defining requirements and creating/approving the project plan.
You know you are in an iterative life-cycle project when people start actually coding, but only a few features. Programming is still being "ramped up", but there's no milestone/gate that formalizes "start-programming" order.
In some sophisticated Iterative projects people will mention Use Cases, customers, experience, business cases and may even have a prioritized list of Use Cases to be implemented. Typically you will find that the Requirements document is quite shallow, delegating the clarification of most requirements to the actual "design" phase of the project.

Life-cycle phase: Middle

Now the teams are working very clearly on the code. Some Iterative projects may even have several sync points (see above) within this life-cycle phase. They will be easier than in a waterfall project, but "integration camps" are still common (although less frequent) and teams actively discuss the "code-line" policy (i.e. version control is an active part of project management.
A dead give away of an iterative life-cycle is that there will be several iterations of about 2-3 months in length at which point many teams try to integrate their code. In some sophisticated Iterative projects the teams will actually try to hold demonstrations of the existing code and in some (far fetched) cases there will be a project-wide retrospective at this point.

Life-cycle phase: End

You'd be tempted to think that Iterative life-cycle projects would have an easier "End" phase, but you would be wrong. Just like in waterfall, the teams develop a lot of their code in their own version control systems and integrate seldom (although much more frequently than in a waterfall project). In large iterative projects there will be "vertical" sub-projects (typically less than 100 people) that will actually integrate their code frequently but the multiple concurrent sub-projects will only really integrate their code at the end.
In this type of projects you will still find a very hectic "test and fix" phase at the end, but it will typically be focused around "complex use cases" as many of the simple use cases have already been tested during the "Middle" life-cycle phase.
You know you are in an iterative project because just like in waterfall Quality is an after-thought and you will find many, many bugs in the end of the project. The defect/error metrics will be the main focus of the project management team in this phase, up to the point that sophisticated statistical models will be created to try to predict when the project will end.
Don't be fooled, in a project where error/defect metrics are used as the main project management tool no one is in control. The project end date is essentially unpredictable and typically management will decide (arbitrarily) when the product should be released.
In some sophisticated Iterative projects I've seen that a length is pre-set for this life-cycle phase based on history and, ironically, that seems to hold pretty well (although not for the reasons you may think! -- stuff for another post)

Agile, aka Incremental Iterative

Life-cycle phase: Start

In an agile project this phase is typically extremely short. A product manager will present a proposal for a new product or a new version of an existing product. This proposal will be discussed and approved. One or more teams are assigned to the project for a short period of time (called Sprint or Iteration -- confusing, I know) and produce a running product, called a Product Increment.
You know you are in an Agile project when the team talks about "releasing" the software starting from day one. Each development team will include testers and each Requirement (aka story or feature) will be discussed in order to create acceptance criteria (aka conditions of satisfaction, aka test cases). These will typically be agreed before the team sets out to design the implementation but can also be updated during the Sprint/Iteration.

Life-cycle phase: Middle

In this life-cycle phase you will notice that the teams start to gel (if they are new) and the project gains momentum. The teams demonstrate regularly what they have accomplished and some teams will have stopped holding retrospectives, because they "have nothing to improve". The pressure is always high in an Agile project, but never too high, and in this phase of the project the team is already used to the pressure level, they know how to tackle their stakeholders.
You know you are in an agile project when the Product Manager is not the Product Owner and is never or almost never present in the Sprint/Iteration planning meetings. That work is delegated to someone in the team that knows the technical product and works with guidance from the product management to help the team manage and prioritize their backlog (aka Technical Product Owner).

Life-cycle phase: End

This is the life-cycle in which the Agile project will differ the most from the other two life-cycle models. In the end of the project the team will still be developing new features at full speed. No one will talk about "code freeze" or "feature freeze" in the project, but the testers and project management team will closely follow the Defect list.
You know that you are in an Agile project when the defect/error list is short and the selection/prioritization of defects/errors is easy. Some teams will just start their iterations/sprints by picking the most important defects out of the defect backlog, others will reserve some time/bandwidth in iteration/sprint planning specifically to improve the quality of their product.
The most noticeable difference however, will be that people will feel the pressure to add more features at the end, rather than the pressure to avoid changing any code. Agile projects typically deliver a large amount of "test" code (i.e. code that is there to test the production code) which makes them confident that they can change the code up to the last minute of the project.

Conclusion

I hope that this collection of patterns from different types of projects will help you talk about the level of agility in your project with your team.
We should really stop bickering about "how agile we are" and start defining how an agile project "feels" like. What patterns do we see, what benefits and constraints we have, etc.

At the end of the day, what matters is that you understand your context. That's the first step to changing it!

Photo credit: Bill Gracey @ flickr

Waterfall leads to 90% ready 90% of the time! (the very definition of unpredictability)

One of the most common anti-patterns I've seen in waterfall projects is the "90% done syndrome". It goes like this: the project will get quite quickly to 90% readiness and will look to be on time until that point. However, after the 90% readiness point is achieved we start to see more and more delays and blast through deadlines like there was no tomorrow.

Why is that? I've spent some time thinking about this question. With Agile projects we avoid this syndrome quite effectively by managing the scope. In practice we avoid the "90% syndrome" by declining to work on the content that would delay the project but not add enough value to justify that delay. But, then some time ago the question came back to me. Why is it that in waterfall projects we have the "90% syndrome"?

The answer to that question just hit me today (a real "d'oh" moment): the reason is that we don't really know about all of the requirements until we are well into the implementation (say 70-90% into it). Therefore we will only really understand the full content of the project when we are close to the end. This, coupled with the fact that waterfall projects tend to be organized architecturally/horizontally (instead of feature/vertical sliced) leads to knowing about critical requirements too late(90% ready for a long time) and not being able to remove content (because we release the lower layers before the upper layers) to meet the schedule.

This anti-pattern is quite simple, but apparently it takes time to understand the forces at play, and most managers (including project managers) constantly miss this point...

Photo credit: striatic @ flickr

Agile Scales, Waterfall doesn't - Video of a talk at Agile Eastern Europe

Sometime ago I did a talk about Scaling Software Development at Agile Eastern Europe in Ukraine.

The slides are up in my slideshare account, but now you can see the video of the talk that Agile EE made available.

In this talk I try to describe what "scaling" means and come to the conclusion that Waterfall-like practices just don't "scale". Check out the video and let me know what you think

Vasco Duarte, "Agile Scales, Waterfall Doesn't", Agileee conference, 2009 from Alexey Krivitsky on Vimeo.

Does Toyota really use waterfall for software development?

I saw this post and could not resist leaving a comment. Here's what I wrote as a comment to that post:

How can they [Toyota] apply waterfall and be predictive, fast and high-quality? Those are all qualities that waterfall lacks.

Predictability is lost because of lack of visibility into quality until it is too late (the test phase). Not to mention that trying to predict how a software project will go is like playing Lottery blindfolded and selecting a ticket to the wrong draw!

Speed is lost because writing requirements up front leads you to create a lot more requirements than you really need because you don't have the feedback of running software.

High-quality is lost because waterfall leads to slow and late integration which leads to many defects being left in the software! Not to mention that you can't inspect quality in, you have to build it in!

Now I'm curious about how you get out of that "they use waterfall but they provide all of the "goodies"!

Oh, and you can't say "they said they used waterfall but did not actually do that"! ;)

What do you think? Do you agree that Waterfall can be used with predictability, speed and high-quality.

The worst decision you could ever make, and will anyway do it...

I'm sure this has happened to you. Probably many times over your long career in software development. You wake up, take a long hard look at yourself in the mirror and courageously make that decision you had been wanting to take for the past few months. You go back to the office, take your manager aside and in a decisive tone deliver the news, the bad, bad, bad news: we need to rearchitect the system. All of it? No, that's the best part! Only this (very critical, absolutely critical) small part. You don't mention the "critical" part of course. You know better.

Your manager looks at you, silently thinks it over for about 0.35 seconds and then solemnly states: that sounds like a plan, make up the Gantt chart and deliver it to me, I'll take it to the boss.

And so it is done. You have taken the first step to re-write (from scratch, obviously) that part of the system. Did you make the right decision? You say only time will tell?

I beg to disagree, and so does Uncle Bob. You just made the most stupid, crazy-ass, fully incompetent, narrow minded, ignorant, and hole-digging decision of your life and soon you will regret it. Hopefully for the rest of your life, so that you don't do it again!

Re-architecting (read re-developing from scratch) a part of your system is the single most waterfall-ish idea that could cross your head -- and it will make you fail...

I've been a developer that made that decision, and still regret it today.

I managed a project where I was the one accepting to re-implement that part of the system, and still regret it today.

And if that was not enough a colleague at work did the same decision and he is already in his 3rd re-implementation of a module that should take 2 weeks to implement (his words) and he has been at it for more than a year... (his boss is still saying "ok" after thinking about it for 0.35 seconds...)

Note to self, use more simulations...

Great simulation description by Elisabeth Hendrickson, and I've got great first-hand feedback from this simulation. I can't wait to see it live!

Check out the post here.

The hard reality of life as developer...

If any of this is familiar, then you should really be reading this blog! :)

Hug a developer today.

Blogged with the Flock Browser

Project smells

Jason suggests that projects, in his experience, face similar problems (I'm assuming he is talking about all projects, including Agile projects).

What is your experience? What are the mistakes that happen all the time as Jason suggests?

In the waterfall world one of my favorites that happened all the time was: requirements change constantly. Can you imagine a world where requirements changes were a bad thing? I can't. I guess i've been exposed to Agile for too long (wink, wink).