The Release Paradox: releasing less often makes your teams slower and decreases quality

Herman is a typical agile coach. He works with teams to help them learn how to deliver high-quality software quickly.

Many teams want to focus on design, architecture, or (sometimes) even on business value. But they are usually not in a hurry to release quickly.

Recently Herman conveyed a story to me that illustrates how releasing quickly can help teams deliver high-quality software much faster than if they would focus on quality in the first place. This is the case of a team that was working on a long overdue project. They had used a traditional and linear process in the past and had been able to release software only very recently, after more than 12 months of work on the latest release.

Not surprisingly, they were having trouble releasing regularly. The software was not stable; once it was live it had many problems that needed to be fixed quickly, and worst of all: all of this was having a direct impact on the company’s business.

The teams were extremely busy fixing the problems they had added to the product in the last year and could not focus on solving the root causes of those problems.

They were in full-fledged firefighting mode. They worked hard every day to fix yet another problem and release yet another hot fix.

This lasted for a few weeks, but once the fire-fighting mode was over, Herman worked with the teams to improve their release frequency. During their work with Herman, those teams went from one year without any release to a regular release every two weeks.

At first the releases were not always possible, but with time they improve their processes, removed the obstacles preventing them from releasing every two weeks and started releasing regularly.

What happened next was surprising for the teams. The list of problems after each release did not grow - as they expected - but instead shrank.

When some problems came in from the customers after a 2-week release, they were also much faster to fix the problem and quicker to release a fix if that was required. When the fix was not critical, they waited for the following release which was, after all, only 2 weeks away.

By focusing on releasing every two weeks, Herman’s teams were able to focus on small, incremental changes to their product. That, in turn, enabled them to fine-tune their development and release processes.

Here are some of the key changes the teams implemented

1. They started with a 4 week release cycle, and fine-tuned their daily builds and release testing process to enable a release every 2 weeks.
2. They invested time and energy to improve their test automation strategy and automated the critical tests to enable them to run “enough” tests to be confident that the quality was at release level.
3. They had some teams on maintenance duty in the first few iterations to make sure that any problem found after release could quickly be fixed, and released to customers if necessary.
4. They changed their source code management strategy to enable some teams to work on longer term changes while others worked on the next release.
5. They involved all teams necessary to complete a release in their iterations. This affected especially: production/operations team, localization team, documentation team, marketing team, and other teams when needed.

This list of changes was the result of the drive to complete each release and learning from the failures in the previous release. Some changes were harder to implement, and especially the testing strategy to allow for 2-week release cycles had to be changed and adjusted several times.

One of the key problems the teams had to solve, was the lack of coordination with departments that directly contributed to the release but were not previously involved in their day-to-day work.

This process lasted several months, and would not have been possible without a clear Vision set forth by the teams in cooperation with Herman, who helped them discover the right way to reach that Vision within their context.

Herman’s work as a coach was that of a catalyst for management and the teams in that organization. He was able to create in their minds a clear picture of what was possible. Once that was clear, the teams and the management took ownership of the process and achieved a step-change in their ability to fulfill market demands and customer needs.

Customers have no reason to change provider as they have an ever-improving experience when using this company’s services.

Today, this organization releases a new version of their product every two weeks. Unaware of it, their customers receive regular improvements to the product they use, and have no reason to change provider as they have an ever-improving experience when using this company’s services.

Picture credit: John Hammink, follow him on twitter

About when I stopped worrying and embraced Engineering!

Product Managers often forget that they can make or break a project. If your team is using Scrum that is even more true, because Product Managers are (or should be) involved regularly in the planning of the work with the team. Either as Product Manager or in the role of Product Owner.

It is therefore very important to carefully craft the messages that the Product Manager gives the team. It took me a while to understand this simple message, and this is my story.

As a Product Manager I was eager to give the team direction and clarity on the goals for the product. I communicated regularly with the team with more information on the market, the product, the customers and generic feedback on the direction they were taking.

Eager to achieve the goals we had set for ourselves the team was churning new features at a very good pace. Then the bomb hit. Every time we were supposed to go to production we faced major hurdles. The Retrospectives did not point to anything that could cause the quality problems we had, so I went on an investigation. Why was deployment to production so hard to this team that was delivering features at such a good pace (uncommon for other teams at that time)?

As a Product Manager I had to forget about what I wanted and had to concentrate on finding the root-cause for the problem. I interviewed the developers but nothing I found would explain the situation. The team was testing regularly in their environments and they were practicing Scrum "by the book".

It was then that it hit me. Having a conversation about the development process with one of the developers I understood that they were neglecting their unit and integration tests, which in turn led them to have a long feedback cycle for integration (some days only, but still too long).

After I heard that, it was fairly easy to trace the deployment problems to the lack of automated, fast-cycle integration testing. In their eagerness to deliver more features the developers would be developing up to the last day and would not have time to do the integration testing for those last changes. In turn, that led to many problems when it came to deploy.

Through that conversation I realized that the problem the team had was that they were being implicitly rewarded for delivering more features every time I praised them about the new feature they delivered. However, they were not being rewarded for building the unit and integration tests that would prevent the quality problems.

The end result was that quality-sustaining practices were being neglected. Having understood that, I changed my communication with the team. From that time on I started asking them if they had the integration and unit tests for every feature they delivered and started giving them praise for delivering tested features, not just any feature.

Before this happened to me I was not aware of how strong an influence the Product Manager's message can have on team behavior. "They are the engineers" - I thought - "they should know what they need to do". It's not that simple!

Photo credit: pcalcado @ Flickr