The software industry as a whole is awaking to a new wave of "process improvement". It's important to look back in history and understand the previous waves in our industry before we can understand the relevance of the latest, Agile, process improvement wave.
Many years ago in 1970, Winston Royce wrote an article where he described (and warned about) the use of the one-pass waterfall ("waterfall" for short). In that article Royce states that if the waterfall was to be used, it should be a "multi-pass" waterfall with many feedback cycles between the different phases, where when you pass through a phase (be it requirements, design or any of the other 7 phases Royce defined in his paper) you learn more about the previous phase and should include that feedback by re-iterating the previous phase.
Later, in 1988, Barry Bohem, also known for his work in the domain of Risk Management came up with the Spiral model of development. This process improvement wave introduced the concept that you do a little bit of everything as you go (e.g. early prototyping), and you repeat often. There was little talk of releasing working software often and interacting with customers, but this was a big push towards the "incremental" process model.
When RUP was heavily pushed by Rational around 1998-2000, their original idea was that the process would lead to incremental and iterative development. But they made a blunder of monumental proportions when they put their process picture in place. It was waterfall all over again -- even if they did not intend it. RUP in effect set the software industry back in time due to it's unfortunate success (and still does in many places).
Of course, there were other waves, I'm sure you will remind me of those in the comments below, but the gist of the argument that I'm trying to lay out here is that there were several waves of process improvement, some less successful (like the spiral model), some more successful (like RUP and Waterfall) but there were no giant leaps in productivity or quality improvements in the software industry during or after those waves. So, the question is -- when one looks at history -- is Agile another wave? And what are the consequences of the answer to that question?
Well, the unpopular answer is: yes. Agile is just another wave. On the other hand, by recognizing that fact we can make the best out of it. We can learn from those that are successful with Agile processes and practices, we can learn from those that are not. We can learn from the importance of Values that Agile emphasizes (unlike other process improvement waves before). We can learn from practitioners who are the ones really advocating Agile. Agile software development represents without a doubt a step forward for our industry, but why? What can we learn from it? Since, Agile does not, no matter how we try, constitute the "silver bullet" Fred Brooks warned about in his Mythical man-month.
The good thing is that we can learn immensely by adopting Agile and using Agile methods. Where Agile methods excel (when compared to previous methods) is in recognizing that whatever we do in the software industry, we do through processes, but these processes depend heavily on people, therefore we need to inspect and adapt constantly to ensure we deliver the best value to our customers despite the quality of the process under use.
That's why retrospectives or reflection workshops are important (like in Scrum and Crystal). In effect the most successful practices from the Agile community (in my experience) are the ones that use the power of reflection -- or "thinking about how we do what we do" -- these include retrospectives, pair programming, TDD, and others. This is perhaps the single most important contribution of the Agile movement to the industry.
However, the benefits don't stop there. We currently have collections of different agile ways of developing software (like it is expected in a "process improvement wave"), now it is time to give room to the scientists and understand "why" those different ways are better than the old ways. This deep understanding of what works, what does not work and most importantly "why?", is the only way to really push our industry significantly further and make sure we don't fall back to our old habits and unproductive processes of the past.
Software industry scientists should also learn from Agile, they should follow the practitioners and shed light onto the practices and processes. The growth of the software industry depends on it. If this essential scientific grunt work is not done we will be here again in 10 years talking about another process improvement wave and why Agile processes do not work...