I am sure that most programmers are nowadays familiar with the term “agile software development”. If it happens that you have experienced some “agile” methodology in a few companies and made a few discussions about this “agile” idea with programmers from other companies, you may have noticed the huge gap that many times exist between the promised improvement and the reality. The following questions have troubled me a lot in the past. Where is this gap coming from ? Why “agile” seems to fail quite often ? Does “agile” equal to Scrum or Kanban ? Why people promoting “agile” are promising great results ? And why this damn thing doesn’t feel that amazing as we were told it would be ?
This widespread rumor of “agile” failing more often than not (sometimes the whole effort to apply an “agile” process was even clashing with common sense or professional experience), something not far from my own experience, made me lately to revisit the roots of the agile movement seeking for a reasonable justification that wouldn’t be based just on a few discussions. Reading the history of this movement and listening to what some of those 17 people (the ones who didn’t monetize the idea) that signed the Agile Manifesto have to say about this, was really enlightening. I tried to help the reader of this post by listing my resources at the end and pointing which parts of this post were being “verified” or “influenced” by which resources, or which quotes mentioned here come from which resource.
The motivation
In order to understand the essence of the agile s/w development movement, let’s have a quick look at the main issues companies, and their programmers, were facing with software development in the 90’s and were trying to deal with:
- shifting requirements: the client was changing his mind about the desired s/w product at a very advanced stage of the project. The change was happening either because the client was not sure from the beginning about the requirements and his expectations were far from what was delivered or because the market needs had considerably changed on the way. That was much more prominent in long-term projects, even when the client happened to be the same company that was developing the software. Failing to handle shifting requirements at the managerial level, often resulted in pushing the problem downwards, to the development team (irrational deadlines, death marches, etc). Cost overrun was also one of the consequences.
- inefficient development process: For example, too much time being wasted in building artifacts that were of less value for the client (e.g manuals) just because the development process dictated by the company required to do so. Or, same defects being fixed and re-introduced several times, on the way.
- bad software quality: Bad written pieces of code remained there and getting stale with time because programmers were afraid to make “unnecessary” changes in the fear of breaking something in the system. Increased pressure coming from irrational deadlines or death marches was also pushing developers to quick-and-dirty solutions.
Of course, there were more but these were some of the major ones.
The original idea
At the late 90’s, many attempts were made by programmers to improve the s/w development process and deal with the aforementioned issues: Extreme Programming (XP), Scrum, Crystal Clear, Rational Unified Process (RUP), the idea of the “Pragmatic Programmer”, etc. Many of the people participating in these attempts were gathered a few times in 2000 and 2001 and formed the idea of agile software development and a more concrete manifestation of this idea, namely the “Manifesto of agile software development”, which was encapsulating the idea in 4 “values”. The manifesto was signed by 17 of these people. I won’t go directly to these values, because without a solid comprehension of the idea, the values can be easily misunderstood.
So, what was the main idea? Is Agile S/W Development a recipe (a set of techniques, practices and tools) for dealing with the main problems of that time ? The answer is, no. The Agile movement, in its original expression, focused on the mindset required by s/w development teams in order to overcome these and any other issues. It didn’t provide solutions. Solutions were meant to emerge as a result of this mindset. [6][7][10] The message was that there are not “one size fits all” solutions and a solution can only be tailored to each company’s project and team needs and temperament. Agility translates to flexibility and adjustability. A flexibility that is driven by the same people involved in the development process and their freedom, ability and willingness to adapt the process on demand (versus an imposed company-wide bureaucratic s/w development process). To modify the process when and for as long as needed. These decisions should fall into the judgement of the development team, not managers. After all, the development teams know (and experiencing) their process and its weak points better than anyone else.[4] From another viewpoint, the intention was to free programming experts from company’s bureaucracy. Business would trust the developers to set their development process and developers would take business needs into account when forming the process. [5] The goal was to close the gap between business people and the technical people. A gap that was bringing a lot of pain to both sides. So, the outcome was supposed to be a win-win game for both sides.
Many attempts were made and the most successful of them led to methodologies that were proved very efficient for the teams that invented them. These methodologies were born from teams that were trying to work in an agile way. These methodologies were the snapshot of the process these teams had developed. And it worked exceptionally…for these teams and…their projects. That’s were the exceptional improvements promised by many people promoting “agile” are coming. From the improvements these teams managed to achieve at that turning point, when they decided to be flexible and adjust their development process based on their experience and their needs.
The “Agile” corruption
And then, business ideas were born and marketing followed (a.k.a opportunities for profit). The Agile Alliance was born in order to spread the idea and all these practices and techniques that have been proposed till then as the result of putting this idea into action. Conferences followed and, of course, the idea of the “Certified Scrum Master”. The idea for agility was commercialized but the developers didn’t follow the trend. The certification courses were mostly taken by project managers. And managers turned the whole idea into a new role. A managerial role. The role of Scrum Master. The original idea of the Scrum Master role was supposed to rotate through the various team members till the idea would fade and the role wouldn’t be needed anymore. [5] But project managers had a different approach. They wanted the role for themselves. So, eventually, everything boiled down to the hiring of the most suitable Scrum Master. [7]
Managers couldn’t foresee the result. Not only the original idea but also the techniques and practices that were there as the manifestation of the original idea (TDD, refactoring, pair programming,…) were lost. Even the concept of technical excellence was sidelined. And when you go fast and you try to finish more and more stories every sprint without paying attention to these practices, the code will rote. And it will be harder and harder to go fast because the code will be in a worse and worse shape. And you can’t have speed without s/w quality. You can’t have speed with technical debt pilling up. [5][13]
The “agile software development” was translated to following one of the advertised methodologies (e.g Scrum, Kanban, Lean, Crystal, ….). But in a very faulty way. Teams were not only adopting the whole methodology package as it was, but they were also being afraid to make changes. They were being afraid to deviate from the official guide of the methodology. The term “agile” was widely accepted but what most companies were doing (and still doing) was not agile s/w development. [4] Flexibility transformed to bureaucracy. Agility was lost on the way. And the gap between the business and developers opened once again, this time within the Agile community.
And it got even worse. Decisions were taken away from the developers. Managers were now deciding to apply a methodology (e.g Scrum) without first consulting with their developers. Without even explaining in advance what is that or asking how they feel about it. Even managers without technical background that couldn’t understand the development process as a whole. Software developers were left out. Their reaction materialized in a new movement. The software craftsmanship movement that focused on the technical side of the development process. [4][5] The result was painful for both sides. Development processes imposed by managers were failing and software techniques and quality improvements desired by developers could not be applied because they couldn’t fit them in the imposed development process.
Allow me here to put a little more emphasis on the corruption of the idea. Agile S/W Development requires expertise. “You can’t buy agility. It comes from practice. It comes from experience. And that’s not gonna happen by getting a consultant in, buying a book or going on a course or whatever…You have to develop that.” [9] Juniors cannot usually see the big picture of the development process and they rarely have as a measure of comparison for most of situations they will found themselves got into. Beginners always need to be told exactly what to do. And someone needs to figure out a solution when a problem comes up. Otherwise, they end up being stuck in the process. Sometimes. because they don’t feel confident enough to figure out what needs to be changed, or they cannot justify it or they even do not feel brave enough to make the changes. That’s what Dave Thomas and James Grenning will tell you.[2][7] And Andy Hunt will add: “Novices have to have rules to be effective… at the other end of the spectrum, the expert cannot be burdened with rules.” [3] And, honestly, this is not far from my experience, at all.
Any experienced programmer feels the needs to understands why he does what he does [10]. For example, let’s say that you are about to choose the length of your development iterations. Here is where, most probably, a scrum master would propose to you the 2 weeks sprint. But why ? The answer will probably be: because these are the scrum guidelines. This is what the Scrum community proposes. Is that enough for you ? For me, it means: someone have read this in a book or have been told this in a SCRUM seminar. Probably, according to some statistics, it is the most efficient sprint length…statistically. But statistics are statistics. Why not 3 days, or a 1 week or 3 weeks or a month ? An experience programmer would ask: what is driving the need behind iterative development ? Probably, the need for frequent integration and deployment. And why ? Because, on one hand, we need frequent feedback from client and, on the other hand, we need to avoid big diversion of the under development features that would make their integration very hard and would force us to rewrite a significant of them. How these 2 factors affect our need for iterative development depends on the individual project, team and nature of the software we develop. So, we should end up asking: how often we need (or we can) to ask feedback from our client ? How fast new features (that can be integrated independently) can be developed by our team ? Blind decisions can only lead to blind results.
The Values of the manifesto
Now, let’s focus on the 4 values of the manifesto. The manifesto meant to provide the intentions behind anything had been proposed or was going to be proposed as a potential solution. This was misinterpreted and many thought that each value or principle was expressing a very specific practice or technique. The values and principles of the manifesto are open to interpretation . And going one step further , we are even allowed to question them. [3] It is up to our judgement and expertise to evaluate whether and to what extend they serve our needs. It is right there in the manifesto[1] : “We are uncovering better ways…” . They don’t say “We hold the one and only truth”. Dave Thomas adds: “These values are not your values. They’re our values. They work for us. And it’s kind of up to you to work out what works for you.” [9] Of course, this is not an urge to question the value but a warning not to the restricted by the fact that this list of values exists. Anyway, before we are allowed to question them, we need to understand them.
“Working software over comprehensive documentation“. The first viewpoint for this comes from Andy Hunt: “Some people say that in Agile we don’t do documentation. It’s not true! There are plenty of occasions that you legitimately need to do documentation. You might need it for regulatory purposes, for training purposes, … That’s the kind of thing that we want to avoid were the only reason you are creating this artifact is to set it on a shelf somewhere because there is a corporate mandate to do so and there is no actual value derived from it….If it creates value, do it!” [3] Another viewpoint of this value is the avoidance of extensive software designs without having any tangible result. Designing and documenting the design in advance for a big software system down to details based on requirements that are going to change is futile.
The application of the “Individuals and interactions over processes and tools” value cannot start with the question “Which processes and tools shall we use ?” . What needs to be answered first is “How are we going to interact ? How are we going to work together ? ” [3] . Tools are not that important. The selection of people and the way they interact and communicate is the real deal. [4] For example, when and how a programmer should interrupt another programmer to ask him/her something ? How can I notify the others that I don’t wish to be interrupted right now ? How the product owner or the project manager should be updated about the team’s progress ? How long can I be stuck in a bug before asking for help ? How can I tell my colleague, in a nice way, that his code has several flaws during a code review ? Can people of the same team communicate their ideas to each other efficiently ? What is more, it is vital that the team chooses the process it thinks it will work better for it. That the only way the team can work as efficiently as possible. [4] Before the agile movement there was a mentality that processes were more important and that if you give a very efficient process to any group of people, things would work efficiently. The idea was now reversed. If you have any group of skilled developers that can communicate nicely, they can find their way through any process. Of course, as long as they have the flexibility to change what they thing is not working well for them. [11]
The “Customer collaboration over contract negotiation” value was trying to deal with the lack of flexibility of a long-term, fixed-scope contract. But it is not condemning this kind of contracts. This value is about building trust with client. If the client feels that he has a good insight on what your company is doing at every moment and feels that his money are being spend efficiently, any kind of contract would work. If both sides play honestly and the quality of work (defined at any way you want) is good, contracts don’t matter that much. Contracts can be modified easily if both sides are willing to. Of course, it is implied that contracts that leave more space for changes (in the time of delivery, in the cost,….) is something that make happy both sides. Because it serves better the needs of both sides. But again, the value is not that much about favoring one type of contract but forging a relationship with the client. Having delivered some s/w according to the contract does not guarantee a satisfied client. [9]
Just for the sake of clarity, let’s add some more context to contracting topic. In most software projects, requirements tend to change as the progress goes on. S/W development companies usually deal with it by adding some (or a lot of) buffer to their offer. Freelancers do the same. But in order to make a competitive offer, this buffer needs to be kept within some limits. However, it cannot be guaranteed that the scope of late changes will stay within these limits. And some unpleasant situations may occur while the client is trying to figure out if and why the contract covers these changes. On the other hand, if the client accepts a large buffer, it may end up paying much more than needed. Most companies and developers go with the second way (large buffer) because not only exact estimations are difficult but also “it is so hard to get a decent set of requirements” from the start. [8] And most of us know that extra cost cannot be effectively pushed down to the one who is building the software. You will end up with heated discussions, lower s/w quality, etc. Of course, we should make clear that there are indeed cases where requirements are well defined and stable from the beginning. In such cases, even everyone’s precious iteration is not a requirement to be agile. Iterations is a just a tool, not a value. [9] Nevertheless, the danger here is that for long-term, fixed-cost and fixed-scope projects the client cannot foresee the upcoming changes of the requirements and he ends up paying for features that, at the end, he wouldn’t like to pay. Because the developers have spent a lot of time for them. Using smaller time frames with smaller scopes, even with fixed price, helps deal with this issue in two ways. Firstly, if the client has to decide every 2 weeks or 1 month what is the next thing he is going to pay for, he will be more diligent in his decisions and the formation of the requirements. And secondly, if client needs to change some requirements at some point, he can always re-prioritize features he has already asked , always focusing at the next more valuable thing to buy based on the current viewpoint. [9] That way, at the end, he will be able if he likes, to leave the least important ones out, based on his budget.
“Responding to change over following a plan“. The same problems, more or less, led to this value. Coming up with a detailed architectural design of a big system in advance and time-planning its implementation was often leading to an illusion that this plan will be followed to the letter. An illusion because, as we said, getting in advance a decent set of requirements that wouldn’t significantly change over the period of implementation happens rarely. But does being agile means that there should be no plan ? No. Having a plan is useful. Mostly due to the process that generates the plan. Because the process makes you understand some things about the evolution, the size, the difficulty and the structure of the project. The plan is just a guideline that needs to be dynamically evolved alongside the project. And so, the previous value is tied to this one by urging for a flexible charging scheme. But do not fault. Being able to respond to change is not only a matter of contracts and charging schemes. It also has a technical aspect. How extensible is your architecture ? How much your developers are afraid to make changes because your software is not sufficiently covered by tests ? And there is more. For example, a “scalability requirement that can only be addressed through in-depth design that will not be visible in early demonstrations. To focus on the visible at the expense of the essential is irresponsible.” [12] So, is there enough s/w quality that will allow you to welcome changes ?
The “agile” methodologies
All methodologies (Scrum, Kanban, Lean, Crystal,…) that emerged as an answer to the concerns the manifesto brought up, were just proposals by different people, or groups of people, tailored to the needs of these people at a specific period of time. Identifying a specific methodology with the notion of Agile S/W Development is a mistake. They were methodologies developed until an agile mindset. No-one says that any of this methodology would fit better in your case and context than the one that you are already following. Scrum is not “agile”. Is a technique that you can use in an agile way, or not. Scrum is a tool. A good tool. But the mindset under which you are using Scrum is what matters. Nor methodologies, not even specific practices (e.g pair programming, unit testing, …) were the subject of the discussions that led to the manifesto. Practices, techniques and methodologies cannot be stripped out of the context. And that’s why selecting one practice or methodology without knowing the context (the company characteristics, the team composure, the domain where the s/w belongs to, the nature of the project that the methodology will be applied), will probably lead to a troublesome development process, if not to failure. That’s why Andy, Dave and Jon [2][3][10] point out that it is wrong to say “we are doing agile”. You can only say “we are being agile”. Being flexible, being adaptable. Being able to change the process and bend the rules when you thing that it would benefit the team and, of course, the company.
[1]. The “Manifesto for Agile Software Development”.
[2]. “Agile is Dead”, GOTO Conference 2015, Dave Thomas
[3]. “Uncomfortable with agility: What has ten+ years got us ?”, COHAA 2013, Andy Hunt
[4]. “Agile in 2018” , Agile Australia 2018, Martin Fowler
[5]. “The Land that Scrum Forgot”, NDC 2011 , Robert C. Martin
[6]. “XP Turns 20 and What Have We Learned?” , Agile2016 in Atlanta, GA , Ron Jeffries and Chet Hendrickson
[7]. “Technical Excellence”, Agile on the Beach 2017, James Grenning
[8]. “ScopeLimbering”, https://martinfowler.com/bliki/ScopeLimbering.html , 2004, Martin Fowler
[9]. “A Retake on the Agile Manifesto” , GOTO 2014 , Jez Humble, Dave Thomas, Tatiana Badiceanu, Martin Fowler, Katherine Kirk
[10]. “Agile, A Perfect Solution to Your Development Needs”, https://www.youtube.com/watch?v=RqtUJQz_n9o , Jon Kern
[11]. “To Agility, and Beyond…” , Startup Lessons Learned Conference 2010, Kent Beck
[12]. “Agile! The Good, the Hype and the Ugly”, Springer, 2014, Bertrand Mayer
[13]. “FlaccidScrum”, https://martinfowler.com/bliki/FlaccidScrum.html , Martin Fowler