Tag Archives: scrum

Standard

Posted by

Posted on

January 19, 2014

Posted under

Methodologies

Comments

7 Comments

Agile is evil

Summary

Real issues encountered in real development organizations while using Agile and Scrum as a software development methodology: no up-front design, no (consistent) user interface design, developers adding too many bells and whistles, no generic infrastructure development, features that are too large to be meaningfully developed in one Sprint, issues with teams located in different physical locations, and poor code quality. The post asks whether Agile is engineering, and promises a follow up post presenting an alternative.

Qualifier

For a very long time I hesitated about publishing this post. It expresses some views which are diametrically opposed to currently popular mainstream ideas about software development. However, I strongly believe that the issues discussed in this post are universal and deserve wide exposure and discussion. I welcome any and all comments, especially if you disagree!

Introduction

In the world of economies of scale, cost saving measures are warmly welcomed as the next savior of the profit margin and sometimes even of the corporation. Unfortunately, cost saving measures often come at the expense of quality. We are all experiencing the same trend: 30 years ago, washing machines would last 20 years; today, we’re happy if that shiny new machine lasts for 5 years. And of course, this is true not only for washing machines: the same exact trend is occurring with televisions, stereo systems, cars, fridges, cameras, you name it. Although generally, products are definitely cheaper now than they were in the past (when correcting for currency  devaluation), they now last only a quarter of the time. Long term products have actually become more expensive to own. And this is without taking into account the consequences for the environment: just imagine the landfills full of discarded appliances growing at an ever increasing rate.

At the same time, products acquire more and more features, options and bells and whistles. It seems as if most product designers feel that more features, buttons and options is always better. 20 years ago, most average people knew how to connect and tune a television; today, only geeks know how to configure today’s screens, and even switching from television to DVD may be a challenge for some people.

Software development has not escaped this trend. Software companies find themselves under extreme pressure to deliver more products, more features, faster than ever before, and provide updates and upgrades with increasing frequency. In many cases, the quality of each subsequent release deteriorates at an alarming rate. This trend has had big consequences for the way software is being designed and developed (the “software development life cycle” or SDLC).

In the past, the software development life cycle usually consisted of several phases; moving on from one phase to the next required achievement of a milestone or checkpoint. The various phases were usually variations of the following: Requirements –> Initial Analysis –> Feasibility –> Detailed Analysis –> Functional Design –> Technical Design –> Development –> Acceptance and Testing –> Release –> Maintenance and Support. This type of software development life cycle is usually referred to as “the waterfall model” because of the way it moves irrevocably from one phase to the next.

There were several serious issues with the waterfall model: it was heavy handed; it took a long time until an idea made it all the way into a release; and there was no feedback from a later phase to earlier phases, although later improvements suggested allowing information to flow back from a later phase to earlier phases, this rarely did more than produce piles of paper records. As software became more ubiquitous and the delivery model changed from local installations to online delivery to cloud based usage, a different software development life cycle gained in popularity. Enter Agile.

A very brief overview of Agile

There are actually multiple flavors of Agile software development:

Some flavors focus more on the practices while others focus more on the project management aspects. However, what all Agile methodologies have in common is a focus on adaptability:Fluxx Goals planning is loose; milestones may be reached in multiple ways and milestones can change mid-way. This allows teams to adapt to changing realities very quickly.

Specifically, Scrum calls for the establishment of self-organizing teams, in which all members should be physically co-located. As a methodology, Scrum focuses on responsiveness and fast delivery of software of incrementally increasing complexity. Communication among team members is almost entirely orally. Each team has a Scrum Master, who is responsible for removing impediments that prevent the team from reaching its goals, and ensures that the Scrum process is used properly. There are no team leads or project managers.

In a typical software development organization, there are multiple Scrum teams, each of which may be physically located in a different geographical location. Each Scrum team should have all of the abilities needed to analyze, design, develop, test and document software. One or more Product Owners write User Stories describing usage scenarios and these are inserted into the Product Backlog sorted by priority. Software development takes place during Sprints. A Sprint is a fixed period of time (typically from one to four weeks). At the beginning of a Sprint, each Scrum team takes the top one or more User Stories off the backlog and then holds a planning meeting to identify the tasks for the Sprint. The team then works on implementing the feature(s) necessary to implement the User Story/Stories. During the Sprint, each team holds a 15 minute Daily Scrum meeting, during which progress since yesterday and plans for today are discussed.  To coordinate amongst multiple teams, there is a Scrum of Scrums: a daily meeting of representatives from the various teams, to discuss issues of overlap and integration. At the end of the Sprint, each team holds a Sprint Review during which the implemented feature(s) are demonstrated to stakeholders and a retrospective meeting is held to review progress and lessons learned. Scrum holds that by the end of each Sprint, the product must be fully working, tested and be shippable, including end-user documentation.

Issues with Agile

Before I deep-dive into the various issues with Agile, I want to stress that the below is not a list of theoretical issues; rather, all of the issues described below were encountered during real software development of large systems by seasoned development organizations belonging to large, multinational corporations, known worldwide, that were generally recognized for their excellence in implementing Agile and Scrum. While it is possible that these organizations had a fundamental misunderstanding of how to implement Agile and Scrum, it is not very likely.

No detailed design up front

The above description makes it clear that in Scrum, there is no room for detailed analysis and (functional as well as technical) design. In fact, the flexibility of the process is largely due to the lack of up-front analysis and design. Some development organizations make up for this by the notion of a “zero Sprint”: one or more Sprints that precede the Sprints during which actual development work takes place. However, generally this does not make up for the vast lack of knowledge about the problem to be tackled due to the lack of analysis and design.

No consistent user interface design

The Scrum teams are supposed to be self-organizing and self-sufficient. Every Scrum team does their own analysis, design, development and testing of the feature they are currently working on. What does this mean for user interface design? While most development organizations do have a style guide which is sometimes quite detailed, there is a lot more to creating a consistent “look and feel” than can be expressed in a style guide. The result is that similar functionality in different features will look and feel different. This is perhaps the one telltale sign of Agile development that is most exposed to end users: the user interface has a typical dissociative feel, as if different parts of the product were developed by separate companies. And in a sense, they were!

Providing too many options

Somewhat related to the previous issue is the issue of user interface design by software engineers. User interface design is a separate profession. It is much closer to industrial design than to software engineering. The one skill that is most frequently lacking in most Scrum teams is user interface design. As a result, software engineers are undertaking user interface design. Software engineers, dare I say it, are geeks, who love additional options. Whenever in doubt, the answer of a typical software engineer is “make it optional” or some variation of this. This results in a plethora of options, a true forest through which the poor end user doesn’t see the trees. Read the book “The Inmates are running the Asylum: Why High Tech Products Drive Us Crazy and How to Restore the Sanity” by Alan Cooper.

No generic infrastructure development

Since development takes place based on User Stories which describe usage scenarios, the infrastructure needed to implement a User Story is generally recognized only during the Sprint. At that point, instead of generalizing the infrastructure needs, only the specific infrastructure to support that particular User Story is developed. The needs of other User Stories which have not yet been taken off the backlog cannot be taken into account. Later, when another User Story comes along that requires similar but not identical infrastructure, either new infrastructure is developed separately or the existing infrastructure is changed to support the new needs. Note that in large development organizations it is quite likely that the development of the new infrastructure or the changes to the existing infrastructure will be undertaken by a different Scrum team. Both of these possibilities result in sub-optimal outcomes: in the first case, infrastructure is implemented in separate ways, in separate classes, and the various pieces of infrastructure that belong together become fractured and distributed in various places in the software, resulting in a maintenance nightmare. In the second case, changes to the existing infrastructure break existing functionality, which then needs to be re-factored.

It is possible to alleviate this issue somewhat by artificially creating dummy “User Stories” representing infrastructure needs; however, to do this properly would require up-front analysis of all User Stories in the backlog to understand all of the upcoming infrastructure needs, and this is quite obviously against the principles underpinning Agile development. Even when the infrastructure development is allocated to one specific infrastructure team, due to the lack of up-front analysis and design it is virtually impossible for this team to foresee all or almost all future infrastructure needs, and therefore infrastructure will necessarily need to be developed piecemeal.

The infrastructure changes experienced by some Scrum teams may be so frequent and so drastic that the teams feel they are busy mainly with keeping up with these changes, both by familiarizing themselves with the often far from trivial changes, and by adapting their existing, already implemented features to these changes. This of course has a negative effect on motivation.

Features too large to be implemented in a Sprint

Since User Stories are not analyzed or designed prior to the start of the Sprint in which a particular Scrum team took that User Story from the backlog, it frequently happens that during the Sprint it is discovered that the feature described by the User Story is simply too large to be developed in one Sprint. Scrum has a solution for this situation: divide the User Story into multiple smaller User Stories. Develop one User Story and return the others to the backlog (this may have to be done via the Product Owner). However, in practice it often is nearly impossible to break down a User Story in smaller parts which still provide meaningful end-user functionality. Remember that according to Scrum, the end of the Sprint must result in a finished product that is shippable. Implementing functionality that is not meaningful to the end user goes against this adagio.

Poor cooperation among teams in different physical locations

An entire list of issues ensues when different Scrum teams in the same development organization are located in different physical locations. Since Agile opposes written documentation (such as PRDs, design documents, etc.) and virtually all communication is oral during daily Scrum or Scrum of Scrums meetings, when teams are in different physical locations this becomes more difficult. When all team are located in similar time zones, modern telecommunications equipment and software such as teleconferencing, videoconferencing and Skype (to name just a few) can help. But what if the teams are located in vastly different time zones? Imagine the mayhem if you have two Scrum teams in India; five Scrum teams across Europe; six Scrum teams on the US East Coast and four Scrum teams on the US West Coast. All of them work on the same largish software project. There is no written documentation other than User Stories and the source code. How are these teams going to coordinate anything orally?

Poor code  quality

When I studied Software Engineering (admittedly a long time ago), one of my professors coined the following statement: “the first rule of optimization is: don’t do it”. What he meant to express with this statement is that it is almost always better to leave working code alone, instead of optimizing it. The more existing code is reworked, the less maintainable it becomes. If you feel you must optimize a piece of code, the proper decision is to throw the code away and write it again, having learned how not to implement it.

We all know what happens with “old” software: after it has gone through several major releases, the code becomes less and less maintainable, until eventually the need is felt for a “major rewrite”. Well written software that has been developed using more traditional methods in which ample time was given for up-front functional and technical design, typically reaches this phase after 6-10 major releases. However, due to the intensive re-factoring built into Agile, it is my claim that software, even well written software, developed using Agile reaches this phase much earlier; in some cases even 2 major releases is too much to expect!

Agile <> Engineering

What is the definition of “Engineering”?

The American Engineers’ Council for Professional Development (ECPD, the predecessor of ABET) has defined “engineering” as:

The creative application of scientific principles to design or develop structures, machines, apparatus, or manufacturing processes, or works utilizing them singly or in combination; or to construct or operate the same with full cognizance of their design; or to forecast their behavior under specific operating conditions; all as respects an intended function, economics of operation or safety to life and property.

In other words: engineering requires planning and design; an essential part of engineering is the ability to foresee the behavior of the engineered system.

Given the above definition, it can be argued that Agile software development is not engineering in the proper sense. It is much closer to “tinkering”: working with something in an experimental manner.

I submit that all of us software engineers have to make a decision for ourselves: do we want to be considered engineers, and be compared to other professional engineers such as construction engineers, aviation engineers, shipbuilding engineers, etc.? Then we should engineer our systems using the same basic principles that underlie all engineering, and that includes planning and design.

Let’s ask ourselves a different question: if we knew a particular bridge had been built using Agile building processes, would we dare to drive over it? I wouldn’t. And if you say that software is different, that by virtue of software being so easily changed it makes sense to make use of that property to the fullest by utilizing a more flexible methodology, I would counter by saying that software is indeed so easily changed that we need extra protection: in fact I would argue that the process and design principles for building a bridge are much better understood and appreciated than the process and design principles for a software development project, and therefore the software development project requires a stricter and more predictable methodology than building a bridge.

Agile ruins discipline

Yes, I will admit it: developing software using Agile can be fun for developers. We get a lot more freedom to design and code each feature. Instead of having to follow a functional and a technical design, not to speak of implementing a user interface designed by some industrial designer who hasn’t a clue about computers, we get to do our own design and implementations. And we can do it differently every Sprint!

In fact, Agile can be so addictive that developers who have tasted Agile development may never want to go back to the discipline required by more traditional software development methodologies.

So how come it works?

By now some of you may be asking “if it is so bad, how come more and more organizations implement it, use it and it seems to work well for them?”

It may surprise you, having read this far, to now read that I do think Agile software development methodologies have their uses. In particular, they are well suited for one-off software development, i.e. custom development, as well as for maintenance of existing code. Once you have a working software product (properly designed and developed using more traditional software development methodologies), Agile can be applied without experiencing most of the aforementioned issues. It is mainly when large scale software development needs to take place from scratch, that the above issues cause severe problems and even cause the entire effort to fail.

However, at this point in time, it has become almost impossible to oppose Agile. It has reached the cult status of “silver bullet” and everybody and their uncle (or aunt) wants to be part of it. It is very much like “The Emperor’s New Clothes”: no one dares to say anything for fear of losing face, or worse, their job! But many of us are waiting for the little kid to say “the Emperor is naked!”.

In addition to this: since, as stated above, Agile development is “addictive”, many professionals in the software industry are simply unwilling to admit to its faults, for fear of having to go back to more traditional software development methodologies.

What is the alternative?

If you have come this far in this, my first blog posting, you definitely deserve a medal for perseverance! Jokes aside, this is a very serious question, which deserves a very serious answer that I intend to give in my next blog post.