Skip to Content

Agile Principles Work, Until They Don't

Since 2001, the principles of the Agile Manifesto has spread from tiny development houses to companies with thousands of developers.

Many companies have benefited by following them while many failed while trying to implement. The common criticism for those who have failed is almost always like “There is nothing wrong in the manifesto, you just do it wrong”

Is that really the cause for all the failed ones ?

Could it be simply:

The agile manifesto works only for the companies whose business model already naturally fits

May be strict compliance to the manifesto caused some of the companies get in to trouble ?

Let’s see how those principles can bring you completely down, if you turn your back to realities and try to follow the principles in a dogmatic, isolated way.


I had been trained with classic Waterfall when I was in university in the late 90’s. It was a model almost carved in to stone. You had to follow that model if you want to make good software product. That model dictated what to do, and how to do. Reasons were never visited.

During my junior years, infamous thought leaders of our industry unveiled the Agile Manifesto. It was like a gift from heavens. It let many early adopters to create higher quality software in shorter time frames. That was not a coincidence.

Majority of early adopters were the ones that are already flexible in their work flow and business.

Initial success of those adopters became example for not-so-flexible environments.

The manifesto has 12 principles. None of them is “completely” wrong. But a lot of companies try to follow principles blindly, without assessing their business, their value flow and their customers. Even some of them become victims of certain type of “agile” consultants who care only about short term gains while inadvertently undermining the business in the long term.

No consultant can save you if you simply seek recipes to follow

Some of the principles have roots from Toyota Production System They are the ones that are generally applicable because they focus on what rather than how, which might contradict with some business model. Those are more like a cultural values.

However, some of the principles have hidden requirements in the business model and customers and they break when the reality doesn’t match.

Toyota factories, which are famous for their efficiency, JIT and sustained successful execution of Kaizen, don’t necessarily have identical assembly lines in all factories. Each factory learns from each other, but decide and develop its own methods in a data/measurement driven manner, even though they produce sometimes precisely the same products.

Automobile manufacturing is among the most precisely defined and complex flows in the world, yet Toyota avoids direct copying of methods and tools even between teams having same job in different locations.

Software industry on the other hand, is very different. Some people sell products, some services. Some develop firmware for highly critical devices while some create highly complex trading software. The business cases, requirements and customer types are endless and unique. Everybody has a target customer type and works hard to keep them happy so they keep paying, letting companies to pay their bills.

Yet, certain parts of the manifesto tries to put everyone in to the same mold, as if everybody face the same situation.

I said customers are unique, but I believe what they want can be summarized with this:

Newsflash: Customers want to solve their problems in order to increase their profits

It almost always drills down to profit concerns. It can be software that brings the costs down, or increase sales, or proactively detect and solve problems. Whatever you sell, your customers buy it because they believe they will make even more profit using that.

The key point of keeping customers happy in long term lies in truly understanding and positioning for them while solving their problems. Unfortunately, the manifesto cares only a certain type of customer.

The type that expects frequent, working deliveries;

The type that wants to keep in touch with anybody in your team continously, and have time for that;

The type that is geographically close to you;

The type that has a business and work environment that can naturally cooperate with a company that strictly follows agile principles;

The manifesto assumes that your customer is also agile

What if your customer is not like that ? Are you going to reject working with them because they don’t cooperate well with the agile principles ?

I hear some of you saying “Educate the customer! Transform all the industry!”

Accommodation booking companies did not try to lecture hotels about merits of the internet when they know some of them have barely working phone line, located in the middle of nowhere without even GSM coverage. They used other methods to on-board them, until the technology reaches out there.

Expecting your customer to automatically pull and deploy your new shiny release works only if your customer can really do that. This is not even always a problem of “ability”. Sometimes this becomes impossible due to regulations and natural complexity of the business domain.

In certain industries, deployment environments must be completely disconnected even from their own corporate network, due to security reasons. How do they manage their own stuff then ?

They go in to physically secured rooms that contain secured systems that can access the infrastructure. No laptop, no vpn. You have to be physically in the room.

Can you transform them ? They are probably already aware of the alternatives. May be they are simply not allowed to do it.

Don’t be a burden on your customers. Solve their problems without putting extra work on them.

Principles emphasize satisfying the customer by early, continuous, frequent deliveries. It assumes customers are satisfied this way. This works more or less well if your software is consumed as a service or if your business allows you to roll out silent updates. It rarely works if you sell your software as product.

Why ?

Whoever owns the operation of the software, also owns when/how it is delivered along with all the risks of doing it.

When you own the life-cycle, as in service, you decide the frequency of delivery, validation of the solution. You have full responsibility and it is up to you to make decisions those areas.

When your customers own the life-cycle, they make decisions based on their best interest. Their best interest doesn’t always lie in weekly deliveries or continuous deployment.

Respect how your customer works in the first place

You want examples ? OK.

Imagine that your software is just one piece in a huge business where tens, sometimes hundreds of companies bring their own software. Yours must always work flawlessly together with them.

In those cases, early and continuous delivery to the customer is rarely possible. You may claim that integration must always be based on API contracts, and they actually are. However in reality, even small changes can have unforeseen affects on other parts of the integration. When your software affects lives of people (in real manner, not like being unable to access your duckface selfies), when penalties that cost six zeroes fly around, when you are not allowed to break, everything has to come together and get verified in a methodical approach.

In some cases, you do not even have access to the other software that you’re going to integrate. You may not even know the vendor. You know which protocols you should be supporting but you don’t know if the other party implemented it in a way that it is defined in specs or RFCs (And RFC allow a lot of space to vendors, enough to create behavioral differences)

Imagine there are other companies in this situation. Your customer executes end-to-end solution verification by taking all the software in solution and putting them together. Once all business cases are verified (you may know only a few of them), the solution is considered to be ready to become production. After that, the new cycle of solution verification begins with newer versions.

Now let’s scale out the idea to all partners. Imagine all partners frequently pushing their updates. Would that make the life of the customer better ?

Good approaches keep working when they are scaled out

Obviously, this is can be manageable if you have absolute control on all components.

Another aspect of this complexity comes from organizational coordination. Your customer may need to align with the corporate division who maintains databases for the update you need. If you are delivering a new feature to them, may be they have to coordinate with the marketing team to align release with the advertisements that can be both online and on paper. May be they have to print the dates on leaflets, newspaper, or even TV ads before you give them new features. May be they have to make coordinated changes to their billing system that will support your new features.

Now, does it really help your customers if you deliver them new version every week when it takes 6 months for them to verify and prepare for the full solution together with the other software ?

It will only annoy them.

Let me extend situation. You have many of those customers and imagine each of them use different version because you release more frequently than the business requires. You will end up maintaining much more versions than you should be, which is a huge cost.

You should align your public release frequency to the speed of your customers and your industry.

If I speak in terms of Kanban, treat your customer as a next phase in your flow and let them pull new software when they need it. Don’t push them something that they can not process or they don’t want. Equally importantly, don’t disappoint them by being unable to deliver a new version when they need it.

Even though this would have been considered anti-agile by many, consistently happy and paying customer will have more effect on your business lifespan.

Another important principle is the ability to welcome changing requirements, even in late development.

This is generally a good advice since it allows adding value to the customer and increase satisfaction. Indeed requirements frequently change in the software world. Old waterfall minded people usually complain about the requirement gathering phase not being done correctly. Even if you do it perfectly, something will need to change.

There are four important points regarding the changing requirements, which often make them very difficult to do.

  1. Not all requirements come from the customer

    There are many cases where requirements are added/changed not by the customer itself but by the organization you live in, or standards/specs that you are obliged to comply.

    It is usually not possible to predict those and it can really harm the roadmap depending on the complexity of the requirement.

    For example, your company may suddenly sign a very advantageous contract with a vendor for a technology that you already use from an alternative source, and you might find yourself obliged to migrate your solution based on the contract in very short time frame. Or, the regulation that you are obliged to comply may change and the new rules might require serious modifications right before your release date.

  2. Not all changing requirements are easy

    There will be cases where you get a late requirement from a customer which requires substantial changes no matter how prepared you are. It might be due to business changes, partnership changes, integration changes, financial issues or many other reasons which are totally out of not just your control but also your knowledge.

  3. Some late changes might require additional changes in some other software that you don’t even own

    This might sound crazy but that can also happen. Such cases will create snowball effect on other people’s plans about their software. If it isn’t valuable for them to make changes in time you need, you are going to have a problem.

  4. Some late changes from a customer may contradict with expectations of another customer

    Customers might have different views and requests on how your software should be solving their problems. In some cases those can contradict. Consensus can be built when you have enough time to visit customers to find a way that satisfies everybody. However a late requirement that creates contradiction isn’t just a technical, but also a business problem to solve.

Not all requirements are equal. Some have higher chance to get a change request. Identify and focus on them.

Principles also ask you to have daily contact with business people and maintain face to face communication. Those are good advice to be followed when possible.

Your development team, business people and customers may be spread even over continents. It means that you will live in different time zones. Even worse, some of your colleagues may start working when you are already out of business hours. The only option left is email in such cases. Face-to-face does not mean Skype. The actual value of face-to-face communications comes only when you sit together in the same room.

Business people may be difficult to reach because they may have busy schedule on visiting customers, meeting with other business people to compare requirements that they gather from different customers, discussing long and boring (for you) contracts.

In some cases, you will not even have a business people to talk to. What if you are supposed to make a software based on a book of requirements and specs ? You will have nobody to speak or make interim demos and this is not your fault. It is simply how some businesses work.

Communicate with your colleagues face-to-face when you can. If you can’t, make communication as effective and short as possible.

The manifesto underlines the importance of having motivated, quality individuals.

The reality is, you can have a good team only if you have budget, if you are close to such people and if you can attract them.

The budget is tight for the majority of companies. By tight I mean many companies try to survive the calendar year, if not month.

Good engineers that combine technical and social skills are rare, like good doctors or good carpenters.

Even if you have budget, it is not easy to find and hire people. You will still get rejected by various reasons even if you offer a very good contract.

Teams should learn how to succeed using the resources that they currently have. It is highly likely that you are one of them.

Remember, when you hire a good engineer, you strip another company from that chance.

The manifesto unfortunately does not say anything about succeeding with a team that consists of normal people instead of extraordinary engineers. This is the reality for most companies and you have to make your way even under this condition.

The principles also mention motivated people. But who is a motivated person ? Is it the person who show deep passion on something ? Does having a deep passion guarantee any adequacy ?

Should I be accepted to work on microprocessors just because I am very motivated even though I am like a potato when it comes to hardware design ?

Having potential or capability to do something is more important. Then, among people who have those, we can favor the ones who have passion.

Try to hire good engineers. But try harder to increase productivity and efficiency of your current team because that is how you will survive. Study literature to find out respectful ways you can achieve this.

Technical excellence and good design is one of the remarks in the manifesto.

But, engineering does not target technical excellence just for the sake of it.

Engineering targets satisfying the customer requirements in a way that the total cost of reaching the solution, without compromising quality, is low enough to make a profit that will let you keep your business going.

After all, this is the ultimate goal of the software production when it is done as business. Making profit out of it.

Do not think that quality implies technical excellence. A solution might have high quality without being technically excellent.

There will be some cases where reaching a good design and excellence is so expensive that you will not be able to make a profit to pay your company expenses. If you can not survive, the excellence becomes an insignificant detail.

Those early perfectionism will already contradict with KISS and YAGNI

For example, if you are supposed to deliver a solution that targets 100 req/s, do not immediately dive in to complex architectures that will let you serve 1000000 req/s. Make your solution easy to refactor (TDD) so you can iteratively improve as sales progress.

Focus on staying on the profitable side. When you are profitable, you will have the opportunity to make investments to improve your product quality and work efficiency. In return, this will help you staying profitable.

Then there comes the emphasis on simplicity. Indeed the simplicity is one of the pillars of the engineering. It makes your product efficient, less costly, easier to maintain, resilient, and sometimes flexible. It allows you to stay on the profitable side, which is the most important gain. It is the most difficult point to reach.

However, sometimes complexity is just part of the business. No matter what you do to keep your solution simple, it can end up being complex due to the requirements that you must satisfy. Actually pretty much everything that makes the world run, falls in to this category. It is important to recognize such situations and avoid simplifying more than necessary.

Oversimplification is as big threat as overcomplication

Or, as in the quote attributed to Albert Einstein:

Everything should be made as simple as possible, but not simpler

Once you fall on the simpler side, you become inadequate, which ultimately dooms.

How can you decide if you are simplifying more than necessary ?

  • Do you break any contract ?
  • Do you bring any risk ?
  • Do you degrade the reliability, security and performance in any way ?
  • Do you make your customer’s life more difficult ?
  • Do you, by any chance, cause an increase in fixed or variable costs ?

Conclusion

Following the principles of the agile manifesto in a dogmatic way, can fail you because it tries to control not just what you are doing but also how you are doing stuff.

Since it is not guaranteed to fit every single business environment, you should learn from it but keep the control at your hands in order to build your own way.

Learn, explore, try, measure but never ever copy

That is simply because running a profitable, quality software business in a sustained way is not something you can achieve by iterating recipes from a cookbook.

It is possible to be an “agile” company or team (in the sense that agilists mean) without adhering to all points of the agile manifesto.

You need to grasp the agility not in the sense of agile software development but in the greater sense of business agility, which will also have positive impact on software development.

Instead of focusing on execution details that are far from being universal, may be you should check Lean Software Development that encourages you to think about few very high level focus points without mandating specific methods on how to do it.


Reading