Find articles from my Blog Archive:

Thursday, 17 January 2013

Simple is Complex

I'm continually bemused by the extraordinary complexity that can be found in virutally all large organisation's IT systems. Sometimes these organisations embark on "Simplicity" initiatives that are often abandoned after much effort because they've had such a limited impact. I started to wonder why this was, and why large organisations seem to find Simple so Complex. Having thought a little, these are my conclusions of the underlying causes of complexity. Understanding the true causes might help in formulating solutions, so I hope this is of some help.

Gaining agreement to a design is an act of compromise

In order to progress a new design its always necessary in a large organisation to gain agreement from multiple people, personalities and departments. Often those parties will have their own “pet” topics or items they wish to see included in the design. As a result, its very often the case that by the time a consensus has been achieved that the design has been compromised and added to in order to reflect those parochial requirements. Any organisation that requires designs to be “signed off” by a wide variety of parties (and in particular, if those parties are in different departments with their own agendas) will tend to result in increased complexity. This is as a direct result of the negotiating process required to achieve signofff. For example, the head of security will want her new security standards included, the head of infrastructure will want the latest server builds, the head of information will want to see the latest data modeling tools being used, etc. Organisations that have such functional silos result in individuals whose only objective is to see the particular area of pet design and requirements reflected. Achieving consensus in such environments is the original "design by committee", which is the antitheses of simplicity.

Simplicity is about saying NO!

Simplicity by its very nature requires the ignoring of certain requirements, the jettisoning of capabilities and the removal of bloat. This means the system will in many respects be less capable than a complex one. By removing capabilities that are rarely used its possible to make the remaining capabilities more efficient. However, this removal of capability can be hard to achieve in a corporate environment. Some people are going to be upset that their requirements weren’t met. Others may choose, with Machiavellian intent, to use the absence of certain capabilities to undermine the political position of the designer or project. As a result, there is nearly always a tendency not to remove capabilities that might represent an unnecessary complexity. How hard is it to say No to requirements, that might be eminently buildable, in your organisation?

Nobody is responsible for simplicity

Aligned strongly with the common need to achieve consensus amongst multiple functional silos is the fact that nobody represents simplicity as a requirement. Security, Infrastructure, Information Architecture, etc  will all have strong proponents, but nobody reflects Simplicity. For many years Steve Jobs was Apple’s “Chief Simplicity Officer” and was able to enforce simplicity through his direct influence. However, few (if any) other organisations have ever achieved such strong influence and support for the requirement of simplicity. Without it, the topic always becomes a second order requirement and with no strong support or voice.

Simplicity is hard

There’s often a common assumption, unwritten or otherwise, that simplicity is easy. However, its not the case. Issuing a corporate dictate that complexity is to be avoided is never successful. As Steve Jobs once said:
 “Simple can be harder than complex: You have to work hard to get your thinking clean to make it simple.”
Apple achieves simplicity in its business and its products by intense focus and effort - unless this is explicitly recognised, planned for and supported then its is very unlikely to happen. How many organisations, once a design has neared agreement, then budget for more intense activity to simplify that design? Few, if any, do so. Simplicity has a cost that is rarely acknowledged or budgeted for. We might not all have Steve Jobs characters to champion simplicity, but acknowledging its difficulty and cost is a first step.

Design evolution will tend towards complexity over time

In the 1970‘s Manny Lehman published his “Laws of Software Evolution” by studying the evolution of complex systems. Although the rules have been updated over the decades, their essential points remain true and un-challenged. The rules state, amongst others, that:

  • As a system evolves its complexity increases unless work is done to maintain or reduce it.
  • As a system evolves all associated with it, developers, sales personnel, users, for example, must maintain mastery of its content and behavior to achieve satisfactory evolution. Excessive growth diminishes that mastery.
  • The quality of systems will appear to be declining unless they are rigorously maintained and adapted.

In other words, increasing complexity over time is inevitable, unless conscious actions are taken to avoid it. Even initially simple systems can become complex as they are added to over time. Originally simple and elegant architectures can become corrupted and complex as they are evolved to reflect changing technical fashions. Unless an organisation takes explicit actions to mitigate these tendencies then its systems will become more complex.

The curse of “I want technology x on my CV”

Designers and Architects are often very conscious of the need to “stay current” and ensure their CV has the latest buzz-words on it. In a world where careers are increasingly built not within an organisation, but by jumping between them, this is even more the case. Contractors wanting to ensure their ability to get the “next gig” will want the latest hot technologies on the centre of their resume. There’s therefore a very real sub-plot at play where the designer of the solution might be bringing in new solutions, design patterns or technologies not because they are genuinely needed, but because they need them on their CV. Often this is done not overtly, but by over-emphasising the need for complex requirements that justify their chosen solution. Which leads us nicely to our next point.

New technologies make it easier to introduce complexity

There is often an assumption that the problem of complexity has to do with the past. Old systems, a lack of good design practice over the decades, etc. In my personal experience the age of a system often has little to do with complexity. I observe too many cases of over-complexity in the very latest designs, whose very purpose is sometimes to remove the complexity of the past. For some of the reasons outlined in this post, its often hard to achieve simplicity in new designs.

With the profusion of technologies available to the modern designer, the potential to introduce complexity is infinitely greater than it was in the past, when the available toolset was much more limited. System designers and architects are often not well-versed in simplicity and they are given a toolkit with which they can sometimes run amok. Simplicity is an art and few technicians are schooled in its techniques. Its therefore perhaps not surprising that some of the worst examples of unnecessary complexity can be found in the newest designs. Treating complexity solely as a problem of decades of legacy therefore misses the target. Complexity is as much a problem for the latest designs as it is the oldest.

Exaggerated requirements

Non-functional requirements are notoriously difficult to accurately document. Often they are little understood and left until late in the requirements cycle to collect. With little emphasis and inexperienced staff there its a common tendency for the resultant requirements to be overstated. Its often safer for individuals to overstate the requirements than to understate them - nobody gets sacked for delivering an overly competent systems, whereas the opposite is the case for an under-specified one. With over-stated requirements, the system will need to be over-engineered for the real requirements. In other words, a lack of challenge over the requirements results directly in a solution that is more complex than it needs to be.

More resiliency features does not necessarily make a system more resilient

Many designers assume that by making a design more resilient on paper that it will be so. However, that is not always the result. The addition of more resiliency features results in more complexity. From a simple statistical perspective, an increase in system components increases the probability that one of them may fail. More features mean there are more features the operators need to be familiar with, and they are often not given the training or familiarisation necessary. In many large corporates, highly sophisticated designs are often operated by lowly payed operators. If there is a gap between the complexity of the system and the capability of its operators, then there is a greatly increased chance that those operators will perform incorrect actions which lead to system outages. In other words, we have children playing with chemistry sets and the results are unpredictable. Sometimes a simpler design, which might look less resilient on paper, is actually the more resilient design in practice. The attempt to increase the resilience of a solution can sometimes result in a misguided, although well intentioned, increase in complexity.

1 comment :

  1. An interesting commentary, I enjoyed reading it.

    I'd add another thought as an addendum to "I want technology x on my CV" that as a contractor (or even internally cross-charged resource for that matter) in the absence of strong (FTE) technical leadership and governance, blinding an organisation with complex technical science can be your friend too.

    Describing what a former colleague would define as a "square of the hypotenuse" solution to an organisation lacking either the skills or the confidence to challenge it gives an illusion of indispensability and gravitas. It also very nicely engineers a long term future (on lucrative contract rates or cross-charge) as the individual responsible may well be the only person who properly understands it. Having used complexity to illustrate the sheer depth of their technical knowledge, the organisation also feels that it would be foolish to release such talent back into the wild.

    I've seen this repeatedly down the years, and until simple really does become the norm (and thus complexity naturally becomes anomalous) I'm sure it'll continue to thrive.