Complexity Theory and Agile Methodologies

I spent the latter part of my time in my Master’s program researching complexity theory and the implications it might have on leadership and product/software development.  As I did my research, I learned that the principles of complexity theory are very well aligned with those of Agile development methodologies, despite the fact that Agile methodologies seem to have been developed with few, if any, formal ties to Complexity Theory (CT) or Complex Adaptive Systems (CAS).  I drew two key conclusions from my work.

1 – Those that recognized the need for new planning and management paradigms in creating great software did so largely based on their own common sense.  They weren’t reading about theory and then attempting to apply it.  They were simply experimenting with new approaches that worked because they represented reality better than their predecessors.

2 – The fact that Agile methodologies have become so prevalent and have been so successful in software development lends credence to the tenets of Complexity Theory.

Complex Adaptive Systems (CAS)

CAS are open systems that constantly adapt, and interactions and relationships between interdependent agents are considered more important to generating emergent knowledge than are the individual agents, or nodes in the network that describes the system.  Cause and effect are non-linear, and multiple structures, behaviors and influences may create significant changes in a Complex Adaptive System.  In many cases, output, or effects, become causes, and a very tangled relationship between input and output exists.

The agents in a CAS interact together and are adaptable, in that they can learn from individual experience, and be influenced both by other agents within the system, and by other pressures, internal to or external from the system.  Feedback loops in the form of direct individual experience and memory, and observation of interactions between other agents contribute to the constant adaptation of both the agents and the system as a whole.

A Complex Adaptive System does not require a controlling mechanism to direct the generation of output, and, in fact, the output of a CAS cannot be directed by a single controlling mechanism.  Emergence occurs naturally, as a result of the interplay between agents.  The nature of a Complex Adaptive System, defined by its adaptability, non-linear cause and effect, and organic evolution without a controlling mechanism all combine to make it complex and irreducible to its component parts.

Agents within a Complex Adaptive System are influenced by a myriad of things: other agents, outside information, pressures from inside and outside the organization, previous personal history, and the emergence of subgroups of agents within the CAS. These influences may cause one agent to respond one way, while another agent responds differently.  Those responses go on to influence both other agents, and the original agents themselves, as outcomes from what may be considered small changes spark new changes or die out.  It’s easy to imagine within a network of many agents that adaptation is impossible to control and the outcomes that emerge are difficult to predict.

Waterfall vs. Agile

Traditional methodologies are often referred to as “Waterfall” methodologies because they rely heavily on execution of a pre-defined and commonly understood sequential series of phases that make up the life-cycle of software development project.  These phases at a high level include planning, development, testing, and deployment, and a project goes through each of these phases only one time throughout its life.  A Waterfall approach relies on top-down, bureaucratic, command-and-control methods of leadership.  It uses front-loaded planning activities which aim to solidify the requirements early for a given software development project.  It can be cumbersome and inflexible, creating conditions in which adaptation and innovation are difficult, if not impossible, to achieve mid-stream while software development is underway.  For some, the Waterfall approach is viewed as too cumbersome and inflexible to keep pace with the demands of modern software applications and their end users.

Change, however, is inherent to the software industry, and customers, users, and development team members themselves often refine ideas as time passes, generate new ideas based on feedback and experience gained throughout the project development timeline.   They are constantly exposed to new approaches to feature functionality by virtue of their membership in the software industry.  This brings them in close contact to other products that may be wholly unrelated to the product or project in development, but which may offer relevant and valuable ideas that could positively impact their development efforts were there mechanisms that allowed for change throughout all phases of the project.

Modern development methodologies, which fall under the umbrella term of “Agile” methodologies, began to appear around the year 2000, and have proliferated in the decade since.  The defining characteristics of Agile methodologies include an inherent flexibility based on a highly iterative approach, where the least amount of planning necessary to understand a software feature is done as late as possible before that feature is built, and then it is immediately tested with end users.  This approach is repeated over and over throughout the software development life-cycle, and thus the end product can be seen as something that evolves based on increasing knowledge of the team, which consists of developers as well as users of the system.

Complexity and Agile

While Agile methodologies have become more and more popular over the past decade, critics believe that Agile approaches lack structure and discipline, and lesser quality deliverables may be produced as a result.  Agile development consists largely of a set of principles and somewhat standardized techniques that have evolved in practice more than they have been developed in theory or by research.  Some scholars believe this weakens the merit of the methodologies.  Agile methodologies are also more difficult to employ when working with distributed teams, especially those that cross time zones and cultures, because the approach relies heavily on frequent communication.

Despite these potential drawbacks to the use of agile methodologies, viewing agile approaches through the lens of complexity science adds value to the discussion.  For instance, iterative software releases managed on a frequent basis are principally aligned with the CAS concepts of adaptation and evolution.  The agile process that includes frequent testing and input from stakeholders maps to the CAS principle of feedback loops, and the more loosely structured environment found in agile teams aligns with CAS concepts of distributed control.

Supporters of Agile methodologies view software requirements themselves as dynamic and changing, and believe they are better suited to adapt to these changing requirements by employing a process that allows for and embraces change throughout the cycle of development.  Supporters of traditional software development methodologies take the stance that non-technical end users don’t necessarily know their requirements and need direction from developers, while Agile supporters believe neither the customers nor the developers know the full requirements of a system, and they must generate that knowledge together in order to arrive at the best possible solution.

4 comments

  1. This is beautifully done. I am about to give a presentation on this very topic, myself. I decided to just Google “complexity and agile” to see what else is out there. Your article is one of the very best I’ve ever seen on this topic.

    1. Thanks, Jeff. I really appreciate the feedback. Complexity theory fascinates me, and I’m always happy to find someone else that is interested in it as well, especially someone that thinks of it in terms of software development and organizations. Good luck with your presentation, and if you’d ever like to bounce ideas off of someone, feel free to reach out anytime!

      1. Your study of the connection is much more formal and more detailed than my own. In fact, I have always just assumed that Agile Development arose from Complexity Theory, but that the practioners failed to give credit where credit was due. I am really surprised to find that Agile developed on its own, but this scenario actually makes more sense to me now.

        I am so glad to find someone else who is fascinated by this and who sees the connection, as I do. It is plainly obvious and very empowering to me. I wonder how many others think about the topic in this way?

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s