The software business is complex.
This is almost a cliché -- but we don't really understand what "complexity" actually means, or how to handle it well. Most big software operations spend a lot of energy trying to get rid of complexity when they should be embracing it. This, more than anything else, is their downfall.
If you don't understand complexity and consciously work with it, it will eat your business for lunch. Here's how to take yourself off the menu.
First, let's define our terms. When I say "complexity," I mean a system where success is dependant on more variables than any one person can hold in their heads, or that can be captured in a spreadsheet or project management tool.
Not only are their a lot of independent parts in a complex system, but they interact and influence each other in innumerable ways.
This adds up to a situation where what we do now determines what happens later, but, because of all the variables, exactly what will happen is not immediately or easily predictable.
The Butterfly Effect (the mathematical concept, not the movie) is a classic example of complexity in action. The idea is that small things (a butterfly flapping its wings in one hemisphere) can lead to big things (a hurricane on the other side of the world).
Mathematician and meteorologist Edward Lorenz coined the term based on his observation of actual data when he saw how very tiny variability early on in a data set could magnify over time and become unimaginably disruptive. He could watch this happen, and even reverse engineer it, but predicting in advance which small things would become big things was impossible.
The technical jargon for a complex system is one that is "deterministic, but not predictable."
The software business may be slightly easier to predict than global weather patterns, but not by much. Gartner, and other industry surveys, have been telling us for years that enterprise IT projects are more likely to fail than succeed.
This isn't because software people are dumb. It's because we don't know how to handle complexity very well. Instead of embracing inherent complexity, we try to hammer it out of our projects.
We're applying industrial age thinking on information age projects, and this almost guarantees failure.
This is one reason tech VCs as a group have an abysmal success rate, and why working on a big software team can feel like you're constantly headed for disaster. Because you are.
- #33: All important decisions must be made on the basis of insufficient data.
- #34: Yet we are responsible for everything we do.
In both life, and the software business, we always have insufficient data, nevertheless our decisions determine the outcome. We can't predict the actual result of any decision, but nevertheless we must decide.
We reflexively deal with complexity by trying to plan really really well. We know that predicting all the variables at the beginning of a big project is difficult, but our human need for certainty means we try to anyway.
This is why big upfront requirements planning is still popular, even though it never works.
Our plans are never right, and they are resistant to change, so we set the project on tracks that lead towards a cliff and it's only through heroic effort on the part of our teams that they succeed.
Success in big software projects is not common and not achieved because of our brilliant plans. It's an anomaly that often comes at a very high cost to the people who manage to pull the project out of the fire at the end.
Instead of trying to manage and plan all complexity out of our projects, we need to build plans, systems, and teams that handle it with grace and responsiveness. This is what psychologists call resilience and it's perhaps the most valuable thing you can have personally and as a business.
Organizational resilience is a core competitive advantage and is housed in the way an organization approaches project planning. This brings us to variability.
This view is a departure from previous lean strategies (like Six Sigma and the Toyota Production System) whose main focus was driving out variation (aka defects, aka uncertainty).
Driving out defects makes sense if you're doing the same thing over and over again -- e.g. manufacturing. It doesn't make sense when your primary activity is problem solving and solution design -- e.g. software engineering and product development.
Reinertsen's insight was that software teams can, and should, learn as they go and evolve a product over time, rather than designing it all upfront and then trying to build it without defects.
From this perspective, value creation is tied to the steady addition of new and better functionality. Value is not churning out of the same widget over and over again, yet this kind of thinking is pervasive in the enterprise software world. Which is why startups consistently win.
When you're working with a process that's designed to exploit variability, you're able to take advantage of flashes of insight and inspiration, as well as happy accidents, to inform the shape your product takes.
Here are four things you can do to build a planning process and organization that's able to exploit variability. I could write an article on each -- let me know if you want me to.
- Finance all projects on a time and materials basis, so you can re-plan regularly.
- Re-plan and re-prioritize at least once a month, if not once a week.
- Build your organization as a collection of small, empowered teams and architect your software to support this. Make sure people on these teams understand and embrace their mission, and that they feel personally connected to each other.
- Have these teams ask "what can we deliver to the customer in the next few weeks that they will find valuable?" and then deliver it, watch the customer use it, and incorporate what they learn into the next plan.
These are foundational to building a great culture in your product teams, and culture is where resilience lives.
A resilient and engaged culture is as important to a software organization as healthy topsoil is to a farmer. With it you can grow incredible things, but without it you're living on borrowed time.
What are your favorite ways to grow an organization that can handle complexity? Write me at bob@citrusbyte.com or reply to this post.
-- thanks for reading! please click here to get more of my writing --