Elephants are hard to figure out for people who are blind, but at least blind people can touch the elephant. Software is invisible, can't be touched and isn't anything like things we're familiar with. It's much harder to figure out, which is why what passes for understanding of software is a collection of unrelated-to-reality myths.
Elephants and Invisible Software
What would we think of elephants if the vast majority of people had never seen one in any way, even a picture? What if the few people who had encountered an elephant were blind and their experience limited to touching one part, for example a leg or the trunk?
And what if elephants were nonetheless a very important part of daily life, both personal and commercial? This is hardly thinkable, but that's what software is.
Myth-conceptions
Software is important and ubiquitous, but also invisible. The information we have about it is mixed, inconsistent and mutually incompatible. The vast majority of people, including programmers, resolve this by accepting a set of myths about what this unseen world is like. The myths tend to make sense and explain what software is about in a satisfying way. Most programmers, with their real but typically skewed experience with software, accept the same myths, but add details. The few programmers who can't or won't accept the common software myths find it difficult to communicate with the majority who hold firm to the myths.
Software "myth-conceptions" of this kind aren't just stories, or ways of talking about software. They are widely shared concepts and metaphors that guide how software is designed, built and maintained. Since many of them have been imposed on software from the outside, they tend not to fit with the reality of software -- like things designed for chipmunks, but imposed on elephants. The result is that the process of building software is really expensive and hard, with crappy results.
I have, through hard personal experience over many years, grasped the mythical nature of many widespread thoughts about software. I hope someone will systematically lay out this world of myths that weighs down and perverts software. I have already described some of the myths in prior posts and hope to go into more depth in future posts.
The Software Factory
The job of a factory is to make high quality things. There are inputs, processes, stages, and work in process. At the end, you get what you expected, when you expected it. Let's make software work like a factory -- a software factory! Hah! Here is more detail.
Software is like a Building
First you have to have a plan. You figure out what you need and work with an architect to lay out the details. Then you put the job out to bid. You pick the best bidder by some criteria. The software builder gives you a plan with dates, and often expects partial payments based on progress. The builder gets permits and approvals.The site is prepared, the foundation is dug, footings are poured, the basement created, and the structure is built and the outside put on. The inside work is done -- first the plumbing and heating, then walls, floors, appliances and finishes. Final inspections take place, and the building is ready for use.
Sounds practically like software, right? Yup! This is one of the prime myths; its fit with reality is marginal. Acting on this myth is a prime cause of software development disasters.
Software is just another thing to Manage
Lots of activities involving different skills happen in an organization. They all need to be managed. Management is a skill that can be taught -- that's what getting an MBA is all about, after all. Some organizations need to build software, just like they need to set up a new office in Chicago. It's just another activity that needs a skilled, experienced person to manage it.
Not! Here's a post that goes into it.
Advances in Software
There's a widespread myth that software is one of those things that advances at a fierce rate. Maybe you were trained in a certain kind of software and were good at it, but after a couple of years you become hopelessly out of date and effectively unemployable.
For certain kinds of low-level software roles like Novell Sysadmin, this is an appropriate thought. But in general, it's dead wrong. Software advances incredibly slowly; often it moves sideways, and sometimes goes backwards. When software is created for a new hardware environment, it is often extremely primitive, with all the "advances" that have been thoroughly proven by widespread use simply ignored. Much of what is presented as software "advances" is little but a fashion.
Software and Security
Securing software is much like securing a building or country. You have to have a wall. The wall should be strong and high. It should have lots of defenses against being breached. It must be constantly monitored. The places where people can enter must have extra protection, and everyone who enters must have ID and submit to a search. There are well-understood methods for accomplishing all this, and conforming to accepted procedure is essential.
That's the myth. The myth guides the vast majority of cyber-security thinking. It is pernicious and wrong. I've written about this already, and will write more.
Modern Software Methods
It's true (according to the myth) that software has been a chancy thing to build. There have been lots of delays, overruns and out-right failures. But now, with method X, we've put all that behind us. We finally understand how to create a flexible, dynamic process that puts all that behind us. We are all (for example) Agile with certified Scrum Masters.
This myth has persisted for decades. All that changes is the value of X. This myth is a perennial favorite. I've written about Agile specifically here, and will write more.
Software Innovation
Software is an exciting field, with innovation constantly frothing about. Software engineers are highly creative people -- after all, every line of software they write has never been written before! It's no wonder that the cup of software is just bubbling over with innovation.
Nice thought. But the fact is, after all these years, software projects fail left and right. Managers grasp at all sorts of things in vain attempts to increase the odds of success. Innovation is all about making something better; the challenge in software is the struggle to rise to the level of adequacy.
Conclusion
The myth-conceptions go on and on. There are a myriad of "solutions" floating around for any given software problem. The vast majority of the solutions make things worse.
There are in fact, fast, effective ways to build software. But the methods are flagrantly at odds with most of the myths that dominate the industry. So the methods remain marginal, employed mostly by small groups of outsiders who are desperate, and really want to win.