Every user you’ll ever have is trying to make you solve an engineering problem that few engineers have ever seen, and fewer yet have ever solved. Nobody hires a programmer because they have a pre-solved problem, because if their problems are all known and catalogued and solved then their solution would be on the shelf of a store already. That you are a developer or entrepreneur is evidence that what you’re solving is unique enough that the benefit is in your favor, but the odds are not. You won’t know what you’re doing because your customer doesn’t know what he’s doing, and your customer doesn’t know what he’s doing because he’s in the middle of discovery as well.

There are good reasons why your problems are underestimated by your bosses and investors, though, and the biggest is that most things are similar. Most people look and behave alike, and most objects can be compared to other things. So any unique problem a man encounters can be miscompared to a similar problem that’s already been solved. And there is such a strong economic incentive to solve a new problem with an old trick that good problems go misdiagnosed.

The problem of vision (“We need faster horses, Mr. Ford”)

Visualizing the nature of a computer program before its built is nothing more than a bit of useful guesswork, and is akin to picking a random house to begin with on your quest to find a new home. Each new house that you visit will show you something that you didn’t know you wanted before. You may have thought you wanted a big garage, fireplace and swimming pool, but by your third or fourth house you’ve discovered that you want a spiral staircase, finished basement and brick construction even more, and you’ve re-evaluated your need for a pool and garage.

The same happens when gathering requirements for a new software project, and very few customers understand that they cannot possibly know what they need the computer to do for them until they’ve seen how it works. The developer must be part of the requirements discovery process, and everyone must be aware that the discovery phase will continue all through the development cycle and into the maintenance cycle forever. Maybe it’s even more apt to say that the discovery phase is the software, and working programs are disposable artifacts found along the way. 

Indeed, if discovering what you want from a computer is like discovering what you want from a house, then our analogy would continue by saying that sometimes–after visiting and living in a dozen different houses–you realize that you’d be better off spending your money on a backpack and camping gear.

The problem of desire (“But I wanna pony”)

False needs and requirements pollute the vision of a program and are the means by which one can force a project to fail. If you wanted to sabotage an undesirable project, then one must merely gold-plate the requirements or demand inapplicable standards. But ponies, unicorns and rainbows don’t really sneak into program requirements because the request was malicious, but because someone really had their heart set on it. In 2009 fantasies take the form of AJAX, Ruby, SOA and REST. Ten years before it was XML, Java, Extreme Programming and Design Patterns. Ten years in the future it will be Functional languages, Solid-State hard drives and schema-less databases.

These problems enter the design because desire was aroused by something that had nothing to do with the problem. When an engineer wants a pony it’s because there’s something interesting about it, and when a customer wants a pony it’s because they feel they need it to be competitive. They hear “relational databases are passe, now” and insist their payroll application has to run on a sharded Map-Reduce cluster.

But the universe itself is the most malicious partner; it works hard to contrive problems that–by nature–will get worse if you try to solve them with the ponies and unicorns of your time. It sets you up for a fall before you’ve even started writing code.

The problem of change (“Now it needs to be a submarine”)

The Enterprise Resource Manager (ERM/ERP) is a classic example of institutionalized cluelessness. Every business thinks it has to have one, and they all think they know what they want from it. But there aren’t any businesses who have a successful installation of an ERM/ERP because the company’s nature will change before they’ve finished configuring it. It will change in at least one way that wasn’t anticipated in the design of the software, and so the company’s employees either begin their inevitable trend of working around and avoiding the software, or the engineers begin their inevitable trend of damaging the software’s design to cope.

Here are some specific examples of changes that broke ERM/ERP designs that I’ve seen:

• the company recategorizes its product line based on utility and lifestyle rather than brand and designer;
• the parent company divests itself of you and withdraws the connection to its parts and inventory database;
• a manufacturer starts using more than one UPC per product because they changed the packaging;
• the law now requires that national holidays must be counted against vacation time if the employee takes a vacation day immediately before/after that holiday;
• a loan company files to become a bank and store deposits that earn interest.

In addition to changes in the business model, there are changes in low-level protocols that frequently force compromises, hacks, or even entire system rethinks:

• CSV files exchanged with a partner were always denormalized, but then one company starts using normalized records with mixed column mappings;
• another partner decides to ditch file transfer as a way to submit data, and creates a web service that you’re supposed to invoke commands on;
• the shipping carrier reveals that their tracking numbers are all recycled at the end of each year;
• new rules require every credit card number to be encrypted, but you still have to be able to quickly search for matches in a database of millions;
• a partner switches to mixed-case alphanumeric PO numbers.

Changes like these are always justified in the short term or immediate scope; they always have a benefit for somebody. And the reason why major changes occur is because business growth itself is driven exclusively by change. A business grows because it convinces a customer to change from wet shaving to electric, or from charge-cards to credit-cards and so-on. And businesses compete by changing their nature to be strong at what others are weak at. 

The problem of organic growth (“I see lots of features, but what does it do?”)

If the solution to fuzzy vision is to experiment and wander, and the solution to desire is to bring in the accountants, and the solution to change is to stay agile, then the penalty for all three over time is a confused product.

Data doesn’t like to live in multiple places, so the programs closest to the data accrue functions to be everything to everyone. Now you get software that’s both a floor wax and a salad dressing, and your training costs for both users and maintenance developers increases–for doing nothing more than explaining to them why things are they way they are.

This is ultimately loss aversion: nobody wants to throw away good code, so they try to keep it, but make it stay relevant somehow by morphing it. Alas there is no anti-wrinkle cream for software, and most code-base owners are far too tolerant of ugliness. 

Software must be treated like it is, like tissue paper. It’s a step on the road to programming humans instead of machines; each development project is an academic exercise with the side-effect of being useful to somebody for a while. In a way, state and community colleges have already figured this out and have their students rebuild the school’s web site and class signup software every few years as part of their CS curriculum. It gets thrown away a few years later by another batch of kids, and they always do a shitty job of it each time, but this is what it’s like in the enterprise, too.

Progress must come from the evolution of computer languages, and not from the artifacts built with them. Each generation folds another layer of experience and insight into new syntax, not new modules. Next year’s batch of kids will still make a shitty web site, and they’ll graduate to make shitty payroll and inventory management systems for the companies they work for. Everything they make will have to be thrown away. But each generation’s produce will get better and better in quality because the tools they use will get better. Since the dawn of computers in business, each re-imagination of software for managing tea shops gets better and takes less time to develop.

A treadmill, forever. But we can make the treadmill nicer.

Example: a voyage of discovery

1. The customer says they need a program to pick stocks based on their complex formula, and it must be able to buy and sell securities on the market in real-time
   
2. The developer delivers said system, but then the customer discovers that the formula needs tweaking and orders more changes
   
3. After many tweaks, the customer now asks for a custom formula editor and a language to express them in, so they can make the changes themselves
   
4. Still unable to get the formula right, the customer now wants to ditch the editor and have the program use a genetic algorithm to auto-discover the ideal positions to take in each stock
   
5. They now realize the cost of all the trades is killing them, and ask for the real-time buy/sell feature to be disabled and swapped for a daily paper report
   
6. Suddenly they discover they can sell these reports, and need the program to maintain subscriptions and send the reports out in email
   
7. It starts to become clear that the genetic algorithm’s picks perform much worse than the S&P 500. The company loses all of its subscribers and asks the developer to add checkbook management features and re-package the software as a shrinkwrapped alternative to Quicken
   
8. After a failed marketing campaign, the company ditches the whole program and puts all its money into index funds