Experienced Engineers and Complexity
Complexity in software is probably the scourge of our age. It can decimate companies, and befuddle end users. It can cost your company millions of dollars of decreasing productivity. The problem is that in most companies, and engineering departments, it becomes the pink elephant in the room, taboo even to speak about. I wrote about complexity once, and my opinion of Sir Ocham's Razor and software development, and if anything my opinion of squashing it has become even more extreme than before, having seen first hand the results of runaway complexity on top of complexity.
Unfortunately, in some shops, this complexity is rewarded by upper management as “creative” problem solving. Basically it is hacking on top of hacking. Hacking together a demo, version 1, or prototype UI is typically fine, except when that prototype becomes production. When budgets get tight, most companies become reluctant to perform that system overhaul. The ironic thing about that is that the overhaul will often save the company more than 10x that periodic budget in recovered innovation and reactivity to marking requests.
At any rate, most good experienced engineers working on a project, whether their own, or someone else's have a “feeling” about 1/3 of the way through the project that the solution they have been working on is too complex. Frequently there is indeed a better solution that they would come up with had they the freedom to stop and think about what they were doing. Usually when they stop and go to their manager to let them know that they have been following the incorrect path, they are penalized, so they stop doing it. Most professional engineers now swallow the impulse to stop and think, and just “make it work.”
It gets worse, frequently by the time they are about to release the product, they have degraded to the point where they are just sticking for and if blocks to handle problems created by the inadequate architecture they have implemented. This isn't the sign of a bad engineer, it is just what software engineering is like. When the command comes down to add a feature, the engineer secretly has a feeling of dread, because they used everything in their bag of tricks to get the initial release out.
The remedy for this is obvious, if one of your engineers comes to you and tells you that they are headed down the wrong path, give it to them, convince upper management that they are much better off allowing the engineer to fix their architectural mistake right then than months down the line when they need a feature and can't implement it.
Some recent issues of this are Twitter and Microsoft. Twitter at least has the intelligence to stop, and build a parallel system, even at the great cost I'm sure they are paying for it. They will end up a much better company from a software and balance sheet point of view. Microsoft on the other hand is a different story. Someone there should really have stopped Vista. I mean, we had all already waited what, 6 years, 8? What would another year or two have done. We all, those of us who run Windows anyway, still use XP anyway. Microsoft has gained nothing by just “getting it out,” but instead they have lost tremendously in their reputation, and now their balance sheet. They are sending people in droves to OS X and Linux on the consumer side, and business is starting to sniff around Mac and Linux to see if a switch can be done. Think about that, the next time an engineer says to you, “we are going the wrong way!” Ask them if they are willing to stick their necks out as to the urgency of this change, if they are, chances are you'd better listen to them.