Software management - The Oregon Trail model

OK, forget reading Peopleware and The Mythical Man-Month. If you want to manage developers, there's only one - no wait! - two things you need to do.

1. Be born in the late 70's/early 80's
2. Play The Oregon Trail on an Apple II

That's it! If you can make it to the end of the trail, you know everything that you need to know about successfully managing software, so go successfully manage the shit out of a project or three!

Wait. You're still here? I should go on? OK.

You have your hunters, guys who strike out in hopes of lucking into a bounty of delicious, delicious meat. Speaking authoritatively as someone who's never hunted or caught anything of note while fishing, I'll say that a good hunter operates off of instinct, innate ("animal") intelligence, skill and luck. Legends tell the tale of the hunter who struck off into the deep of the jungle with nothing but a knife and came back carrying a gorilcopterous (work with me people, I live on the edge of the tundra) or some big tasty animal that lives in the jungle⁰.

Then you've got the gatherers who plant crops and (hopefully) eventually got to harvest the bounty. It takes a deep investment in time and resources to see a field from seed to harvest, and even then there's any number of natural disasters that can beset you along the way. This isn't to say it's easy. On some level, we all understand how plants grow. They eat sunshine, water and carbon dioxide and crap oxygen and chlorophyll, right? On another level, there's a hell of a lot that goes into it - seeds cost money. Machines to till and harvest cost money. Irrigation costs money. You've got to know when to plant. What to plant. When to rotate your crops.

Developers are an impatient bunch, which means that we don't generally have the patience for the small, far-off, rewards and repetitive work that it takes to grow a crop. But we get hungry, so we're going to strike out and hunt, hoping to bag the next seminal idea. OK, it's probably not a seminal idea by any stretch of the imagination, but for a brief moment in time, you really feel like you've crafted something brilliant and have found something more than you could have hoped for, something that's somehow bigger than you could have imagined.

But eventually, the quarry of the once-fertile plains and forests of imagination!!!!!!! will run dry. Your hunters will pick up on the fact that there's no more ideas to be hunted, only tedious fields that demand regular, monotonous maintenance, and they'll migrate elsewhere. This is all well and good, but every now and then wouldn't you like a steak to go with those potatoes?

But again, I get ahead of myself - let's take a step back to see what a dated educational game has to do with any of this.

In The Oregon Trail, you're in charge of seeing a family¹ make the trip on the Oregon Trail from Missouri² to Oregon³.

You notice how they emphasize "try"? It's because of a simple fact that you don't want to admit to yourself - no matter how simple it seems like it should be, that shit is fucking hard to accomplish⁴. To the outsider, it's as simple as getting from point A to point B. That's really all there is to it, right?

Go figure - there's nuances that you hadn't even considered when you started out. Could this mean that you'll eventually discover that there are nuances upon nuances? This makes the decision of something that seems as simple as when to leave a dizzyingly difficult one.

Leave early and you'll be freezing and making slow, painful, progress⁵. Leave too late and you'll face the unenviable, super-difficult, task of over-wintering⁶. Face it - you've tried to make a trip like this before and you've probably still got the bruises and scars from the last one, but you keep telling yourself that this time will be better.

You need supplies to make the trip. Oxen to move your cart⁷, clothing to keep you warm⁸, food⁹, replacement parts¹⁰... and bullets for hunting¹¹. And naturally, you're constrained to a budget that's tighter than you'd like.

Making matters worse (you mean it gets worse?), you're probably making the trip with a bunch of egomaniacal (wo)man-children. I mean, we all strive to be egoless in our tasks, but we all take an undue amount of pride in an elegant solution and invariably take it the wrong way when someone points out the glaring flaws in our implementation. Or maybe I'm the only one, who knows?

But... about those supplies. Some of them are effectively interchangable. If you've got a good hunter (you have a good hunter, right?), you can exchange bullets for food in the wild - that's the biggest bang (pun unintended) for your buck. Then you can trade food for just about anything else you need along the way¹². You can run pretty lean-and-mean¹³ but you need enough food to keep your bellies full.

You're going to want to hunt (because it's fun!), but not every hunt can be a winner.

What can I tell you? Times is tough out there. Even with a gang of elite hunters firing on all cylinders and a pile of food (ideas!) as high as the eye can see, sometimes nature just doesn't smile on you. As in any pursuit, it's entirely seemly that you can do everything right and still fail.

Not only are times tough as hell, but you're fairly constantly reminded of your own failures and the failures of others in a big, somber way.

Is there any wonder why Einstein up there didn't make it? Wasn't cut out for it. You can't teach people how to hunt - they're born with it or they'll never get it, now matter how blue in the face you get trying to explain it to them.

The trip itself? It's possible that your people are entirely happy in Missouri right now. You can crack that whip all they want, but if you don't instill a powerful longing in them to reach the promised land of Oregon, you'll never make it. In fact, you might find yourself unwillingly invited to a Donner Party. And not to ruin the surprise, but everyone at the party will be eating but you.

---

0. you know, your run-of-the-mill whispered about in legends superhacker
1. your development team
2. the start of the project
3. the project's end (you do have a concrete end-point in mind, right?)
4. #9 on Software's Classic Mistakes - Wishful Thinking
5. sometimes you have the toolchain you need to complete your project, other times you're kind of winging it as you go along - bootstrapping too much of your own technology stack only shrinks the chances that you'll ever make it
6. running out of funding
7. computers
8. technology stack - compiler, language, yadda yadda
9. ideas
10. source control and the other niceties of modern development (build server, unit tests)
11. ain't nothing more dire than running out of bullets - there's no metaphor here
12. dumbest way of describing the point of open source ever?
13. agile??!?

This week's dispatch from the seventh circle

"I don't understand why FxCop is complaining about this static property on a static class."

"Probably because static classes are stateless, so exposing properties doesn't make a whole lot of sense."

"Yeah, but I want to expose properties so that a caller can set the values that they want."

"But that's really not such a great idea. Either overload methods with the parameters you're trying to read from properties or let callers instantiate the class and set the properties themselves."

Why did I find this so disturbing? Probably because it took two days for the chap to make the change.

The other reason I found it disturbing? I did the math in my head and decided that there would be approximately zero point in trying to get him to understand why globally mutable state can quickly lead to misery (false negatives/positives) when you're dealing with the MTA that NUnit runs in.

Design patterns as religion

I've come to an understanding of and guarded appreciation for design patterns.

They're not at all useful in and of themselves, but they are useful at a level of abstraction - they give us a common language to explain our implementations with ("...so basically, the model objects are just kind of structs and we cajole them into doing interesting things by Decorating them...") and, more importantly, they let us learn about the boundaries, limitations and realities of the design space that we're working in by revealing the problems that get solved and solved again and again and... you get the picture.

If your objects start to look like a pattern, it's a curiosity. Patterns were discovered "in the wild" to begin with, so don't tire your arm out patting yourself on the back for rediscovering one. No gold star for knocking out a Flyweight, sorry. On the flip side, don't flagellate yourself when you realize that you've implemented a Singleton - if it emerges from solid design, why would you throw it away?

It doesn't matter if you're blindly implementing a pattern or blindly rejecting an implementation because it looks like a pattern that someone on the Internets doesn't like - it's cargo cult programming either way.

Design patterns are there and were there before anyone started looking for them or writing about them. People have classified subroutines as a design pattern - like other "patterns", this isn't a reason to use or not use them, it's further establishing where they come from - introspection on how we make programs work. There's something to be said for taking a stroll through that meta-field, but I don't linger there too long. Most of the time.

All this is a long-winded way of explaining why my monocole absolutely popped out the other day at work. I was busy doing something or other at the time (transliterating objects to an XML format I don't control using unhealthy amounts of Reflection, I think?) when I saw one of the other developers explaining something or other. I popped out my earphones to catch the explanation midway through.

He had the Head First Design Patterns book open on his desk and was walking through his ideas.

"Why are we bothering with inheritance? The domain model you guys put together is too convoluted and doesn't work in this case."

"If there's something we missed, it's not like it's set in stone. If it's really hairy, we can walk through it and see if there's some reasonable way to get it factored in and if there isn't, we can go further back and see how we can make the model work for us."

"The <I forgot already!> pattern says that you can use hashtables to store the data for your objects, rather than inheriting from a base class."

"Well, yeah. But think about why we use objects and I think you'll see why that's probably a bad idea."

...

"For our model objects, we're use them not only to encapsulate associated data but to make it easily discoverable - when you're working with them internally, it may seem pretty convoluted that your parent base Foo class has property Bar that's exposed by a child Buz object (which has more properties and methods from the IBaz interface that Foo implements), but that's all readily discoverable by anyone who's consuming a Buz object. They could kind of care less how it works, they just need access to have access to its state and be able to send it messages to have it do interesting things."

He was not sold by my explanation and the discussion closed in on arguing (something about our respective tones and the other developers in the office getting quiet and looking uncomfortable) about whether it makes more sense to have the classes representing data internally like a struct or as a hashtable - I scoffed (and still scoff!) at the idea but told him that he was the one implementing it, so if he had a good reason for properties exposing a hashtable behind the scenes, I was fine with it.

I started to look up hashtables and figure out whether he was really on to something or whether my gut feeling that it was adding an unnecessary level of complexity to the design (OK, I was premature optimizing in my head and wondering about the relative weight of sparsely-populated hashtables, too) when, a few minutes and pages in the book later, he announced to another developer in the office (!!!) that the book said that you should only use hashtables when you're dealing with a lot of inheritance and a lot of properties.

I don't think there was any amount of selling I could have done to get him to walk away from the curious implementation - it was design pattern as dogma. Blinders on, reject facts, implement on faith and The Pattern will see you through.

I'm still trying to figure out if I was wrong (probably, yeah) for knee-jerking away the use of hashtables or not. I'll let you know when I turn that page.

Aging codebases, a postscript

Ron Avitzur's comment on my post wondering aloud as to whether code can age gracefully done tickled my brain. I mean, I think about code a lot. I think about software and making software a lot.

But for some reason, I hadn't ever pondered how truly strange software is - if you look at it a certain way, it's never truly broken. It may be old, it may be ungainly, but it will always do something.

Sure, the hardware that it ran on may disappear or cease to function and the compilers may evaporate... but that doesn't mean that the code's broken. It just means that you've got a little work ahead of you porting that software to the latest and greatest hardware and toolchain and bingo bango your software's back from the dead.

Of course, there's always the question of "should we?". Times move on and inevitably reach a tipping point where maybe it makes sense to let that sleeping codebase lie.

Steve McConnell refers to this as a "doghouse problem" - when it comes to building a doghouse, everyone has an opinion on what color to paint it, what the roofing tiles should be, the exact dimensions, yadda yadda because it's a problem so eminently tractable that we can easily wrap our heads around it. Hell, it's within the realm of reason that we could build it ourselves. We have a pretty good idea of what it'll cost to make and how long it'll take.

But ask us to price the cost, materials, resources and time required to build a skyscraper and... good luck on estimating that one. How hard can it really be though? They're both buildings, right?

A couple of real-world buildings brought this into clear focus for me.

First is an old mill building not too far from where I live. For the years I've been here, it's been uninhabited - a hulking, crumbling mess of a building. 100 years ago, I'm sure it was a bustling center of commerce, an economic powerhouse. Now it's gone past creepy and into sad. It's being demolished to make way for... I don't know. Probably another drug store or something.

(zoom on the crumble)

Second is This Old House, or what's left of it. There was a house there once, nothing here now but the bones of the old building rising like so many dinosaur bones from the ground.

(check them bones)

Either one of them could be broken down as much as possible to be built back up to work again, but it's the house whose bones will remain in place while the mill's ground into the dust. Looking at it, the possibility of fixing it piecemeal doesn't even enter into your mind - it's over a hundred years old, frigging brick and huge. The house, on the other hand? Over a hundred years old, but the frame and chimney are still sound - we can rebuild around them, putting the walls and stairs and windows in different this time.

When looking at open source projects, it's not uncommon for one of the mentioned features to be their "clean codebase" - on some level, this is kind of laughable because who but a geek gives a shit what the code looks like as long as it runs well enough and does something useful for you?

On another level, it shows that there's people who have taken pride in their work, that fix broken windows, and have a vested interest in making sure that a year from now, it will have crumbled less than that mill building.

Think about building something that will stand the test of time and it just might. Make it concise and useful enough and who knows? Its framework just might live a decade or so.

I hope to someday build something so good that I won't be able to hear the lamentations of the developers responsible for supporting it because I'm busy watching Wheel of Fortune too loud or whatever it is that incredibly old people do. Did I mention that I'm watching The Wheel in my rocket car? Because ideally, I'm incredibly old, reasonably well-to-do and the future finally delivers on its promises.

Postscript

I biked by the house again earlier this week. Nothing remained.

I still hold out hope for a rocket car-filled future.

SQL Server, we need to have a talk

So I'm working on normalizing the database the other day.  I want to move some columns from a table that I'm about to make vanish to another table and I can't remember the syntax for it.  I should have hit up The Google (the SQL syntax for adding tables/columns to tables really is dead simple) but instead I was extra lazy and fired up the table designer inside of SQL Server Management Studio and scripted out the changes to the tables.

I finish that up, finish modifying the stored procedures, everything looks good.  I run the SQL script, fire my black box tests against it.

Hmm.  Things ain't working right.  The column that I just defined to hold a decimal is causing problems.  I check my stored procedures to make sure I'm not doing anything insane, check my business logic to make sure I didn't bork anything there, it all looks good.

I take what I'd just scripted and distill it down to its essence.

create
 table scratch
 (
   percentage decimal (18,0)
 )
go

insert
  into scratch
values (0.5)

select *
  from scratch

Not-so-curiously, the scratch table now contains a lone value of 1.0.

When you look at it, it's obvious what went wrong here - I just defined a decimal with zero digits of decimal precision.  Or, as I like to call it, a fucking integer.

I obviously biffed in the first place, but this left me scratching my head - why on earth would you make the default for a decimal (effectively) an integer?

As an added bonus head scratcher, there's no warning about the loss of precision/silent rounding when you store the value in there.  I don't know what should happen there (do you want an exception because of that sort of thing?) but I do know that it rhymes with "leaky abstraction."  I'm not an RDBMS guru, people.

Times like these, I understand why it's regarded as best practice to treat money as two integer values (dollars and cents!) - databases are a bit funky when it comes to storing out your decimal values.

Can codebases gracefully age?

Terrible shame. Monstrous shame.

I really need to stop looking at my old code. More than that, I need my old code to work all the time in the face of needs and requirements that didn't exist when I first wrote it. Is that even possible? It isn't when I put it like that.

Probably not. At this point, I'd say "almost certainly not," but it's not like I've seen a whole lot of other projects' codebases (it's not something that I do for fun).

The sad thing is that, not to sound like a monstrously arrogant prick (even though I am), I'm the best developer where I work, by far. When Larry O'Brien made the observation that bad developers are actively counter-productive, it riled some people, but I'm in complete agreement. I'm the best developer at my office (sometimes by more than an order of magnitude) and I'm out-and-out embarrassed by code I wrote six months ago. As in "how the fuck did this shit ever work?" embarrassed. As in "if I ever come across a time machine, I'm going back in time to punch myself in the throat for that mess" embarrassed.

And it only gets worse.

This isn't humility, this is common sense. Microsoft doesn't want me, Google wouldn't fart in my general direction. I'm pretty sure even Fake Steve Jobs would prefer that I not have any Apple products.

I'm certainly no superstar developer (do they really exist in nature?), but even developers way smarter and better than I am have problems with their codebases, especially as they age.

Microsoft's not having such a smooth time of things with Vista - people are hating it so much that they're apparently seriously thinking about doing away with backwards compatibility, regarded by some as the albatross around Vista's neck. But... backwards compatibility is what sells Microsoft products for a whole lot of people. Microsoft's got scary smart dudes working to insure that your old software will keep working, even if it means that old, really questionable stuff, stays in the OS to keep it working.

They must make bunches of money off of it (there's a school of thought that says that the solid support/backwards compatibility fuels the virtuous cycle that keeps large companies from jumping ship from Microsoft), but with the time overrun and backlash about how assy Vista is, they're thinking about sacrificing a cash cow for a restart on the accretion of bugs they have to support in their codebase.

Apple's another company that's got scary smart dudes working for them. Like, even the guys that don't really work for them are scary smart. Another place where I'm sure I'm the janitors are smarter than I am. Even they've got problems with their codebase.

Even there, when it came time for OS X to come out, they broke backwards compatibility with System 9. There was an emulator or something for it (I could swear my PowerBook has it on it) but they made a clean break of it. With point releases of OS X, they brutally deprecate features and functions, leaving developers to clean up their act or not run in the new version.

It's kind of rough, but it's probably also the right way to go. Yes, as a first course, you want to refactor the hell out of your code. But they've accepted the fact that their codebase doesn't age gracefully and when they find really objectionable code, they rip it out as soon as they can. Developers who support Apple apps seem to not mind it all that much (maybe they do?). As for the users... compare and contrast people's reaction to Leopard and Vista.

I just don't think that it's possible for code to age well. Bits and pieces may stand the test of time, but the whole will become a not-so-secret shame. Well, maybe TeX has aged gracefully, but (and no offense intended) nobody who's as smart as Donald Knuth is wasting their time reading this blog.

(click for more bigger brick)

I think that's a totally lovely sight, but if that were software, a bunch of people (the developers supporting it, the people trying to sell it) would be in a lot of pain because that is unfit.

I mean, OK. First it was designed as a brick edifice to hold a building up. Then someone decided that we could slap an ad for Coca-Cola on it. Then a real genius figured "hey, it worked for Coke, why not make it work for Gold Medal", and in the process, they lost sight of what it could reasonably support.

If that were software, I'd be scared to touch it - if I do anything with it, I'm going to lose the Gold Medal ad or the Coca-Cola ad or both or the lovely interplay between the two. I don't want to be responsible for fucking them up.

It only overtly serves one of its intended functions well at this point, when, if it were software, people would be demanding it support all three. At this point, we can all agree that there are simpler ways to support multiple ads (they're called billboards, have you heard of them?) and maybe we should just strip it back to what it can do well (be a brick wall).

In the face of all this, why do I keep developing? In the hope that next time I'll get it right.

So what about you out there - do any of you work on applications that only get better with age?

Python Hack - Unicode Sniffer

Unfortunately, I sort of built the build system we use at work.

I say unfortunately because, since I'm the one that built it and I'm cheap, it was rolled with a minimum of brains and a maximum of free software.

On the database side of things, I use a handy little program called GSAR (global search and replace) that does what it sounds like it does - searches and replaces strings in files.  Unfortunately, it isn't exactly Unicode-aware, which means that files not saved as ASCII can slip through the cracks, causing subtle problems (depending on how subtle of a problem you see a missing piece of functionality being).

This normally wouldn't be a problem, but SQL Server Management Studio seems to flip a coin on installation to decide whether it's going to save files as ASCII or UCS-2/UTF-8/UTF-16 encoded by default.  Most computers in the office save files as ASCII, but there's a few that like to emit Unicode (especially when scripting out tables and stuff).

Like a hawk, or a ninja, or a ninja hawk, I quickly figure out who the offenders are and keep an eye on their check-ins.  I've got an RSS feed for the repository and, to be honest, they don't check in a whole lot of code so it's not a monstrous burden.  I keep an eye on check-ins anyway because, well.  I'm anal like that.

But why bother grepping the files manually when I've got a computer to do the grunt work for me?  I've been putting off knocking together a C# program to do it for me (why should I do it?  I'm not the one who sucks!), Ruby isn't so Unicode-savvy... but wait!  I'm a world-famous Python hacker now and Python knows Unicode!

So I put together a script that will recurse down the directory tree you put it in and, if it finds a file that isn't saved as Unicode, print its name out.  As a word of warning, it's not the greatest thing I've ever done and it catches binary files in its trawl line as well.  For my purposes, that's just fine - there's just text files living in the /Database portion of the central repository.  Now I can have it shut the build down when it finds the file straight-away.  Everyone goes home happy!  Except for the developers who got the bum install of SSMS and have to do some Save As... gymnastics every time they touch a file.

You can grab the little script here.  Feel the magic!  Feel the power!  Marginal utility!

Oh, and I updated the links to my world-famous marginal utility IsDebug for .Net 2.0, too.

4/23/2008 update - I've gone ahead and slapped the WTFPL on it. So do WTF you want with it now. Or don't. You won't hurt my feelings either way, honest.