Software development - the journalism model

You probably don't know this about me, but I've studied journalism extensively. Back in third grade (for my vast international readership, that's the grade level you're in when you're ~9 years old), I learned journalism well enough that I think I wrote an article or two for my school paper. Maybe I even edited an edition of it - obviously, my journalistic credentials can't be called into question.

We learned that the essence of journalism can be found in the "Six W's" - Who?, What?, When?, Where?, Why? and, uh, hoW?. We also learned that teacher couldn't spell that day.

I humbly submit that these same Six W's should be applied a bit more rigorously to software development and project management, just in case you're trying to figure out if your project will fail or succeed (and if you're not trying to figure out, it will fail).

The Important W's

(in no particular order)

• What (are we doing)?

This one seems so obvious that it should require no explanation, but the flip side of that is that it's so essential that you're dead in the water if you and everyone else on your team can't explain what you're doing. The team should know what you (collectively) are doing in the macro sense and they should have direction in the micro sense.

If you or any of your team can't accurately (and hopefully, clearly and concisely) verbalize just what you're doing, your project's probably well on its way to being necrotic.

If you're a manager, you should be able to define achievable goals in the macro sense and the tasks needed to achieve them in the micro sense. If you haven't and if your people can't either, see you at the unemployment office.

• Why (are we doing it)?

As a lazy software developer, I pose this question to the project management team a lot more than they'd like me to. It's reflex and I think it does everyone a lot of good. I don't want to try to wield phrases I won't understand in a million years like "cost-value proposition" but they're probably applicable here.

In hackneyed colloquialisms I can wrap my pea brain around, whose itch will your software scratch? Will anyone take one look at it and immediately "get it"? Will it make people spontaneously say "this is fucking cool"? These are all really, really good things - if you know that people will think that it's fucking cool, you and your team probably does too, and will be excited and take pride in what they're doing, and the end product will reflect this.

Alternatively, will people take one look at what you've done and shake their heads and gasp in disgust at the turd you've fished out of the shitter and slapped on the table?

• Who (is doing it)?

As in - do you have the people to accomplish your goal on hand already? Or are you expecting that you'll just subcontract this out and everything will work out just fine? Do you already have a team of studs or do you expect scrubs to "step up to the plate" and get things done?

If you don't have faith in your team's ability to accomplish it, listen to that nagging voice in your head and start running for the hills.

Projects can overcome a lack of manpower, but you probably don't have the stomach or the wallet to grunt your way through it. With amorphous direction and unclear goals, even a team of ICFP-certified development gods are going to flounder.

The Unimportant W's

• How are we doing this?

This applies to those "little things" like IDE, language and framework. I don't mean to underplay the importance of all of these; you'll certainly make your work more difficult (OK, orders of magnitude more difficult) if you choose something pretty inherently unsuited for the task (RoR for an embedded system, C for a web application), but I also think that one of the other W's easily steamrolls it - Who. You put your team of studs up to the task and it stands a reasonable chance of succeeding in spite of itself.

• Where am I doing this?

This doesn't speak so much to physical location as to metaphysical location. Oh, did I just blow your mind?

By that, I mean things like version control (where the project lives), and build management. I'd lump bug tracking in here too. I know that Bill de hÓra would disagree and the developer in me wants to agree with him, but then I got to thinking - "hmm... VCS hasn't been around forever and I'm sure some working projects have teams that still pass around spreadsheets with a list of outstanding bugs on them." More importantly, the Who will overcome it - good people know good practices. If we walk into an shop without version control, we'll set it up. It's not hard to find a bug tracker out on the internets, it's a couple days to roll your own and if all else fails, spreadsheets aren't the dumbest way to go about things (OK, they kind of are). All this goes out the window once you meet a certain critical mass of project size, but the Who will obviate it again, as you'll get good people in there that will make you hire build engineers and all that jazz.

• When

Uh, now? I've lived through multiple dot-bombs, so spare me the "the market just wasn't ready for our innovation" speeches. See Why, jackhole.

Boiling it down to Solomon's Law (YES!), "Code cannot overcome reality." If the reality is that you've got an inept team, are lacking direction, are developing something that nobody wants, it doesn't make one goddamned bit of difference that you've got an A-number-one build system, magical bug tracking, distributed version control and an immaculate code base built on the right language and framework for the job.

No matter how beautiful the package, you're still boxing up a turd.

Oral Traditions in Software Development

The requirements document will be the death of me. I don't have a whole lot of faith in the requirements document if only for the fact that I've yet to work with one that concisely and accurately expresses the needs and intent of everyone that it represents.
I can't stress this highly enough: I'm not very smart and I don't have much of an attention span. That's why "concisely" matters so much to me. You can make that requirements document accurate, but in the process you'll probably end up with something resembling the Yellow Pages for Manhattan. It's accurate up until the point where it puts me to sleep (about three paragraphs that resemble legalese in) and sooner or later, people will get hip to the fact that I didn't read it.
OK, I'm taking that a bit too far. I do read it as much as I think I need to in order to get the flavor of what I'm developing. People sweat way too much over getting into the down and dirty of how things are supposed to work, pre-chewing the details. For downstream consumers of this product (QA, client-facing folks and beyond) maybe this is a good thing? For me, I just want the broad strokes and I'll be able to fill in most of the blanks.
To fill in the gaps between the broad strokes, there's back and forth where I talk it out with the person who wrote the requirements document, because they know what we need better than I do. This works out great for everyone as we quickly hash out details and get back to what we need to do.
Well, it's great until months from now, when QA will have a new set of questions that weren't reflected in the original documentation because they didn't sit in the series of little meetings that development had with the requirements crew. And we'll have another sit down and chat and they'll see what the real intent of the functionality is or I'll go back to the drawing board.
Here's the thing - I really like how this works. Little meetings (30 minutes or so) when needed and when the particulars can fit it into their schedule. I don't have to read much of the documentation which is good for so many reasons, but mainly because people can tell me what they need better than they can write about how things should work.
But here's the other thing - I can see how one would look at this and be calculating a bus number of "one." The person that wrote the requirements document leaves the company and all of a sudden, you're back to just that dry document that's a subset of the expertise that they brought to the table. The developer leaves the company and all the discussions that the product developer had with them that they never inserted back into the original document disappear with it. People are left to scratch their heads over what's going on in with the application, not knowing that this is expected behavior.
Or QA or someone else walks into the application with the expectations laid out by the requirements document and finds that the fantasy of the documents doesn't meet the reality of the application.
I don't like documents.
I like talking things out.
I have no idea how to blend these two competing desires and turn what amounts to an oral tradition into a sustainable development model but if I figure it out, I figure I'll be a billionaire and then it'll be time to settle down to my life's ambition, hunting the most dangerous prey of all - MAN. Just fucking with you, I'll be so rich I'll pay someone to hunt MAN for me.

Agile Development - Can We Stop Designing?

Over on some programming forums I frequent, someone posed the following question that resulted in some heated development (from Agilists and others).
"We have a team that tells us that because they have adopted 'agile development' they no longer need to do any design work before writing code.

To me this seems bizarre, I can't fathom how one could build any reasonably complex application without a modicum of design (hell, even pencil sketches of UIs or something), but they claim it works.

Is it reasonable to develop complex applications with zero design work?"

BUFD (Big Up-Front Design) has gotten a hell of a black eye, but damned if I can think of a better way to wrap my head around a project in a business domain that I'm completely unfamiliar with.
Generating documents is a mixed bag - I think they're great for the process you use to create them (collaborating with everyone involved in the project, picking brains, exposing flaws and needs and hopefully constraints early) but I think they're awful for the fact that the end product is a document that's outdated a week into the development process (give or take; the point where the bulk of your time is developing rather than documenting) and at that point, the different partners have vastly different ideas about what the document means.
Developers think it's a rough roadmap (do I update the documentation with the way things will work in our system? I'll get that later... OK, not really. Have a meeting? That gets old REAL fast), most everyone else (project managers/QA/stakeholders) think it's litany and that's a sure-fire recipe for friction.
Eschewing the design portion of this and flying by the seat of your pants elevates this from an exercise in friction between developers and, well, everyone else to an exercise in sure-fire catastrophe for everything but the most trivial of implementations.
This isn't to say that reasonably complex applications can't be done without BUFD - I'm sure that they can, but with a big fat caveat: the developers have had a reasonable amount of experience with the business/application domain that they're developing for. There you can come away with a working application in the void of BUFD but with yet another gotcha: no one has any solid expectations of what to expect when the developers pull away the veil.
If you're working in an "agile" fashion, I guess this means that you're going to have rapid iterations from developer to project managers/stakeholders at which point they'll yea or nay it and talk about what else needs to be done, which probably means throwing away a fair bit of what you're doing, which means that instead of doing BUFD, you're doing Smaller Bits And Pieces Of Design All Allong (SBAPODAA). On the upside, there's that boost of "look what we're getting done". On the downside, there's that "holy crap I had to throw out another fucking day's work because Joe Blow From Product Management couldn't have told me yesterday that's not how things work?" and on the real big downside, there's the very real possibility that you're going to scuttle the project wholesale because developers are arrogant and think they know how to do everything so much better that you end up with a brilliant application that just happens to be absolutely useless for what the stakeholders need it to do.
I'm pretty sure that pair programming doesn't solve that one.

The case for McMansion architects (part 2)

So I think I might even have reasonably established that handing off a specification to an architect (no matter how good) is necessarily going to be nothing but an unmitigated disaster if you're expecting something arable the first go-round.
But these McMansions, these big, generic, samey houses. These enterprise software applications, this big, onerous, pay a million and pay a million more for consultants to get something nominally useful that no one likes using programs. Why do they get built in the first place?
I keep coming back to a PBS Frontline program called The Persuaders and in particular, the interview with Clotaire Rapaille, a French cat who isn't afraid to pose like he's in a rap video in front of his mansion with his cars (seriously, check the picture on the front page of his internets site) and proffer a bastardization of Jungian archetypes and Lacanian linguistic dynamics for mad profit.
As he explains in the program, the car companies came to him, looking to get in touch with their consumers and give them something that would jump off the lots, so he started interviewing people and came to a revelation - bigger automobiles, taller off the ground, with tinted windows were "code" for "wealth", "power" and "domination."
Rapaille talks about drilling through the mammal brain and speaking directly to the lizard brain. John Coltrane put it more eloquently - "The emotional reaction is all that matters. As long as there is some feeling of communication, it isn't necessary that it be understood." Rapaille sure was on to something with making them SUVs bigger and taller and whatnot, because they provoke gut responses in people. You ask people who drive them why they drive them and they'll tell you "because I feel safer in them" and it really doesn't matter if you were to put the crash tests that say otherwise in front of them. They're safer and that's all there is to it. You ask me, who would just as soon drive a goofy European-small car, about them and I hate them because they're gaudy monuments of conspicuous consumption.
But underneath my upturned nose, I'll admit it - there's the seeds of jealousy in there. I can't explain why, either. I've driven them and hate driving them. I don't like being up high, they handle like tanks and jesus christ the gas milage. But still, when I'm driving behind some jackhole and I can't see through the blacked-out rear window to the car in front of them, wouldn't it be nice to be able to see the guy in front of them?
Why the McMansions? In this context, I think it's easy to see where the infatuation with more house and less soul than you need comes from. For the same money you're putting out on a McMansion (and probably more because deviation from the crowd is gonna cost you), you're ultimately getting "less" house. There's safety in numbers and everyone else is doing it, so it takes an extra dose of courage to get it done.
You can't define soul, but square footage is easy to quantify.
Think about it - deep down inside, you've got that sense of unease gnawing at your stomach. You're spending more money for ultimately less house. The garish abomination that they're building is more than they need, isn't built for them but rather some generic approximization of them, they can't afford it, but... they're getting a whole lot of square footage and maybe I could make it my own by decorating it nicely and choosing the fixtures I'll have the place built out with? It's easy to slide back to the road more travelled.
The lowest common denominator is hard to argue with for a reason. IBM may not be the greatest, but nobody ever got fired for buying it. Budweiser and McDonald's suck, but won't you come off like a snob if you tell your friends you want to get something better?
I'd like to think that I have the courage to admit to myself and others that I'm OK with smaller and better. You probably have the same courage... but do we have the courage?
Product development comes up with a giant specification. You may have the courage to question whether a service-oriented n-tier architecture implemented in .Net 2.0 and redundant Oracle databases communicating with your Oracle Financials OLTP system with the data cached in XML for quick retrieval and the requisite smattering of patterns talk makes any sense whatsoever for this hypothetical product, but how convincing of an argument can you make to the contrary
Product development probably isn't really interested in hearing that these dozens of features that they've spent weeks and months painstakingly realizing and documenting probably aren't needed. Management isn't going to be sold on you telling them that less being more - they see their career riding on hundreds of pages worth of specification and who is this programmer to tell me that less is more and we don't need these features that product development does? We paid that architect good money for a good reason.
McMansions, SUVs and enterprise class software are all about us being sold what we want instead of what we need. It awfully hard to argue with the inner child that's having problems rationalizing away the fact that we're paying 15 percent more money for 15 percent less square footage. Peopleware costs a boatload and wow look at all of the things that it can do (I've tried to figure out what it is and near as I can figure, they've managed to assemble what I can only describe as a Rorschach application), so it can obviously do what we want (nevermind that we need to use the thing and that may not be possible). It's hard enough to argue with our inner child; you get a room full of inner children together and the odds of common sense rearing its pretty head rapidly approach zero.
A quote I've heard that's apt - "in a community of saints, we are all sinners."

The case for McMansion architects (part 1)

So I'm reading an interesting piece decrying "McMansion" software architecture and totally nodding my head. Raganwald's a better writer and smarter than I am (PLEASE DON'T HATE ME MR. WALD) and given that a recurring theme in his posts is casting "best practices" aside and being brutally intellectually honest about what you're doing and how it's failing and can be ameliorated, he's right. But as the devil's advocate with a massive failure of vision, here I go sticking my foot in my mouth.
It's an interesting article to be sure, but at the end of the day, it feels like it's restating Brooks' thesis on essential and accidental difficulties in software development (yes I just reread The Mythical Man-Month, what of it?).
There's a part of me that's totally nodding in agreement at being saddled with cookie-cutter
concepts and then there's another part of me that's scratching my head about what exactly is wrong with using databases and XML and all the other unsexy (proven) technologies that the anonymous architect has proposed to bring the app from a "telephone book sized specification" of requirements to an implemented product. There's a part of me that's giving an e-high five for sticking it to consultants who are paid too much to use a metric ton of buzzwords to not quite tell you what you already know and a part of me that's wondering what exactly is wrong with using COTS products and technologies since I'm not a world-class developer and don't work with world-class hackers and maybe I won't see as many exciting things on the road more travelled, but I won't fall off a cliff either.
I honestly feel bad for the architect. Think about it for a second - a specification that probably runs upwards of a hundred pages. I haven't seen the spec or the proposal, but I feel safe in asking the question - who could possibly put together a proposal that would sanely satisfy something like that? If I'm Raganwald, I'm probably going to outwardly scoff to anyone unfortunate enough to listen to me because I'm an asshole like that, but at the same time I've got to be happy that there's progress that's been made. OK, the initial draft sucks and the architect looks like a chump and here's a dozen reasons why, but at least we're moving from something proposed by a group of people (I doubt it was one person who was put together that specification; I'm assuming that it was spec'd by a committee and written by a couple of people) into a proposal that I can digest and critique and iteratively improve upon to a point where we have a good enough idea of what we're all talking about that we can actually make something.
The initial architecture proposal sucks? I'd be astonished if it didn't. Using XML is the wrong decision? Thank god we're far enough along that I can definitively say that. That's better than we were a telephone book ago.
Above and beyond the obvious problem (a mammoth spec), in handing it to an outside entity, you're dealing with some unspoken problems. That specification is ostensibly the crystallized desires of a group of people, but hiding deep beneath and in-between the lines are the unspoken desires of people that never made it in there. How much of what ended up on the cutting floor could be picked up and stitched together into a fabric that details the essence of what they're after? Was the spec handed off to a single person to trim the fat off of everything? Did they have the courage to ask the terrifying question that nobody wants to ask - do we really need to develop this and the added fortitude and singular vision to come to that conclusion on their own?
Maybe I'm just missing the bigger point because at the end of the day, I doubt I'll ever be the Le Corbusier of software development and I'm just totally projecting here.
I believe that it is possible to overcome the fantasy/reality impedance, but nobody's going to be happy when you do.