Meaningful Certification is Hard

When I read Raganwald's post on certification a while back, I was impressed by what laid underneath the surface. What he's looking to certify out of the gate is a first-pass cut to weed out the scrubs and find people with a shared set of values that is not up for discussion - no matter what you're doing or how you're doing it, if you can't prove that you're doing it correctly and safely, it hasn't been done. I like the idea because it implicitly weeds out the people who can't program all that well in the first place; if getting it to compile is a chore, tacking on unit tests and being able to talk coherently about why injection attacks are rendered irrelevant is certainly going to be beyond your means. If it's a certification done on a small scale, you're going to be finding an exclusive group of developers and you're going to have a high degree of confidence in any of them and as such, you've pretty much rendered the cover letter+resume then let's small talk thing moot - you're hired.Of course, this ignores a harsh reality. If there's an economic incentive to having that certification, there will invariably be ways to game the certification. I've interviewed (and worked with) enough incompetent but acronym-accredited developers (IBAADs, copyright MEEEEEEEE) to pay certifications little mind at all. They're not bad, but all that's being certified is that the person's achieved some encylopediac familiarity with the terms of the field rather than a true understanding of how and when to apply them... more on that later.

More recently, Joel's posted about just how lame undergraduate programming courses are. Did you know that people come out of school patently unprepared to do development work? This is, frankly, SHOCKING. That last sentence was, frankly, sarcastic.But don't despair people - software is hardly the only profession where this sort of thing is endemic.

I'm friends with some lawyers and I generally hear the same thing - going to law school and passing the bar is tough, but all it does is certify that you've achieved a base level of competence; they come out of the process thinking that they're ready to take on the world and are pretty immediately humbled when they realize just how unfit they are to do any lawyering. It takes 18-24 months for a freshly bar'd lawyer to be able to do much of anything beyond researching case law, being mentored by more experienced lawyers most of the time. Doesn't this sound familiar?

I can't help but see the similarity with getting my degree in information systems - I graduated with good grades (cum laude, what!) and was sure that there wasn't anything I couldn't do. I look back on my first few jobs/years of professional work and shudder at how awful of a developer I was. I look back on code I wrote a year ago and shudder at that too. At this point I don't think that any amount of education could fulfill all values of "rigorous and practical", let alone most of them. This isn't to give JavaSchools a free pass or whatever, because I've interviewed people from them and their abject lack of passion for the craft is a ferocious turn-off for me.

In that regard, I'm with Joel. I don't expect that people are going to step out of school ready to do it all (not gonna happen) but I do hope that they'll have been exposed to more than one way of doing things. In my time, I brushed up (and a few hours a week for a couple of months is about all that I got in most cases) against x86 assembly, C, C++, COBOL and, yes, Java in my four years of college. I'd like to think that I took a little something away from each of these languages about how you can make software work and the lack of LISP or some other functional language is the reason I'm learning Haskell and going through The Structure and Interpretation of Computer Programs these days.

Lawyers go through as much college as I did and then 3 more years on top of that and then a certification on top of THAT and at that point, they're still so green that they require a year or two of more or less intensive mentoring to become productive. Word on the street has it that doctors do all that, except that they're putting in hundred hour weeks while they're mentoring to boot.

I understand Joel and Raganwald - I wish that there was some way to certify that the developers have a shared set of values, a shared passion and a capability to rise to the task that's put in front of them, but looking for this to come from schooling or certification is, or so it would seem to me, praying for a magic bullet.

And even those JavaSchools that churn out mediocre developers serve their purpose - we can't all be superstars. A great certification or four years of a great school won't turn mediocrity into a hacker god, and not every piddly project needs a superstar developer. Certifications and education are both fine signifiers, but it'll take something more rigorous than either to signify what we're all after.

In Praise Of Being Too Clever

Strolling into my local bank the other day to get some cash from the ATM, a little something caught my eye. No, not the ATM what with the fuzzy, flickering screen, although the fact that my local branch in podunkville has a newer model with a nicer screen than the one in the big bank in the BIG CITY left me scratching my head. The nameplate on someone's desk.

Director of First Impressions

I know, and I agree with you - what the hell? That job title's too clever by half. But at the same time, it tickles me in all the right places.

It's more pleasant on the ears and brains than "ombudsman." What kind of word is that, anyway? No fair asking internets or dictionary. Beats "receptionist" too (which is what they are). For whatever reason, it speaks to a lot of things. A company that takes itself more seriously than I do, down to the point where they've got egregious nameplates like that out in public view.

It got me to thinking about Peopleware, the anecdote about The Black Team in particular. If you've never read the book and you work with software in any professional fashion, when you finish reading my drivel, go out and get it. Seriously.

But anyway, it was an anecdote about a team of testers at IBM that was good at their jobs to the point where they were unafraid to loudly announce to the world I am better at my fucking job than you are by dressing all in black. It sounds corny and like a complete non-issue in the age of casual attire, but I remember watching a documentary in high school (not Triumph of the Nerds but something around that time) where they interviewed OGs from IBM and they talked about how rigid the culture was, down to the point where on the first day of work, a co-worker lifted leg of the guy's pants up and critiqued the fact that he didn't have sock garters on. Taken in that light, dressing all in black does take a pretty big pair (but then again, maybe they were all black down to the sock garters).

Sadly, I started wondering to myself - where's my application's director of first impressions? For that matter, where's most software's director of first impressions? Most software out there (the stuff I do for work included) doesn't greet you with a handshake and a smile so much as scowl at you from its office before it runs back inside and slams the door shut, hoping you didn't notice it and if something goes wrong, leaves you to fend for yourself and maybe throws you a bone with a wiki or something lame.

Why doesn't most software, most teams developing software, take what they do so seriously enough that they can get away with doing something goofy like rolling into work dressed like an execution squad when everyone else is in a regimented 3-piece suit and get away with it because they're the shit and everyone knows it? Director of First Impressions too precious, too clever?

Hell no - I'm jealous. I hope that the software I develop is too precious too.

Marginal Utility - IsDebug for .Net 2.0

Unfortunately for my co-workers (and especially the QA department), I play build engineer because we don't have a dedicated build engineer or even build server. Small company, that's how it goes.

In recent months, I've stepped my game up. Batch files, playboy. I know how retarded that sounds - I was this close to having Cruise Control up and running, but I got fed up with the memory footprint it left on my labtop. It's not that trivial a matter when I can time out database calls by having a debugger up and running the same time as I have SQL Server Management Studio with a couple of windows open. Any way you slice it, that shit is s-a-d.

But in the past, I wasn't quite as lazy as I am now. Rather than spend a couple of hours cobbling together a full build script (dear me from 3 years ago - is it really so hard to Google csc.exe?), I did it all by hand from inside of Visual Studio. Clean out the destination directory, swap from Debug mode to Release mode, compile, zip up, yadda yadda.

But wouldn't you know it? It's so simple and yet I managed to screw it up. Multiple times. Handed QA releases that were compiled in debug mode or that had assemblies left over from debug mode and they got clued into it (because our app works slightly differently in the two modes (BY DESIGN!!!)). How was I to know? DLLs look like DLLs, people.

Eventually, I found a marvelous little utility called IsDebug out on the wild internets. Look at that! You drag them DLLs onto it and it tells you whether they're debug mode or release mode! Total win!

When I finally sat down and knocked out our app's upgrade to .Net 2.0, I was kind of bummed to find out that IsDebug didn't work anymore. I recompiled, nothing. Checked file properties, still not working.

But wait! Internets to the rescue again with the clue-by-four on what new properties I should be examining!

So I got Jeff's peanut butter in Jim's chocolate and here we are - IsDebug for .Net 2.0.

I've got a compiled version if you're lazy or don't have a compiler (I imagine this utility is especially marginal if you don't) and if you distrust stuff I write because I'm on the internet or because you know that I cut and pasted it together, there's the solution with source code for you too.

HOORAY!

DebuggerStepThrough Considered Arrogant

It happened again working with vintage code today. I'm tracking down a problem, trying to step inside a getter on the object and all of a sudden, I've skipped over it and find myself in the next statement.

I'm a little confused. I drag the execution point back up to the line I was trying to inspect. Hit the function key to step into it, find myself skipped over it again. What the hell?

So now I'm mired in a meta-problem - in order to get to the bottom of why this (the original problem) doesn't work, I have to get to the bottom of why this (stepping into the getter) doesn't work. I look at it sideways for a little bit and nothing looks out of order, so why isn't that breakpoint being hit? It's definitely being used.

Oh wait, I've seen this before. [DebuggerStepThrough], the most useless code attribute on the fucking planet. The MSDN reference fails it hard, so if you've never run into this abomination, here's what it does - tells the debugger that there's nothing of interest here, move along.

In a best-case scenario, you're saving one mouse click or key press to step through that dumb getter. My hero!

But wait! You don't specify the attribute on a property level, you specify it on a class level. If you're a better programmer than I am, you've managed to not screw up public exposure of private instance variables through getters and setters. Kudos.

But an entire class that's bug-free? And not causing any side effects anywhere else in the application?

Even if you manage to pull that miracle out of your hat, will it never have to live side-by-side with code that isn't quite on par with it? You're still causing problems for people because your code doesn't act like everyone else's code does and that looks awful fucking fishy.

So do me a favor - write that gorgeous, airtight code. Write it as correctly as you can the first time.

And leave the fucking [DebuggerStepThrough] out of it.

Internet Is The New COBOL

I've been trying to quantify exactly what it is about software as a service that bugs me so much. Yeah, my experiences haven't been all that winning with it so far, but I shouldn't let my anecdotes taint the promise of the paradigm, right?

Like Joel pointed out, computers will continue to roll out faster and with more memory, so don't sweat the micro-optimizations and build more features. No way I disagree with that; Moore's Law keeps on chugging away and today's monster rig is 18 months from now's Dell.

But as computers inexorably get faster, more memory, bigger hard drives, our line to the outside world remains fairly constant. Putting aside the comparatively minor problems of pretending that software standards ever worked for anything beyond the trivial (can you name one standard that gave you anything approaching wood?), SaaS seems like a plug-ugly mistake for that reason.

• The success of SaaS is predicated on the use of a scarce resource, the network.

This problem was driven home in performance testing our services. Internally, on a modest (that's a nice way of saying "hand-me-down") application server and database, we were able to push pretty good numbers through. We're feeling good about ourselves and how it's performing, so why not point the load servers at our staging environment up on production? It's a managed site, the hardware isn't vintage, so we're expecting to see some solid throughput.

We start the ramp-up, get about halfway to where we peaked in our internal testing when things start shitting the bed. The load servers start timing out and they drop out of the test.

We saturated our T1.

Oh. Network pipes don't double in size every 18 months? So we set off scrambling again. Do we order another T1 or two? They'll take 30-45 days to be installed and then we'll be paying to have them lay fallow after we run our few days' worth of tests. Maybe more importantly, we're not sure that our clients have any fatter pipes than we do.

Do we find someone with bigger pipes than we've got and tote our load machines over there for a few days? They'll gladly let us set up shop there for a perfectly unreasonable price. Oh. But our connection to our back-end won't work from there so we'll need to be teleconferencing with someone on-site monitoring the servers. That complicates things and we still don't know that performance is a problem.

What we do know is that the network is causing a lot of problems that we can't easily throw more hardware at. When it comes to what a computer can do, the graph trends up and to the right.

When it comes to the stuff backing your service calls, how much shit can you stuff in that five pound sack?

XML is bloated. Really, really bloated. It was designed as a human-readable markup language (it's what puts the ML in XML) but basing communications protocols on it was a dubious decision, hindsight or otherwise. Five pounds.

JSON is less bloated, but JSON parsers aren't as endemic as XML and business people will object because they can't juggle two acronyms at the same time and their tech guys don't know JSON but have a sneaking suspicion that it means way more work for them so they're getting doubly steered back to XML. Six pounds.

You can compress the HTTP that either of them is getting shooted out over but, like JSON, not all clients are going to be able to deal with compression. Six and a half pounds.

In college, professors told me that in the Bad Old Days of computing, you didn't own the computers you worked on. You paid oodles of money to lease an IBM rig and keep it running and even then, it shipped with more hard drive space and more CPUs than were turned on at any point (over the phone line that you pay for).

"But professor, that's awful! You pay all that money and you don't even get all the computer that you could be using? And you have to pay for a phone line so their techs can dial in and turn the magic on?"

"No, that's a good thing. You built your applications under constraints and when you ran into a wall because your app was running too slowly or you were running out of disk space, a call and a few hours later, magic happens and your app's running fine and disk space is no longer an issue."

Curiously, Amazon's following IBM's lead with their S3 and EC2 offerings. Need more space? Got it. More computational power? Bingo bango.

God help you if you need more bandwidth to make those calls to S3 or EC2. Not even god can help you if your clients are running into a brick wall because they saturated their pipes calling your services.

Like buying IBM, basing your architecture around a decentralized network server with flexibly vast resources won't get you cockpunched for making an impossibly wrong decision by most people, but I'll still hate on you because that's how I do.

1. We already knew that storage space and computational power were cheap and vast. Amazon's maybe made it moreso, but that's nothing new.
2. For what it is, the pay-as-you-go model isn't awful. You wouldn't consider it if you could build your own disparately-hosted server farm, but you don't got the bankroll to roll like that which is why you've gone this route.
   
3. Wait a fucking second. You knew that the network wasn't going to get any faster and you designed your application around using it extensively anyway?
   

Congratulations. You've discovered the brave new frontier of decentralized internets architecture and it looks a whole lot like a fucking mainframe.

Web 2.0, meet Web 0.7. Web 0.7, meet UNIVAC.

Playa Hatin' on Oracle in the 2K7

Growing up a young nerd, I spent a lot of time in my formative years on BBSes. OK, entirely too much time that should have been spent playing in the mud and socializing with people instead of keyboards, but I digress.

Along with BBS doors which were (are; I have a computer that supported a Telnet-connectable BBS collecting dust) awesome, I regularly found myself getting into inane flame wars about how much WINDOWS SUX LOL and MACS ARE GAME SYSTEMS ROFL to bump up my system credits so that I could spend them downloading t-files.

I'm not normally one to wax nostalgic (FUCK CDS! BRING BACK ACETATE!), but when I get a telephone book-sized requirements document (or hell, one that fits on a double-sided printout), boy howdy do I start to wish for The Good Ol' Days, when Telling You How It Was was the domain of hackers.

I miss those precious t-files and what they connote to me.

Profanity. Dry, cutting wit. No mincing words, no dumbing it down for people who won't (and can't) "get it". A pretty powerful stench of superiority, and you'd better believe that as you pore over the electrons of that tome, you start to feel better than the assholes out there that don't know shit from shinola either. Goddamn, I love those things. I still go back and read the Cult of the Dead Cow from time to time. It can feel like a product of its time, like a zine handed off with a wink and a nod in a suburban parking lot, but it sure as hell ages better than that requirements document - pick a requirements document, any requirements document.

The cDc got its name from a little programmer joke (I think; please don't hack my site folks, I am 31337 and k-r4d to the bone!!!!) - people would use hexadecimal "magic numbers" to flag specific memory segments so they could find them easily. People figured out that you could spell things with the few letters that afforded you - 0xDEADBEEF is, to my thinking, one of the swankier iterations that hackerdom mustered.

The fine folks at Oracle, the database company responsible for the ungodly machines required to run Oracle and the $400/hour consultants that are required to "tune" your systems so they don't run like raw ass (does raw ass run?) haven't forgotten about these magic hex codes. I can't speak to whether they've forgotten how to tell the difference between an empty string and a null one, but what do we care for 40 years of relational database theory? There is only one true path to Database Enlightenment and ORACLE IS IT. But seriously, watch those fucking spaces when you work with it.

Why so bitter about Oracle when I should know better than to get all frothy about a technology that I don't use and that probably has no effect on my life? THEY KILLED MY MOTHERFUCKING MOTHER, MAN! No they didn't. But I did have to deal with an Oracle salesman once (and if you've ever had to deal with an Oracle salesman, you know it isn't just once) and it left a foul taste in my mouth ever since.

Which is why I'm so glad to see that the way of the t-file is alive and well - this is some quality-ass hatin' on Oracle right here, folks. A few choice quotes (but really, go read it!):

We are talking libraries of 30 Megabytes and more linked in as well as sitting next to the binary, just in case.

[...]

One can only assume that Oracle uses the Intel compiler because no other compiler would produce efficient enough code to run this behemoth of a binary in acceptable speed.

[...]

And we would like to welcome Oracle Corp. in the year 2007, the century of highly advanced, mixed-case passwords.

When I was young, after getting over wanting to be an astronaut and paleontologist, I wanted to be a guy who dug deep into the cruft of software and systems, ripped the secrets out of them and brought them back to the world.

I never became that guy (and doubt I ever will because I spend too much time playing video games and I'm not that smart), but I am glad to see that there are people out there hacking away and still producing quality t-files. That they're straight hatin' on Oracle is just a triple word bonus.

Now if you'll excuse me, I have to start praying that no one ever looks at my code ever because I'd probably break down sobbing like a big stupid baby if it ever received that kind of brutal scrutiny.

Software as a Service - Oy Vey

Software as a service (SaaS), we need to talk.

You had so much promise. Mashups! Loose coupling! And other buzzwords/phrases that architects, CIOs and developers could somehow all get behind. Tangentially, does anyone still say "the network is the computer"?

I guess it was a pretty cool idea. Rather than having to worry about the boundaries between Widgets Foo and Bar, you just wave your stupid hands, utter things like "SOAP!" and "XML-RPC!!" and presto! Those fuckers are working together perfectly because of the magic of standards-based communications.

No more fugly COM calls. CORBA? It's dead to us. What's old (piping text files between widgets, Unix-style) is new again! Hooray XML!

It's so simple, how can this possibly go wrong?

Glad you asked. You produce a service. The person on the other end - do they know how to consume the service? Do they take into account things like "am I trying to read this file before you've finished streaming it over the tubes to me?" That web services definition that you published - are they really adhering to it? Will those line breaks you put in to make it readable throw a wrench in the works (oh if only I saved that godforsaken e-mail chain to forward along to The Daily WTF).

Let me try putting that another way - you're selling your software as a web service. What happens when, for any reason, people find themselves unable to consume it? If you're like me, you'd like to curse at them for their brazen incompetence and write them off. If you're also like me, you realize pretty quickly that you're cannibalizing your own bottom line by writing off these clueless retards in DRAMATIC FASHION because they're the ones that are paying for your stupid service.

Their problem suddenly becomes your problem. Managing one project at a time is enough of a nightmare; now, in addition to the one you're only barely managing, it's your job to asymptotically manage another.

But I don't mean to exclusively hate on SaaS - this problem extends, in one form or another, to any product that you sell. At the point that it leaves your hands, no matter how fully-rendered, it enters the payers' hands and no matter how drop-dead simple, how intuitive the interface, someone is going to fuck it up and then your headaches begin.

Just that when the inevitable fuck ups happen to be piped over SSL with proxy servers and firewalls and misbehaving routers between points A and B, life seems a little less rosy.