Python Hack - Unicode Sniffer

Unfortunately, I sort of built the build system we use at work.

I say unfortunately because, since I'm the one that built it and I'm cheap, it was rolled with a minimum of brains and a maximum of free software.

On the database side of things, I use a handy little program called GSAR (global search and replace) that does what it sounds like it does - searches and replaces strings in files.  Unfortunately, it isn't exactly Unicode-aware, which means that files not saved as ASCII can slip through the cracks, causing subtle problems (depending on how subtle of a problem you see a missing piece of functionality being).

This normally wouldn't be a problem, but SQL Server Management Studio seems to flip a coin on installation to decide whether it's going to save files as ASCII or UCS-2/UTF-8/UTF-16 encoded by default.  Most computers in the office save files as ASCII, but there's a few that like to emit Unicode (especially when scripting out tables and stuff).

Like a hawk, or a ninja, or a ninja hawk, I quickly figure out who the offenders are and keep an eye on their check-ins.  I've got an RSS feed for the repository and, to be honest, they don't check in a whole lot of code so it's not a monstrous burden.  I keep an eye on check-ins anyway because, well.  I'm anal like that.

But why bother grepping the files manually when I've got a computer to do the grunt work for me?  I've been putting off knocking together a C# program to do it for me (why should I do it?  I'm not the one who sucks!), Ruby isn't so Unicode-savvy... but wait!  I'm a world-famous Python hacker now and Python knows Unicode!

So I put together a script that will recurse down the directory tree you put it in and, if it finds a file that isn't saved as Unicode, print its name out.  As a word of warning, it's not the greatest thing I've ever done and it catches binary files in its trawl line as well.  For my purposes, that's just fine - there's just text files living in the /Database portion of the central repository.  Now I can have it shut the build down when it finds the file straight-away.  Everyone goes home happy!  Except for the developers who got the bum install of SSMS and have to do some Save As... gymnastics every time they touch a file.

You can grab the little script here.  Feel the magic!  Feel the power!  Marginal utility!

Oh, and I updated the links to my world-famous marginal utility IsDebug for .Net 2.0, too.

4/23/2008 update - I've gone ahead and slapped the WTFPL on it. So do WTF you want with it now. Or don't. You won't hurt my feelings either way, honest.

It's official - I'm a Python hacker!

Lately, it hasn't been building the application that's captured my imagination so much as figuring out a way to validate the correctness of the application. When you're dealing with a sprawling behemoth of an application of varying levels of quality (this isn't a dig at co-workers - I've got more than my share of turds baked in there) that's so absurdly data-driven that... well. I've come to grips with the fact that chunk by chunk, we've built a borderline unmanageable enterprise application.

There's blocks of the application that I'm flat-out scared to touch because of their fragility. There's a reason that Jenga never took off as an application platform.

There's so many tables (I'll ballpark it at 300, but it could be more than that) that I have to walk through stored procedures and table definitions to figure out why many of them exist. Thankfully, at least they're well-normalized even if, again, some are really, really arcane.

More interesting to me than anything else lately is testing the application. When it comes to a giant database-driven web application, there are no easy answers to "how do you prove it is working correctly?". Well, at least none that The Google is willing to impart on me.

As I've mentioned before and will be talking about again soon (I'm busy hacking, d00dz!), I heart the hell out of Watir. You run WatirMaker, you walk through your application, you're 80% or more of your way to a test case. But... it runs slow. It takes over 3 hours to run 130-ish test cases of varying complexity on the old desktop I snuck the regression automation environment on to. I love it, but I worry about its scalability.

Over on programming.reddit, I saw a new tool announced that tickled me in all the right spots - Pylot. It was designed as a web load-testing tool (I've seen the QA folks at work use Quick Test Pro, but it's a fragile piece of shit if I've ever seen (and built) one) and has lightweight validation features built into it. On that end, it's not as fully-featured as Watir (I've updated WatirMaker to do 31337 things like generate automagic tests when you click on text in table cells/spans/the body) but... speed! Free replacement for Quick Test Pro (maybe)!

I saw it and immediately got to wondering about writing tests for it - is there anything that will get me 80% there? Not yet. Could I take WatirMaker and update it so that it emits Pylot-friendly XML? Sure, but doesn't that feel just a lot unholy?

So I downloaded Python and got to work doing the crappiest, stripped-down-est translation of a useful Ruby meta-script to Python that you've ever seen. A few hours of grunting and banging my head later, I've got something to show for it, thanks in part to the duders over in #python on EFNet.

Does anyone have any experience working with buffer objects? I'm getting one handed back to me and I'm completely lost as to how to coax it back to a string.

um, str(yourbuffer)

that's what makes python so difficult to pick up -- its simplicity

If I were them, I'd clown on me too. In my defense, The Google wasn't all that helpful when it came to working with buffers in Python and I'm a complete new jack and calling yourbuffer.tostring() wasn't getting me anywhere useful and enough ball-aching.

So here's my first stab at PylotMaker.py. It requires the Win32 extensions for Python to run and you run it the same way as WatirMaker - pop open a command prompt and "PylotMaker.py > test.xml", browse around the Internet Explorer window that opens up, close IE when you're done and test.xml is set up for you. Minus tests.

Did I really spend 5 hours knocking out 44 lines of Python instead of playing Team Fortress 2? There's something way wrong with me.