Vampires (Programmers) versus Werewolves (Sysadmins)

Kyle Brandt, a system administrator, asks Should Developers have Access to Production?

A question that comes up again and again in web development companies is:

"Should the developers have access to the production environment, and if they do, to what extent?"

My view on this is that as a whole they should have limited access to production. A little disclaimer before I attempt to justify this view is that this standpoint is in no way based on the perceived quality or attitude of the developers -- so please don't take it this way.

This is a tricky one for me to answer, because, well, I'm a developer. More specifically, I'm one of the developers Kyle is referring to. How do I know that? Because Kyle works for our company, Stack Overflow Internet Services Incorporated©®™. And Kyle is a great system administrator. How do I know that? Two reasons:

1. He's one of the top Server Fault users.
2. He had the audacity to write about this issue on the Server Fault blog.

From my perspective, the whole point of the company is to talk about what we're doing. Getting things done is important, of course, but we have to stop occasionally to write up what we're doing, how we're doing it, and why we're even doing it in the first place -- including all our doubts and misgivings and concerns. If we don't, we're cheating ourselves, and you guys, out of something much deeper. Yes, writing about what we're doing and explaining it to the community helps us focus. It lets our peers give us feedback. But most importantly of all, it lets anyone have the opportunity to learn from our many, many mistakes … and who knows, perhaps even the occasional success.

That's basically the entire philosophy behind our Stack Exchange Q&A network, too. Let's all talk about this stuff in public, so that we can teach each other how to get better at whatever the heck it is we love to do.

(Sometimes I get the feeling this idea makes my co-founder nervous, which I continually struggle to understand. If we don't walk the walk, why are we even doing this? But I digress.)

The saga of System Administrators versus Programmers is not a new one; I don't think I've ever worked at any company where these two factions weren't continually battling with each other in some form. It's truly an epic struggle, but to understand it, you have to appreciate that both System Administrators and Programmers have different, and perhaps complementary, supernatural powers.

Programmers are like vampires. They're frequently up all night, paler than death itself, and generally afraid of being exposed to daylight. Oh yes, and they tend think of themselves (or at least their code) as immortal.

System Administrators are like werewolves. They may look outwardly ordinary, but are incredibly strong, mostly invulnerable to stuff that would kill regular people -- and prone to strange transformations during a moon "outage".

Let me be very clear that just as Kyle respects programmers, I have a deep respect for system administrators:

Although there is certainly some crossover, we believe that the programming community and the IT/sysadmin community are different beasts. Just because you're a hotshot programmer doesn't mean you have mastered networking and server configuration. And I've met a few sysadmins who could script circles around my code. That's why Server Fault gets its own domain, user profiles, and reputation system.

Different "beasts" indeed.

Anyway, if you're looking for a one size fits all answer to the question of how much access programmers should have to production environments, I'm sorry, I can't give you one. Every company is different, every team is different. I know, it's a sucky answer, but it depends.

However, as anyone who has watched the latest season of True Blood (or, God help us all, the Twilight Eclipse movie) can attest, there are ways for vampires and werewolves to work together. In a healthy team, everyone feels their abilities are being used and not squandered.

On our team, we're all fair-to-middling sysadmins. But there are a million things to do, and having a professional sysadmin means we can focus on the programming while the networking, hardware, and operational stuff gets a whole lot more TLC and far better (read: non-hacky) processes put in place. We're happy to refocus our efforts on what we're expert at, and let Kyle put his skills to work in areas that he's expert at. Now, that said, we don't want to cede full access to the production servers -- but there's a happy middle ground where our access becomes infrequent and minor over time, except in the hopefully rare event of an all hands on deck emergency.

The art of managing vampires and werewolves, I think, is to ensure that they spend their time not fighting amongst themselves, but instead, using those supernatural powers together to achieve a common goal they could not otherwise. In my experience, when programmers and system administrators fight, it's because they're bored. You haven't given them a sufficiently daunting task, one that requires the full combined use of their unique skills to achieve.

Remember, it's not vampires versus werewolves. It's vampires and werewolves.

What's On Your Utility Belt?

Like any self-respecting geek, I'm mostly an indoor enthusiast.

But on those unfortunate occasions when I am compelled -- for reasons entirely beyond my control -- to leave the house, I do so fully armed with my crucial utility belt items. Yes, you heard me, I transform from the geeky Bruce Wayne to the gosh-darned Batman!

At least, that's how I like to think of it.

I've been talking about this every-day carry stuff for quite a while now. The 2010 edition of my personal utility belt is mostly subtle tweaks, but I daresay it's the best one yet.

The art of every-day carry must go on. What you see here is the contents of my pocket:

1. Patriot 32 GB USB flash drive
   
   Now you can have a whole freakin' hard drive worth of files in your pocket. Just in case, you know, you have an emergency need to upload a virus to an alien mothership, or something. Beware the many cheap, slow USB flash drives out there; this one is a real gem. It's inexpensive, and per my measurements, about as fast as they get. This is important because the larger the flash drive, the more important speed becomes. Hard to believe I've gone from carrying a 512 megabyte flash drive in 2005 to a 32 gigabyte flashdrive in 2010.
   
   
2. Leatherman Squirt P4
   
   Ounce for ounce, nothing beats the utility of the Leatherman Squirt. This time I opted for the plier (P) version instead of the scissors (S), and after seeing how much more generally useful the pliers are, I am now a little ashamed to admit I ever carried the wussy scissors version. Pliers all the way, baby. And yes, that is a Pulp Fiction joke you see on it.
   
   
3. Fenix mini AAA LED flashlight model LD01R2
   
   Since I've been carrying them in 2005, the average LED flashlight has gone from bright, to very bright, to amazingly bright, to ridiculously blinding laser-like bright. It's scary how bright these fancy milled aluminum AAA LED flashlights get now. What I like about this one is that it lets you trade off stupid-brightness for something practical, like greater runtime: you can twist the top to switch levels: 9 Lumens for 11 hours, 28 Lumens for 3.5 hours, or 85 Lumens for 1 hour.
   
   
4. Small Nite Ize s-biner
   
   These little nite-ize carabiners are awesome for quick attachment and detachment of your EDC items, but I'll warn you: resist the urge to put everything you carry on a carabiner, because if you do, the weight and "jangliness" goes up a lot -- and this way lies madness. Consider how many items you actually remove from your keychain regularly. For me, the only item I frequently removed to work with was the Squirt, so that's the only one I put on a carabiner.

Rest assured, everything here is carefully selected with the appropriate levels of monomaniacal attention to detail. For this weight and size, I don't think you can do better. (And don't think I've forgotten about optimizing my wallet, either. Oh no. Quite the contrary.)

However, I have to add a special category this year for the other must-have EDC utility belt item: the smartphone. What self-respecting superhero would leave the house these days without their smartphone? I'm not religious about it, but I use and rather like the iPhone 4, and I'm continually amazed how many things it does that I used to carry separate items for:

• cell phone (obviously)
• "Nintendo DSwhatever" for portable gaming
• GPS
• point and shoot digital camera
• near-desktop quality mobile web browser and email client
• mp3 player with speakers
• audio and hi-def video recorder
• DVD player
• ebook reader
• watch, alarm
• emergency flashlight (via front facing LED flash control)
• scanner
• level and ruler

Smartphones really are the ultimate gadget. The list of functions is already enormous, and I'm sure I'm leaving out a few other things that you can do with a modern smartphone.

In a pinch, I could conceivably drop the AAA LED flashlight and the USB flash drive from my EDC kit and substitute the smartphone. Not exactly, mind you, but it's getting closer every year. At this rate, Apple could introduce a flip-out blade on the iPhone 7 and reduce my entire EDC kit to one item.

Anyway, that's what's on my utility belt in 2010. What's on yours?