I always travel ready to get stuck and be forced to work remotely. My tool of choice for that varies, but has recently been a third-generation iPad armed with my Nokia 800’s old folding keyboard, PocketCloud, and Prompt. With these four simple tools, plus Azure and AWS in a pinch, I can pretty easily get a good day’s work done anywhere. So when I got stuck in Los Angeles this past Saturday, I wasn’t worried: I knew I’d still be able to help Fog Creek get stuff done.
You know what an iPad, PocketCloud, and Prompt do not help with?
I’ve been at Fog Creek seven years. I’ve worked through emergencies before: I’ve been there moving systems around, rebuilding databases, getting emergency code fixes out to work around infrastructure problems. I’ve even written code on an airplane to fix a bug in the account registration system on Thanksgiving, pushing it out right after we landed. When stuff breaks, I’m there.
But this time, there was nothing I could do to help. With the exception of providing some very minor assistance with the shutdown and power-up when we thought the datacenter death was immanent, all I could do was to sit and watch.
I tried to think of something to do, anything, to make up for my absence. I spooled up machines on AWS and Azure so I could…write code if I had to, I guess? And nabbed copies of the deployment system for…some reason. I wanted to do something to help out, and was, briefly, being a noisy jerk in the chat room trying desperately to find something to do.
One thing I’ve come to realize is that, sometimes, the best thing to do is to shut up and stay out.
I love that at Fog Creek, everyone I work with is bright, focused, and eager to build amazing stuff. But that means that my individual absence, while not ideal, is not going to make or break anything, and right now, the help Fog Creek needs is 100% people on the ground. The best way I can help from LA is to quietly monitor the chat room, pipe up if I know a specific answer that no one else currently available knows, and otherwise, keep to myself.
The annoying thing to me is that I’ve been in the reverse situation before, many times: trying to get a system back online and answering every five minutes if it’s back yet, or whether someone can help, is ludicrously frustrating. But it was incredibly hard to recognize that I was doing the same thing when my role was reversed. I was so used to being able to help that it took awhile for me to genuinely understand that I couldn’t.
The next time this happens, I’m going to follow a strict check-list before interjecting:
- Make sure I try to understand the full problem first. That includes not asking, “what’s the situation with ABC, and have you tried XYZ?” until I’ve read the full chat logs (if applicable) or talked to someone on a private channel or face-to-face who is clearly currently idle, and therefore interruptible.
- Evaluate whether I have any relevant skills or expertise to help. Is the team trying to figure out how to quickly ship 10 TB of data to S3? Unless I actually have real experience trying to accomplish that, I’ll be quiet. Anyone can google the random comment I saw on Serverfault if that actually becomes relevant.
- Even when I do have relevant experience, if someone is already providing accurate, relevant information, I should be quiet. The random guy on the other team describing the migration process for PostgreSQL may not have as much experience with it as I do, but if he’s right, no one needs to hear me validate it. If they’re actually unsure if the information they’re getting is accurate, they will ask for confirmation from someone else, and I’ll provide it.
- Evaluate whether what I’m about to say is actually productive. Even when I have experience at hand, if the comment I want to make isn’t going to move things forward, it’s useless. If something will take ten days to complete, I know a cool hack that’ll cut it to eight, and the actual problem is any solution has to be done in five hours, then I can keep it to myself. And finally,
- Do not 4chan the conversation. People stress out and need to blow off steam. When I need to do that, I’ll do it off-channel, to keep the main forum clean. When other people do it, I’ll ignore it, and not add fuel to the fire.
Instead of trying to help with the data center, I’m responding to public forums, working on marketing campaigns for a new product we’re releasing when this is all over, and doing some performance work on that same new product—basically, the stuff I’d be doing if there weren’t an emergency. And I’m also making sure I’m well-rested so that, whenever I actually manage to get back to Manhattan, I, too, can help out with the bucket brigade.
I’m annoyed it took a hurricane to teach me to stand back and not throw myself in an adrenaline-fueled craze of help when something goes wrong, but I’m happy it took.
To all my fellow Creekers, and the awesome folks from Stack Exchange and Squarespace who are helping out: you are all mad awesome people, and you deserve serious accolades. I’m sorry I can’t help, but I can’t think of any group of people I’d trust more than all of you to get this done. Good luck!