« Lessons learned after a major system crash | Main | INND and crashing machines; what you don’t get at first gloss »

who needs coffee, when you’ve got power outages

So I get to my office this morning and find multiple calls from my customer who has a data center in LA. Evidently they still have power problems in LA this morning, because when I call up the NOC guy in our data center, he says that he’s been up for 48 hours dealing with this.

Still: where are your battery backups? Where are your diesel gensets?

Now I get to rebuild history files again, figure out why MySQL replication isn’t automatically reconnecting (I think I know why now, though, and it smells like pilot error...) and watch as the load on our master servers jumps well into the double-digits due to the inrush of MySQL replicants and file replications...



In other news, I did find a really cool firefox extension called Scrapbook, which allows you to save copies of web pages, annotate them, and even edit them before and after saving. It’s the answer to another customer’s desires, who loves to save web pages “as they are”. It’s also cool to save amusing Craigslist ads for posterity. (For the record, that particular ad, which will expire in about 20 days, was a paid ad. And, of course, I’ve saved it.)

TrackBack

TrackBack URL for this entry:
http://www.jbaltz.com/mt/mt-tb.cgi/23

Listed below are links to weblogs that reference who needs coffee, when you’ve got power outages:

» INND and crashing machines; what you don’t get at first gloss from Entropy Reducers Amalgamated
After the past few days of multiple outages at one of my customer's LA datacenters, I got to learn a few things about the resiliency of popular packaged unix software. Read this all the way, since the very last step... [Read More]

Post a comment

(If you haven't left a comment here before, you may need to be approved by the site owner before your comment will appear. Until then, it won't appear on the entry. Thanks for waiting.)