Friends

EDIT@08:16 UTC/GMT. Wow. That was ugly. I expected it to go for 30 minutes and have maybe 1 minute of broken connectivity. Instead it lasted over 4 hours and we had 10 minutes of downtime directly related to the load balancer upgrades and then another 5-10 minutes of downtime when our primary Pingback database server crashed and the secondary couldn't take over; which could have been indirectly caused by the network upgrade missing a self-VIP.

Anyways, we're up, we're working, the load balancers are barely breaking a sweat right now and I need some food and a shot of whiskey. I don't even *like* whiskey!!

Thanks mhwest and dnewhall for helping out!

---

On Saturday the 14th at 4AM UTC/GMT we will be upgrading the operating system of our network load balancers to a newer version, one that will allow us to use both CPUs! Nifty, because multiprocessing is nice.

Since we have 2 load balancers, the plan is to upgrade 1 at a time, and there really should be very little impact to our website. Hopefully you won't notice a thing and I'll get to go back to the hotel and watch some wonderful late night infomercials.

We've got a lot of exciting projects coming up for 2010 and we're hoping that we'll be able to deliver them all to you, that you will find it useful/cool/lovely and then you will use the site even more. Behind-the-scenes work like this will give us the capacity to handle the anticipated traffic, so expect a few more maintenance windows especially in the beginning of next year as we've got some neat ideas to improve performance around here! We had the recent 30-45 minute outage yesterday due to one of our logging databases filling up disk space -- not so great design coupled with my human error in handling the initial problem -- and it looks like we're going to finally have some resources to eliminate stuff like that. I can't wait!

As usual, I will be updating status.livejournal.org before and after, just in case you are not able to reach our main website during the work.