Category: Work

…to both me and to Firefox. 🙂

One year ago yesterday, I joined the Mozilla Foundation as a paid sysadmin, just in time to wrestle with the server overload as we released Firefox 1.0 the following day. It was pretty cool to see Firefox 1.0 get released on my birthday last year. Now both Firefox and I are another year older.

Here’s to another great year!

Most of this was written by Myk, but since he doesn’t have a blog, I’m paraphrasing it and posting it here. 🙂

Our firewall upgrade at the Meer colocation facility this evening was successful. The upgrade caused 23 seconds of downtime while we switched firewalls and several minutes of slowness until we fixed a misconfigured firewall parameter. The new firewall (plus redundant failover firewall) is now up and running, and normal access to all services is restored.

We’ve been in the process of moving from one facility to another, and for the interim, there is a direct ethernet link between the two facilities, but behind our firewall. There are only 6 machines left at the old facility currently, and until tonight our traffic was still being routed there (and then across the cross-colo link to get to the other 50 or so servers at the new facility). The new firewalls we upgraded to are located at the new facility, so our Internet traffic now goes directly there, and only those 6 machines are still depending on the cross-colo link now.

Many thanks go to Alex Polvi, Joe Rhett (of meer), Myk Melez, and Chase Philips for their excellent work making the upgrade happen!

We’re going to upgrade the firewall in our meer.net colocation facility this evening (Friday) at 6pm PDT (Sat 1am UTC). The upgrade affects many services, including Bugzilla, Bonsai, Tinderbox, LXR, CVS, mozilla.org mail, mozillafoundation.org mail, devmo, the wikis, IRC, and the mailing lists. We expect no more than a few seconds of downtime, but we’ll work for up to an hour to resolve unexpected problems before rolling back to the current firewall, so there may be up to an hour of downtime.

If you have any problems trying to reach any mozilla.org services after the changeover has occured, try first to find us on IRC at irc.mozilla.org #bmo, or if the IRC server is unreachable, mail me at justdave@bugzilla.org (which isn’t handled on mozilla.org’s network).

As Alex points out, the clusters are finally coming. I’ve updated the outage schedule to reflect the first round of services being moved over. Everything in this round should be transparent, it’ll just be DNS changes, so you probably won’t even notice it happening.

We will be moving the ftp-staging server physically from one colo to the other (since it’s the only machine we have with enough disk space) at or shortly after 10:00am on Friday. If all goes well, it’ll be up and running again at the new location within 15 minutes, but we’re allowing a window of about an hour in case of any unforeseen circumstances. This could potentially disrupt automated nightly build uploads from tinderboxes.

You can continue to watch http://nagios.mozilla.org/outages/ for up-to-the-minute information about the moves.