Category: Thunderbird

Posted on

The hardware behind bugzilla.mozilla.org

I recently did up a diagram of how our Bugzilla site was set up, mostly for the benefit of other sysadmins trying to find the various pieces of it.  Several folks expressed interest in sharing it with the community just to show an example of how we were set up.  So I cleaned it up a little, and here it is:

Bugzilla Physical Layout Diagram
Click the image for a full-size (readable) version

At first glance it looks somewhat excessive just for a Bugzilla, but since the Mozilla Project lives and dies by the content of this site, all work pretty much stops if it doesn’t work, so it’s one of our highest-priority sites to keep operating at all times for developer support.  The actual hardware required to run the site at full capacity for the amount of users we get hitting it is a little less than half of what’s shown in the diagram.

We have the entire site set up in two different datacenters (SJC1 is our San Jose datacenter, PHX1 is our Phoenix datacenter).  Thanks to the load balancers taking care of the cross-datacenter connections for the master databases, it’s actually possible to run it from both sites concurrently to split the load.  But because of the amount of traffic Bugzilla does to the master databases, and the latency in connection setup over that distance, it’s a little bit slow from whichever datacenter isn’t currently hosting the master, so we’ve been trying to keep DNS pointed at just one of them to keep it speedy.

This still works great as a hot failover, though, which got tested in action this last Sunday when we had a system board failure on the master database server in Phoenix.  Failing the entire site over to San Jose took only minutes, and the tech from HP showed up to swap the system board 4 hours later.  The fun part was that I had only finished setting up this hot failover setup about a week prior, so the timing couldn’t have been any better for that system board failure.  If it had happened any sooner we might have been down for a long time waiting for the server to get fixed.

When everything is operational, we’re trying to keep it primarily hosted in Phoenix.  As you can see in the diagram, the database servers in Phoenix are using solid-state disks for the database storage. The speed improvement when running large queries that is gained by using these instead of traditional spinning disks is just amazing.  I haven’t done any actual timing to get hard facts on that, but the difference is large enough that you can easily notice it just from using the site.

Posted on

Outdated extensions are depressing sometimes

There’s a Thunderbird extension that I use that still hasn’t been updated since Thunderbird 1.5, that I have a hard time living without.  It’d be awesome if it got updated to work with Thunderbird 2.

It’s Sync On Arrival.  This one makes Thunderbird download all of the new messages in an IMAP folder as soon as it sees notification from the server that new messages have arrived.  There’s two main benefits this gives me. 1) Thunderbird feels a lot faster, because 90% of the time when I go to look at a message, it will have already downloaded it and can show me the cached version, instead of my having to wait for it to download the message from the server when I open it.  2) It’s essentially “always offline-ready”.  Since the messages are always being downloaded when they come in, I can put the laptop to sleep, wake it up somewhere else where I don’t have a network connection, and already have the messages there to look at, without having to have told Thunderbird that I wanted to switch to offline mode before putting the laptop to sleep.  Switching to offline mode then becomes just toggling the state and being done with it instead of having to wait for Thunderbird to download everything before traveling, or even better: having to look at an email at a random time when you weren’t expecting to be offline, and having it already be there instead of being out of luck.

Now the situation isn’t quite as dire as it sounds…  the extension still works with Thunderbird 2, it just doesn’t know that it does.  But having to use the Nightly Tester Tools to override the maxVersion on it every time there’s a Thunderbird update gets annoying.  There’s been plenty of comments in the discussion board for it on the addons site saying it works in 2.0 (and a few that say it doesn’t work, but I haven’t experienced any of the problems they’re reporting).  I’ve also emailed the author about it, and never got a response.  If I knew more about writing extensions and had time to play with it I’d love to fix it, but unfortunately I don’t.  It’d be really awesome if someone did though.

Alternatively, if anyone knows of another extension that does a similar job that also works on Thunderbird 2, let me know in the comments here.

I remember the original ExtendFirefox contest had a category for updates to existing extensions that hadn’t been updated for Firefox 2 yet.  Maybe Thunderbird needs to do something like that to encourage people to update their extensions.