Category: Work

Posted on

stage.mozilla.org moving *finally*

Migrating stage.mozilla.org to a new host has been in the works for a long time. It’s been stalled for a couple months while we worked with the developers of the unionfs filesystem (mostly by providing them with crash reports) to stabilize it to the point where we felt comfortable putting it to use on a production service. That time has finally come. The alternative to unionfs (keeping 9 terrabytes of disk space online at once, completely dedicated to the FTP staging process) was really not cost effective, and it was really worth waiting for.

This Thursday, March 13th, we’ll be switching the DNS for the stage.mozilla.org domain name to point at the new box. The SSH host keys have been copied over, so most people will never notice. The existing box, effective immediately, is accessible using the name stage-old.mozilla.org. If for any reason you’re nervous about switching your upload process to the new box this coming Thursday, you can use the stage-old name to keep using the old one, for now. Stage-old will go away on Tuesday, March 25th, so you’ll have until then to resolve any issues you have (or get IT to resolve them if you think it’s our problem). Feel free to file a bug against Server Ops if you need help.

Be sure to read my previous post for details of how the new system will work (it is a little different, though most well-behaved upload scripts shouldn’t be affected). The most noticeable change will be that there will now be a slight delay between when you upload something and when it shows up on the ftp server, since we’re now virus-scanning what you upload before it gets made available on the ftp server.  On the current system that we’re moving away from, the virus scan runs after the files are placed on the ftp server, and then the files are yanked afterwards if they get flagged. This of course, could let something get out there briefly, which we obviously don’t want.

Posted on

Trying to find that long lost tab

I’m very much a power user.  I use my web browser constantly, for both work and play, and get extensive use out of tabbed browsing.  I keep web pages that are related to the same task in tabs in the same window, and open a new window when I’m shifting gears to work on another task.  And I often go back and forth between tasks as things come up or I need a break from the routine, or whatever, so eventually I wind up with a situation like right now where I have 12 windows open, and about half of them have 5 or 10 tabs in them.

Now I’m looking for a specific tab, and it was sort of a one-off thing, and I don’t remember which window it’s in.  And it’s not the frontmost tab in the window it’s in, so I can’t just look in the Window menu to find it.

Now I’m thinking it would be really cool if the Window menu had submenus for each window that had multiple tabs in it, which listed the tabs in that window.  Then I could just mouse over the windows in the Window menu and glance through the submenus looking for it.  Bug 405933 filed.

Posted on

Outdated extensions are depressing sometimes

There’s a Thunderbird extension that I use that still hasn’t been updated since Thunderbird 1.5, that I have a hard time living without.  It’d be awesome if it got updated to work with Thunderbird 2.

It’s Sync On Arrival.  This one makes Thunderbird download all of the new messages in an IMAP folder as soon as it sees notification from the server that new messages have arrived.  There’s two main benefits this gives me. 1) Thunderbird feels a lot faster, because 90% of the time when I go to look at a message, it will have already downloaded it and can show me the cached version, instead of my having to wait for it to download the message from the server when I open it.  2) It’s essentially “always offline-ready”.  Since the messages are always being downloaded when they come in, I can put the laptop to sleep, wake it up somewhere else where I don’t have a network connection, and already have the messages there to look at, without having to have told Thunderbird that I wanted to switch to offline mode before putting the laptop to sleep.  Switching to offline mode then becomes just toggling the state and being done with it instead of having to wait for Thunderbird to download everything before traveling, or even better: having to look at an email at a random time when you weren’t expecting to be offline, and having it already be there instead of being out of luck.

Now the situation isn’t quite as dire as it sounds…  the extension still works with Thunderbird 2, it just doesn’t know that it does.  But having to use the Nightly Tester Tools to override the maxVersion on it every time there’s a Thunderbird update gets annoying.  There’s been plenty of comments in the discussion board for it on the addons site saying it works in 2.0 (and a few that say it doesn’t work, but I haven’t experienced any of the problems they’re reporting).  I’ve also emailed the author about it, and never got a response.  If I knew more about writing extensions and had time to play with it I’d love to fix it, but unfortunately I don’t.  It’d be really awesome if someone did though.

Alternatively, if anyone knows of another extension that does a similar job that also works on Thunderbird 2, let me know in the comments here.

I remember the original ExtendFirefox contest had a category for updates to existing extensions that hadn’t been updated for Firefox 2 yet.  Maybe Thunderbird needs to do something like that to encourage people to update their extensions.

Posted on

Free SIP on Leopard exists after all!

So it’s been a few weeks since Mac OS X 10.5 (Leopard) came out.  One of the major things that immediately hit a lot of people was that every known third-party SIP client stopped working.  I work from home.  I have an extension number at work that rings on a Polycom phone in my home office, thanks to the magic of VoIP.  This same VoIP technology (namely SIP) has allowed me (up until I upgraded to Leopard) to run a “soft phone” program on my laptop to allow me to connect to the same phone system when I was out of my office.  With the help of a set of headphones, a laptop actually makes a halfway-decent phone.

The two free (as in beer, not freedom) products previously available that I knew of were SJPhone and X-Lite.  Both of these broke on Leopard.  SJPhone hasn’t been updated in years, and the material on their website makes it look like the Mac version was an afterthought anyway, so I don’t hold high hopes for them ever updating it.  X-Lite is a pared down version of a commercial product called eyeBeam.  eyeBeam just got updated for Leopard this last week.  An X-Lite update is expected “by the end of the year”.  The obvious reason for the delay is to encourage people who are frustrated enough to throw money at it to upgrade to eyeBeam instead of waiting. 🙂

There *are* two free products that have been updated for Leopard which do SIP to a generic PBX of your choosing. Those are SightSpeed and Gizmo Project.  Unfortunately, both of these require you to register with their service, and sign in on their service, and your generic-PBX-of-your-choosing account is a secondary login (if you don’t log into their service, your generic one won’t connect either).

The world is in really dire need right now of a good open source solution for SIP on the Mac.   If any of the above programs were open source, I would bet we would have had patches posted somewhere within days (if not hours) to make them work on Leopard.

UPDATE: I was going to post this hoping to get some discussion going and/or someone to point out something that works that I missed.   But before I could post it, I found one!  XMeeting not only works on Leopard (it apparently didn’t break — there hasn’t been a release since July), but it also supports DTMF (touch tones) during the call, which is the one thing that’s been missing from all the other free stuff I’ve tried so far.  Touch tones during a call are pretty important for things like entering the password for a conference call.  XMeeting supports video, too (and so does Mozilla’s phone system).

Posted on

stage.m.o, shift-reloaded

Just like ftp.m.o, shift-reloaded, stage.mozilla.org is getting an overhaul as well. If you read through preed’s post that I just linked, you’ll see the plans for stage included in there as well. For various reasons (most of them including LDAP and testing issues) they ended up not happening at the same time, and we ended up doing only the ftp/archive part during that last outage window. We’re finally ready to proceed with the stage.m.o half of this, and the best part is for this chunk of the puzzle, we don’t even need an outage window. We’ll be running the new stage along side of the old one for a little while to let people try it out, make sure their upload scripts still work, and bring issues to our attention before the old one goes away.

The items listed on preed’s earlier post that still have to happen are:

  • All files will be virus scanned before becoming available. We currently virus scan all builds, but depending on a number of factors, it was possible for unscanned-builds to appear on the FTP site for a window of time; we’ve removed this window.
  • Interactive shell accounts on the FTP farm will be replaced with sftp-only accounts.

Non-interactive Accounts For Uploaders

One of the changes we’re making is that shell accounts are going away. The new server will have you chrooted into the staging area, and you will be limited to scp, sftp, rsync, and a small subset of file management commands (such as mv, cp, chmod, chown, chgrp, etc) invoked via ssh. This is the reason that if you have any scripted uploads, you need to test them on the new server to make sure they still work, and whether you can modify them so they will if they don’t. If you are doing something on stage currently that really needs full shell access, we’ll probably be happy to accomodate you, just on some other machine. Come talk to us or file a bug.

We are also moving from local accounts on the staging server to LDAP-based accounts, to make management of permissions easier. In a few cases, this might mean your username will change (in most cases, the affected people have been contacted already). Accounts that haven’t been used in the last year will not been ported over. If you haven’t connected to stage in the last year you’ll need to file a bug in the FTP: Staging component to get your access back. Accounts will be getting enabled over the course of the next 24 hours. If you want to get in sooner than that, come find me on irc and I’ll manually toggle your account. The new machine is located at stage-new.mozilla.org.

Virus Scanning

The ftp file tree was almost a terabyte in size before. Now, with it combined with archive.m.o, it’s 2.2 terabytes. Keeping just one copy of that sitting around is decently expensive. Keeping multiple copies of it live is just cost prohibitive. On the current staging system, for lack of disk space, there’s no way to prevent the files from going to the mirrors before getting virus scanned. The virus scanner would come along and scan the newly-uploaded files and then yank any with viruses found back out. So there is a small window of time when something with a virus in it could make it to the mirrors and then would disappear off the mirrors again the next time they synced.

With the new staging system, we’re making use of a new (to Linux — BSD has apparently had it for years) filesystem technology called unionfs which allows us to layer the filesystem. If it helps to visualize it, think of it like a multi-layer photo in Photoshop, where each layer is transparent by default, and each pixel in the photo is a file on the filesystem. When we make changes to a file, we’re only really changing the pixel in that position in the topmost layer (that pixel is now no longer transparent). From here on out, this post gets a little technical, to explain how it’s set up for those that are curious. If you’re not the technical type, you can feel free to stop reading here. There’s nothing beyond this point that will affect your ability to upload if you have an account.

UnionFS example AThe main ftp tree (which is NFS-mounted from a huge disk array) gets mounted read-only as the “base layer” of a unionfs mount. We then put another layer on top of that which contains the chroot jail environment (the executables and libraries needed for jailed users to have minimal functionality while connected, but don’t need to be visible to the mirrors). There is another read-only layer on top of that (Layer B on the left) which is used for the virus scanning (more on that in a moment). The top layer (Layer A in the diagram) is the one that the end-users actually write to when they make changes to files. This last layer records all of the additions, deletions, and modifications to files that the users make so that those files appear changed to the unified filesystem view that the users see, but the underlying filesystem that the mirrors rsync off of isn’t touched.

UnionFS example BThe layers can be reordered and changed from read-write to read-only at any time, so when it comes time to push the files to the mirrors, we move Layer B to the top and make it read/write, change Layer A to read-only. This way, nobody can make changes to it while we’re scanning it. We then virus-scan layer A directly. Once the scan completes, and any infected files have been removed, we then move the changes from this layer down to the real live ftp/archive layer at the bottom which the mirrors can see when they rsync. Layer A is then cleared off so it’s ready to be swapped with Layer B again on the next pass.