Category: Work

Every so often, there are security updates for the Linux Kernel. Most security updates you get for Linux don’t require a reboot. Installing a new kernel is one of the few things that does. Well, we have 16 RedHat servers in production, and 11 of them had kernel updates in the queue (the other 5 were just put in production within the last couple days, and had the necessary updates installed already), so I got to spend a few hours tonight rebooting servers. And Myk and Chase got to spend that time at the colo facility watching them reboot in case anything went wrong. They also had other things to do while they were there (there were some tinderboxes being moved, and three of the above-mentioned new machines were actually getting put in tonight). And something did go wrong. Megalon (the CVS server) didn’t come back up after rebooting. After an hour or so, Myk power-cycled it, and it came back up this time. It seemed to have stalled trying to initialize the RAID controller the first time. Ah, what fun.

We have a small bit of data corruption with the Operating System field on bugzilla.mozilla.org currently, which has been around forever, and we really need to get it fixed, because there are upcoming schema changes in the next version of Bugzilla which will get tripped up if it isn’t fixed. Unfortunately, it involves a large number of bugs, so I’m asking for volunteers to help out. More than one person can do this, so don’t hesitate even if someone else has jumped in and said they’ll do it 🙂

The problem with these bugs is that the Operating System field in the database is NULL, that is, it doesn’t actually have an operating system listed. This is illegal in Bugzilla, so Bugzilla will fill in whatever the first available OS in the list is in the popup menu, and the next person to touch the bug will wind up inadvertantly setting it (if they have editbugs), or getting a scary warning that they aren’t allowed to change the OS (if they don’t have editbugs) even though they didn’t actually try to change it. Many of them are likely botched imports from Netscape’s old Bugscape system, others are probably Mac OS 9 bugs that someone forgot to change the OS on before the Mac OS 9 entry was removed from the available OS list a few years ago.

What needs doing is for someone to examine each bug individually, find out which OS really best fits it, and set it correctly, or change it to “All” or “Other” if it doesn’t appear to be something OS-specific. You will need editbugs privs on Bugzilla to be able to fix them.

This query will return the list of affected bugs. At the time of this writing, there are 534 bugs on that list. If you can grab a few of those and fix them, I’d be grateful. You don’t even have to let me know you’re doing it, just grab some 🙂

And thanks in advance to anyone that can help!

UPDATE 1/7 2:00pm: As of sometime around 11:00am this morning (according to my IRC logs and people commenting about it) these are now all gone!!! Thanks to everyone that helped!

Happy New Year to all!

And it’s quite happy for me because as of today I’m now a full-time employee of the Mozilla Foundation.

Since I’m no longer part-time, I’ve removed the donate link from my blog… your money is better spent donating to the tsunami relief funds right now anyway.

One of the really promising new anti-spam technogies which is really starting to gain some ground on the net (lots of people are actually using it now) is SPF. It works by having ISPs create DNS entries which list what servers are allowed to send email using their domain name. Since most spammers forge their headers, this makes it really easy to block spam that claims to be from an ISP that has those entries set up.

One of the minor problems with this is it completely breaks the existing notion of a “mail forwarding mailbox.” A little known fact is that we don’t have an in-house mail system at the Mozilla Foundation yet for people to pick up email from (there isn’t really a need for one yet). When you see someone with an @mozilla.org email address, that mail is redirected somewhere else (usually to that person’s home ISP or their own server) after it gets to mozilla.org. With traditional mail forwarding, the mozilla.org mail server would take that email with the @mozilla.org recipient on it, and just change the recipient address to the final destination address and send it on its way, without touching the sender address.

Now enter the age of SPF. The person sending this hypothetical email from @somedomain.com is using an ISP that has one of those SPF records set up. One of the ISPs that several Mozilla Foundation employees use, just started blocking mail if the SPF records don’t match. They receive the email and it says it’s from @somedomain.com. They look up the SPF record for somedomain.com, and get the list of authorized servers to use that domain. But guess what? It didn’t come from that server, it came from rheet.mozilla.org. Bounce.

They have a solution for that, but the solution is causing yet more trouble for us, and I could use some ideas. Continue reading “SPF and SRS and friends”

I spent a good chunk of today triaging old bugs in the Server Ops component on bugzilla.mozilla.org. I haven’t finished yet, but I closed out a huge number of bugs that were specific to things in the AOL infrastructure that we haven’t been using in the more than a year since the Foundation got created and mozilla.org moved to its own facilities. 🙂 Some of the comments on those old bugs make me really glad we don’t have to deal with AOL’s Information Services folks anymore.