Why did it take Bugzilla 9 years to get from version 2.0 to 3.0?

Bugzilla 3.0 was released last night!

I was too tired last night to post anything, and I’d be remiss if I didn’t post something today. 🙂 All the good information is on the release announcement on bugzilla.org.

It was almost 9 years ago when Bugzilla 2.0 was released. We made a big deal about that in the release announcement, and since then I’ve seen the question asked “Why did it take 9 years to get from version 2.0 to version 3.0?”

Well, one of the little known secrets (some of the old-timers might remember this) is there was another attempt to create Bugzilla 3.0 several years ago. The original idea was to start over from scratch, and design with a completely object-oriented back-end in mind, with complete separation between the program logic, and the user presentation, allowing many different types of front-ends to potentially talk to the back end. You can see the remnants of this project in LXR. Many people knew of this project at the time, and since Hixie had already taken the version 3 number, we didn’t want to confuse anyone, so the existing codebase continued to evolve adding numbers to the minor version component of 2.x, even though several of the 2.x releases had fairly major new features in them.

Bugzilla has been using the even/odd version numbering scheme, where released versions always have an even minor version number, and development versions always have an odd number. So since 2.0 came out, we’ve had 2.2, 2.4, 2.6, 2.8, 2.10, 2.12, 2.14, 2.16, 2.18, 2.20, and 2.22 using the 2.x versioning scheme. Almost every one of these had major feature improvements. Something else that happened along the way: The vast majority of the design goals from Hixie’s Bugzilla 3 proposal got met along the way, by iterative development of the existing codebase. Bugzilla now uses a templating system for any output going to end-users (be that email or HTML pages, or XML, CSV, RSS, etc). The majority of the back-end code has been refactored into a comprehensive set of Perl modules that do all the dirty work of interacting with the database. The vast majority of the “sloppy code” was rewritten by necessity over the last 5 years in order to make Bugzilla work in Apache’s mod_perl.

So this last summer, some of us realized that Bugzilla really had progressed a long ways, and was certainly deserving of a major version bump. A bunch of us sat down at one of our Bugzilla team meetings and put together a roadmap for what it would take to make us all agree to call it Bugzilla 3.0. By this point, the original Bugzilla 3 project spearheaded by Hixie had long ago died off, and hardly anyone remembered it, so we decided there was no need to worry about confusion (otherwise we would have called this Bugzilla 4.0 instead 🙂 ).

It’s been discussed elsewhere, but very few open source developers ever want to do complete rewrites. Everyone likes scratching their own itch, and usually that’s one or two features they’re missing, and it’s much easier to make changes to existing code than it is to write something completely new from scratch. This little adventure to obtain Bugzilla 3.0 is a good example of that. Hixie’s design for Bugzilla 3 that was posted 6 years ago last week was really pretty good. But very few people joined him to work on it, and the folks that were working on it eventually didn’t have time for it and it died. But most folks understood it was the right way to go, and the Bugzilla 2 codebase eventually evolved into that in little tiny pieces over the course of 5 to 6 years.

Cron job output overload

We have a mailing list at Mozilla which receives mail sent to root at any of our servers. The majority of this mail is cron job output. I have filters set up in my Zimbra account to filter the cron job mail specifically into a folder separate from the rest of the mail to that mailing list. I was on vacation last week, and the last day before I left, I completely deleted the contents of that folder. On my return, that folder contained 26,373 messages in it, all dated within the last week. Trying to separate the nuisance mail from the real problems is kind of impossible by hand with that volume.

Obviously one task is to eliminate the nuisance mail. This has to be done carefully, because typically you still want to get errors from cron jobs, but you don’t want the general output. And not all jobs are good about their use of standard error and standard output, so often you can’t just devnull the standard out and expect to only get mail when there’s a problem. So fixing the nuisance mail sometimes means writing a wrapper script for a cron job that does some grep or awk work to filter the output. But even with the nuisance mail gone, it’s a lot of mail to sift through to find any possible real problems.

So, I filed bug 377043 with an idea for a tool to do some automated analysis of all this cron job output. Keep track of patterns and point out things that need looking at, etc. Unfortunately both cron jobs and data analysis are pretty popular topics (and usually not related to each other) so Google isn’t helping me much trying to search for existing tools. Does anyone know of any existing tools that do something similar to this that we might either be able to use, or build upon?

Vacation and work travel

So this last week I’ve been on vacation, but just hanging out at home hoping to catch up on some things.  One of the projects I’ve been working on this week is trying to write a driver for lirc to use a USB-attached IR receiver on Mac OS X.  One of my MythTV boxes is running on Mac OS X, and it’s annoying to have the little white Apple remote be the only one that works on it (it’s a nice simple remote, but there’s just not enough buttons on it to be useful for a full entertainment center).  I’ve been hoping to get that driver working before I left so my wife could use a real remote while I’m gone.  Not quite there yet, not much time left.  I made major progress on it this afternoon though while the kids were watching the new movies they got in their Easter baskets.

Tomorrow afternoon (Monday) I leave to head out to Mountain View for our quarterly all-hands meeting at Mozilla.  The following week I’ll be attending an Asterisk training program put on by Digium in San Jose (teach me everything I need to know to run the PBX system at Mozilla), so I’ll be away from home for two weeks.  It’s always fun visiting Mozilla, but it’s not going to be fun being away from the family for that long.

CVS Checkin mail

I just checked in a change to the script the Mozilla cvs server uses to send email. It’s one that’s been a long time coming, to get it to use a more reliable way to send the email so that high load conditions won’t prevent the message from delivering. I don’t expect any trouble with it, but if you’re on one of the mailing lists or newsgroups that get new check-in messages, do let me know if you notice anything strange.

Thanks!

Get paid to work on Bugzilla!

We now have two projects approved for participation in the Google Summer of Code this summer.  If you’re a student and you’d like to get paid US$4500 to work on one of them, head on over to Google and sign up.  What better way to spend a summer than improving your favorite bug-tracking system and getting paid for it?

If you have an idea for a project other than those two that you’d like to suggest, you can add it to the brainstorming page.