Localizing Asterisk for China

This last week, we set up a bunch of the IT infrastructure at Mozilla’s new Chinese office. My primary part of the process was setting up their phone system. We used Asterisk, of course, seeing as how we already use that for our phone systems in Mountain View and Toronto. Asterisk has a really cool feature that lets you put localized sound files in for the voice prompts, and each device and incoming phone line can be set up to default to a particular language. It will use the localized files for that language, if available, and fall back on the English ones if they aren’t. You can also allow users to change which language they get with a little careful scripting (“Press 1 for English”, etc). We set it up so that dialing from any phones in the Beijing office will get Chinese prompts, dialing into the Beijing office from outside will get Chinese prompts, and dialing into the Beijing office via the links to our other offices will get English prompts.

One of the difficulties we’ve run into is that there’s no official Chinese language pack for Asterisk, and the only unofficial Chinese language pack we could find is fairly incomplete. You’ll be listening to something in Chinese (like the instructions for voicemail) and suddenly get a word or two of English in the middle of it. 🙂 I ended up spending a fair portion of this week trying to set up a nice friendly web app the folks in the office can use to easily see which files have been localized and which haven’t, and allow them to record their own localized files and have them automatically go where they need to go. Other folks might find it useful, so I’ll try to get it posted somewhere once I get it fixed up a little (it’s a bit of a quick and dirty hack still right now, but it’s getting there).

Vacation, life, hard drives, and basements

So two weeks ago I had scheduled to take a week of vacation from work. My parents’ 40th wedding anniversary was coming up, and my sister and I had planned an open house for almost everyone they knew to come and party that first weekend of the vacation. My parents were then going to take my kids with them up north to spend a week at their cottage on Secord Lake. Swimming, fishing, pontoon rides on the river, all that fun stuff. While we were kidless, my wife and I were going to work on cleaning out all the junk in our basement in preparation for another round at our seemingly never-ending quest to get it remodeled.

Well, the night of the party (but after the guests were long gone), my mom gets a phone call that her father had passed away. We’d kind of been expecting it, as he’d been in and out of the hospital a lot recently, and had been in the hospital and deteriorating for the previous week. So the kids came home with us instead of going up north, while my parents went down to Ohio to be with my grandmother the following day. I made arrangements with my boss to extend my vacation by a few days to buy more time for the original basement cleanup plan.

While at home, the computer in the family room is acting weird (won’t launch one of the games, hanging at weird times, etc). It just felt wrong to me, and stupid me, I rebooted the computer. I was then greeted with the little icon that indicates that it can’t find a disk to boot from. Oops. I managed to mount the disk over Firewire from another computer though, and started rsyncing the contents of the user home directories to the fileserver in the basement. It did not manage to grab everything though, and we lost a lot (the kids bookmarks, my wife’s email, 4 or 5 months’ worth of digital photos). I know, backups. I’d always intended to, and never got around to setting it up. Fortunately for the photos most of them got uploaded to Shutterfly, so that wasn’t a total loss. As of this writing, I now have an automated daily rsync of the /Users directory to the big fileserver downstairs.

But there wasn’t much time to deal with the computer, that would have to come later. Tuesday meant heading to Ohio ourselves for my grandfather’s funeral. The funeral and the family luncheon were quite nice. There were a lot of family there that I hadn’t seen in many years. After the funeral, my parents took the kids, finally off to their trip up north. My grandmother also went with them. Spending a week with the great grandkids on a river/lake system with lots of fun stuff to do… what a great stress relief for her after all that happened the previous several weeks.

As the kids headed up north, Lori and I headed back home, in theory to finally work on cleaning out the basement. But now we’re back to that eMac with the dead hard drive. I made a quick trip to the local Circuit City (I don’t like them either, but they had the best locally-available deal and I was in a hurry) and picked up a new hard drive, 4 times the size of the one it was replacing. Now came the fun part of actually replacing it. The eMac (like many modern Apple products) was not designed with hard drive replacement in mind. Just getting to the hard drive required an almost complete disassembly of the eMac, complete with discharging the CRT to make it safe to work with so I wouldn’t get a 10,000 volt shock touching the wrong part, since the hard drive is right underneath the bare CRT tube. The entire process of disassembly and reassembly took most of a day, between research to make sure I could do it without killing myself and quick trips to the store for required tools I was missing. I then formatted the new drive and spent the next day or two installing Mac OS X 10.4.6 (that’s the install DVD I had), upgrading it to 10.4.10, and reinstalling all the software we use.

Now we’re up through Monday of that vacation that now ends Wednesday. Suffice it to say that I did get a little bit of cleaning done in the basement. But it was nowhere near the goal, and nowhere near being able to allow contractors to touch the place, so I think I’ll be finding myself requesting additional vacation time from work in the near future as soon as I get caught up there again.

Bugzilla @ OSCON 2007

We’re hoping to get some Bugzilla folks hanging out at the Mozilla booth at OSCON this year. Mozilla is blocking off a number of 2 hour slots throughout Wednesday and Thursday where they’ll be advertising specific topics to be discussed at the booth, and we’ll be doing at least one of those, and would like as many Bugzilla folks as possible there during that time slot (probably 3:30 – 5:30pm on Thursday, but yet to be determined). It would also be nice to generally have at least one Bugzilla person there throughout both days (doesn’t have to be the same person the entire time 😉 ). If you’re planning to be at OSCON and are willing to help out with staffing the booth, let me know.

Digging people

So yesterday afternoon, Alex Faaborg blogged about some new features in Firefox 3. No big deal until it got posted on digg.com. The blog server could take it. It’s in the load balancing cluster behind a caching proxy server, which doesn’t even notice this kind of traffic. But Alex had posted his images in his personal space on people.mozilla.com, which is a single server which isn’t really considered production critical on IT’s priority list. Now, even though this is a single server, it’s not exactly sucky hardware. The machine should have been more than capable of handling a slashdotting and getting dugg at the same time. So we were all pretty surprised when it fell over.

Apache kept dying, and spitting out errors about failing to setuid to the apache user. After much banging of heads, Justin Dolske found a relevant forum post in one of the Gentoo forums of all places, which pointed the finger at per-user process limits, and using ulimit in the initscript to override them. Using ulimit turned out not to be necessary, but it did get me looking in the right places.

Mozilla employees get shell accounts on people.mozilla.com (makes it easier for them to manage the webspace there, and several folks use it to run irssi in screen to keep a session to irc.mozilla.org open). In order to keep users from bogging down the machine, we had used pam_limit to limit user logins to 100 processes per user in /etc/security/limit.conf. Well, it turns out that this limit applies to both root and apache as well. So when apache spawned that 100th process to handle that many concurrent connections, it hit that limit and died. Now, root is immune to process limits, however, limits set for root still apply to any setuid processes spawned by root, if that limit is lower than the user being setuid to. So setting a specific (higher) limit for apache in limit.conf wasn’t enough. Had to bump it up for root as well.

But that did the job. The site was back up in no time, happily serving all the images any Digg user could want to go with Alex’s blog, and still keeping an 0.03 load average. Next time someone’s images get posted to Digg or Slashdot, people.mozilla.com will be ready.

Bugzilla download stats

Every so often we run stats for Bugzilla downloads just to see how we’re doing.  The numbers that came up today just blew me away.  I knew Bugzilla was popular, but that popularity seems to be growing exponentially recently.

All-time downloads as of a few minutes ago:

Bugzilla 3.0 (released 09 May 2007) :  10,957

Bugzilla 3.0rc1 (released 26 Feb 2007) : 33,227

Bugzilla 2.23.4 (released 02 Feb 2007 – devel snapshot) :  2,945

Bugzilla 2.22.2 (released 02 Feb 2007 – stable) : 73,897

Bugzilla 2.20.4 (released 02 Feb 2007 – old stable) : 9,835

The numbers for 2.22.2, 3.0rc1, and 3.0 (considering it’s only been out for 6 days) were much higher than I was expecting to see. 🙂

Of course, we all know that downloads != users, but it’s still interesting to look at.