I say nothing that should surprise diehard Mac users if I say that Mac OS X lacks decent power-user mail clients. Thunderbird suffers from a lot of the same problems as Firefox, Entourage is…well, Entourage, and Mail, pretty though it may be, is heavily underpowered in a lot of key areas. (E.g., you can’t even create nicely formatted lists, which is something I need to do quite frequently.) From this motley crew, I’ve traditionally opted to use Thunderbird. Yes, it’s ugly and doesn’t integrate with OS X, but it’s got the features I want, it runs quite fast, and I get to use the same app on both my Windows and my Mac machines.

Like a lot of people, I had been keeping a few thousand emails in my inbox to be sorted out at some point in the future (usually, “tomorrow”). Some of those were useless spam, and many were just trivial notes I should have deleted but didn’t, but a few were more important to me, like emails from my trip to Germany back in eleventh grade.

This morning, I brought my G5 out of sleep and noticed that Thunderbird had locked during the night. No biggie; force-quit and reopen. As I heated some tea and prepared my breakfast of burnt eggs cereal with spoiled milk instant oatmeal and a grocery list, I sat down to read my email, and noticed that Thunderbird was reporting that I had zero messages. It took my brain a few moments to fully process that Thunderbird was telling me that I had no messages, period. Thankfully, Thunderbird stores its emails in mbox format, and I actually do back my data up fairly regularly using rsync, so I didn’t worry too much, but I was fairly annoyed. The only other app that has ever lost my mail is Microsoft Outlook Express 4 for Mac—not good company to be in.

When I got home today and actually had a chance to look at the problem, though, things were weirder. The file for my inbox was not only fully in-tact, but Thunderbird was dutifully appending new messages to the end of the file. It simply refused to show me what it had downloaded. A few Google queries later, I had identified the problem: Thunderbird’s index file had gotten corrupted. The solution was to delete the index file.

This is dumb in so many ways I don’t even know where to begin.

  1. My biggest complaint here by far is that Thunderbird has all the information it needs to fix this problem itself. It knows how many emails used to be in the account the last time the program ran. It knows how many emails the index file thinks there are now. If these numbers don’t match, rebuild the index.
  2. It turns out that this is a known problem if your inbox gets big and you don’t compact it. What does compacting mean? It means actually deleting on disk what you deleted in Thunderbird. By default, Thunderbird simply marks certain messages as deleted, meaning that they won’t show in the GUI, but doesn’t actually expunge them from the file on disk, since that would require rewriting a large amount of data. Lots of programs use this shortcut for normal use, occasionally running a compacting cycle when the machine’s idle or at least not terribly busy. What’s unique to Thunderbird is that Thunderbird, by default, won’t compact, ever. That would be ludicrously stupid, but at least vaguely pardonable, if Thunderbird worked okay in this configuration, but the truth is that Thunderbird’s indices get corrupt if you don’t compact them often. To even discover this setting, let alone change it, you have to go to Tools:Options:Advanced, where it’s notably under “Disk Space” and not “Settings That Can Crash This App And Destroy Your Emails If You Don’t Change Them.”
  3. Normal end-users are never, ever going to realize that they have any hope of recovering their data. To your normal end user, when the indices get corrupted because all that image-based spam they’re deleting doesn’t actually get deleted and corrupts the Thunderbird database, their email just vanished. Not what I’d expect from an application that bills itself as “a robust and easy to use client”.

At this point, I’m seriously considering reviving my earlier attempt to write a cross-platform email back-end in Squeak (or, these days, maybe Factor) with powerful threading and search, and then writing a nice GUI over it in Cocoa and .NET for normal use. In the meantime, I’m migrating back to Mail. It may be underpowered and it’s IMAP support may be an exercise in frustration, but at least its never deleted my email.