Have thunderbird reindex mailboxes instead of compacting
I'm running Thunderbird 115.12.0 64-bit on Ubuntu 22.04. I recently had a hard drive crash. I've had to put together my email data from two separate backups to restore everything. The problem is I reorganized the mailbox folders between the backups, so the metadata thunderbird maintains about the data storage will be wrong in my new setup. I have my imap and pop mailboxes restored correctly now under my profile storage and would like Thunderbird to just leave them alone. The problem is, every time I start thunderbird, it automatically wants to compact my mailboxes and free up 5.7 GB of space, which means it will delete some of the newly copied emails it doesn't recognize.
Is there a way to tell thunderbird to start over and reindex the mailboxes that are there? I've tried deleting the global-messages-db.sqlite file, but that does not fix the problem. This is a common problem as I've had it happen in the past as well when reorganizing the folder structure. It seems there should be a way to tell thunderbird to reset its indexes and leave the mailboxes as-is.
Any help would be appreciated.
Asịsa ahọpụtara
David, thanks for the reply. I know it’s been a while, but I thought I would post what I’ve figured out as to how to reorganize Thunderbird data after, for example, a disk crash with an incomplete backup. Some of this is somewhat detailed, not for the average user. My operating system is Ubuntu 22.04, although much of this should work on Windows or a MAC as well.
- Always have a backup of the Thunderbird data before starting. It is possible to really scramble the mbox data if you’re not careful. - It helps to work offline so that Thunderbird doesn’t try to update things while you are reorganizing. This can be done with File → Offline → Work Offline. It’s best to exit Thunderbird completely when editing the files manually. - Don’t touch anything under the ImapMail folder. Anything changed here will revert as soon as you resynchronize with the server, by design, and you will lose your changes. Any mailboxes that need to be changed under Imap should be first transferred to a folder under the Mail directory, such as something under Local Folders. - Email is stored in mbox files as plain text. The files can be edited if you are careful. There’s no need to edit the .msf index files, as they are created automatically. - If bringing in a new mbox folder from outside the profile, start Thunderbird, right click on an account under which the new mbox file will go, and choose New Subfolder. Enter the name as desired. Then exit Thunderbird. Copy the actual desired mbox file and its associated .msf file, if available, to replace the empty files Thunderbird just created in the profile. - If an .msf file for an mbox is not available, it will be created the next time Thunderbird is started. Right clicking on an mbox folder, choosing Properties, then clicking on Repair Folder will do the same thing, as mentioned by David above. It looks like creating the .msf file and/or updating it leaves the mbox file untouched, which is good. It won’t trash any edits you’ve made until you compact the file or add mail through the gui. If the file is formatted correctly, it should be fine. - I found several mbox files on my system that appeared to be corrupted. There were several emails that appeared toward the end of the list in Thunderbird, when viewing the folder, with garbage date, subject, and/or correspondents entries. I found this on some very old emails from 2000 – 2010, and also found some Yahoo emails from 2019 – 2020 with a similar problem. The problem turned out to be the emails were somehow stored without escaping “From “ at the beginning of lines within message text. The mbox file format requires emails to be separated with a line that has the characters “From “ (including a space at the end) at the beginning of the line. I found several instances where “From “ occurred randomly within email text at the beginning of lines, and it caused Thunderbird to misidentify where to split the emails, resulting in phantom emails with garbage data. I fixed this with a text editor by changing “From “ to “>From “, that is with a greater-than symbol at the beginning, where "From " occured as the first characters on a line. This seems to have fixed these problems without any observed side-effects. A text only editor like vi or wordpad should be used, in order to not add extra, unnecessary formatting. - I found a folder that appeared to have 50 emails when viewed in Thunderbird. I checked the mbox file text with an editor, and the text from 50 separate emails was there. However, when I clicked on repair in Thunderbird, the displayed items dropped to 25 without it changing the mbox file. I found that 25 of the emails had been deleted earlier and were marked for deletion. However, the emails don’t actually get deleted until Thunderbird compacts the mbox file. Emails ready for deletion have a field toward the beginning of the text, “X-Mozilla-Status” with a flag of 0x00000008 (or possibly in combination with other flags). This is the expunged flag. Emails with this flag set are set for deletion. If repairing a folder causes the number of displayed emails to drop in Thunderbird, or Thunderbird says there are fewer emails in a file than there should be, it’s possible this is what happened, so check the X-Mozilla-Status lines.
I hope this information is useful to someone. It took me a while to figure it out.
Gụọ azịza a na nghọta 👍 0All Replies (4)
rightclicking on a folder, selecting properties and then click repair causes thunderbird to reindex the folder.
Asịsa Ahọpụtara
David, thanks for the reply. I know it’s been a while, but I thought I would post what I’ve figured out as to how to reorganize Thunderbird data after, for example, a disk crash with an incomplete backup. Some of this is somewhat detailed, not for the average user. My operating system is Ubuntu 22.04, although much of this should work on Windows or a MAC as well.
- Always have a backup of the Thunderbird data before starting. It is possible to really scramble the mbox data if you’re not careful. - It helps to work offline so that Thunderbird doesn’t try to update things while you are reorganizing. This can be done with File → Offline → Work Offline. It’s best to exit Thunderbird completely when editing the files manually. - Don’t touch anything under the ImapMail folder. Anything changed here will revert as soon as you resynchronize with the server, by design, and you will lose your changes. Any mailboxes that need to be changed under Imap should be first transferred to a folder under the Mail directory, such as something under Local Folders. - Email is stored in mbox files as plain text. The files can be edited if you are careful. There’s no need to edit the .msf index files, as they are created automatically. - If bringing in a new mbox folder from outside the profile, start Thunderbird, right click on an account under which the new mbox file will go, and choose New Subfolder. Enter the name as desired. Then exit Thunderbird. Copy the actual desired mbox file and its associated .msf file, if available, to replace the empty files Thunderbird just created in the profile. - If an .msf file for an mbox is not available, it will be created the next time Thunderbird is started. Right clicking on an mbox folder, choosing Properties, then clicking on Repair Folder will do the same thing, as mentioned by David above. It looks like creating the .msf file and/or updating it leaves the mbox file untouched, which is good. It won’t trash any edits you’ve made until you compact the file or add mail through the gui. If the file is formatted correctly, it should be fine. - I found several mbox files on my system that appeared to be corrupted. There were several emails that appeared toward the end of the list in Thunderbird, when viewing the folder, with garbage date, subject, and/or correspondents entries. I found this on some very old emails from 2000 – 2010, and also found some Yahoo emails from 2019 – 2020 with a similar problem. The problem turned out to be the emails were somehow stored without escaping “From “ at the beginning of lines within message text. The mbox file format requires emails to be separated with a line that has the characters “From “ (including a space at the end) at the beginning of the line. I found several instances where “From “ occurred randomly within email text at the beginning of lines, and it caused Thunderbird to misidentify where to split the emails, resulting in phantom emails with garbage data. I fixed this with a text editor by changing “From “ to “>From “, that is with a greater-than symbol at the beginning, where "From " occured as the first characters on a line. This seems to have fixed these problems without any observed side-effects. A text only editor like vi or wordpad should be used, in order to not add extra, unnecessary formatting. - I found a folder that appeared to have 50 emails when viewed in Thunderbird. I checked the mbox file text with an editor, and the text from 50 separate emails was there. However, when I clicked on repair in Thunderbird, the displayed items dropped to 25 without it changing the mbox file. I found that 25 of the emails had been deleted earlier and were marked for deletion. However, the emails don’t actually get deleted until Thunderbird compacts the mbox file. Emails ready for deletion have a field toward the beginning of the text, “X-Mozilla-Status” with a flag of 0x00000008 (or possibly in combination with other flags). This is the expunged flag. Emails with this flag set are set for deletion. If repairing a folder causes the number of displayed emails to drop in Thunderbird, or Thunderbird says there are fewer emails in a file than there should be, it’s possible this is what happened, so check the X-Mozilla-Status lines.
I hope this information is useful to someone. It took me a while to figure it out.
I have similar problems with multiple corruptions. My Profile had grown (i suspect due to looping re-indexing/repairs and slow copying) to some 600+ GB. I have now restored this to a reasonable size, having manually transferred carious MBOX files out of the Local folders (more later), by - deleting ALL msf files - deleting the global index - using an add-on to remove duplicates - archiving much old data. I have then rebuilt the saved (corrupt) MBOX files using Thunderbird Viewer Software. I now have several MBOX files and have verified the contents (and number of emails) of each file, both using the software to view the mbox contents, and when using this to create individual EML files. The problem is importing the recovered EML and/or MBOX files (using Import/EXPORT TOOLS NG) or manually copying files into Local Folders.
When importing MBOX, for example, only 1 of 28 messages in the mbox are imported, and when copying EML files in, these are corrupted, showing nothing in subject/Correspondence, and rebuilding results in 27 of the 28 vanishing.
Can you assist?
I have no magic tricks, but I would exit thunderbird, and in windows file explorer copy the mbox file to the Mail\Local Folders folder in the profile and then restart thunderbird. I always prefer the fewest number of apps to get things done.