Bulk migration of thousands of mails from Thunderbird POP to IMAP (GMAIL) and Server Time Outs
Hey folks, I recently started the process of migrating my accounts from POP to IMAP. I have several accounts set up on Thunderbird which have been getting mail via POP for years. The idea is to migrate all these accounts to IMAP and I've started the process on one of the smaller accounts, but have already ran into issues with server time outs due to the quantity of mails I'm trying to copy over.
I set up both the POP and IMAP account for the same address on Thunderbird and manually set up the folders (or labels on GMAIL) on the IMAP account to replicate the set up of the pop account. Setting the folder structure manually is not a problem, the problem I've run into however, is that when copying any large amount of mails from the POP account to the IMAP account, it just doesn't seem reliable.
The progress displays in the status bar, but on transferring a folder with about 2,000 emails, it seemed to time out and not transfer everything over. I then had to go through a long process of comparing emails in each folder to know what was left to transfer.
I'm yet to transfer much larger accounts, some with tens of thousands of emails and I need to be sure everything gets transferred properly.
Can anyone suggest a tool or add on or method that even if the server times out, that it can just continue where it left off and I can be sure that all the mails will transfer eventually? I understand the upload process can be slow and I don't mind having to wait or doing it overnight, just as long as it's done right and I'm not left having to compare thousands of accounts, folders and mails.
Any help would be greatly appreciated!
All Replies (11)
OK, why are you doing this migration? I ask because there may be a better way to get to where you want to be. Moving large quantities of data via email protocols is never easy nor reliable.
Is the mail still on the POP server? If so, you can transfer it to Gmail using built-in Gmail tools. If the mail is only on your local computer, some quick research indicates there are a few programs that upload mbox files (TB stores mail folders in mbox format), but I cannot attest to their usability or reliability.
Thanks for the replies folks.
I'm doing the migration to switch from POP3 (where local storage is essential and changes aren't reflected on the server) to IMAP so I can access my mail from anywhere and all changes such as sent items, deleted items etc will be reflected everywhere.
The mail is not still on the pop server and actually the pop mailbox is quite small, much smaller than the total I've downloaded over the years from it.
I actually managed to transfer a few mails over today without an issue and deleted one of the POP accounts on Thunderbird, but a lot of the mails were small so I'm thinking I might still run into issues with any big moves.
Would love to get some recommendations on anything that's been tried and tested as I'm not so technically minded and if I run into any issues, could be in deep trouble :o
I've been trying to copy mails manually since and they mostly seem to be transferring ok, but I do get several time outs, so then I just need to check where the transfer stopped by the last mail.
There's one big problem though and when I check from one mail in 2007 to another mail in 2011, there were about 6,400 mails between the two, but then in the folder I put them into in Gmail, when I check the amount of mails between the same two emails, there's only about 6,100 and I haven't a clue where these 300 are going.
The same happened transferring another folder before too :(
I've done that or helped someone else do it years ago... But I don't remember how.
Still, the e-mail was strictly ordered in folders.
Maybe if you do so, you do less manual work by dragging and dropping folders or the content of folders instead of trying to Move Inbox with 7000 e-mails you move Inbox
2012 2013 2014-1 (first semester) 2014-2 (second semester)
Hey, thanks a lot for the reply!
So you mean, instead of trying to make sense of a folder with 7,000 emails in it, to first split the POP folder into several sub folders and then create the same folders on the IMAP account and just select the whole lot to move and then compare later to see what went and what didn't? Then, once I'm happy I have everything on IMAP, I can put them all into the one folder on IMAP?
Sort of.
You may be able to drag and drop the folder instead of the messages.
I'm don't know the IMAP protocol. But I wouldn't trust (or force) a sync protocol to have to deal with 7000 (headers) every time an update is made (a new message comes in/out is moved, deleted, etc)
Having everything split up in folders makes some folders "permanent" and they will not have to be synced. Keep your inbox, sent, trash, junk small, and all other folders as ordered as possible.
If there is an issue with a file/folder/message you may loose all or most of the 7000 e-mails. If each folder is ~500 messages and something happens you don't loose more than 500.
Ok, thanks a lot for your help. I might try that so and see if it works any better :)
I was doing some more searching on this last night and found this program which seems to have some good reports and reviews: https://www.ghacks.net/2011/01/07/google-webmail-filer-for-thunderbird/
From the description it seems to be what I'm looking for, but I'm a bit apprehensive to use it as I'm not sure how it will work given I'm half way through all transfers already so I'll probably just continue to do it manually, but maybe useful for anyone else reading this.
The developers website seems to be down though so I'm not sure if it's still safe to use or what and the discussion linked to above is over 3 years old.
The program is still available for download here: http://www.softpedia.com/get/Internet/E-mail/Mail-Utilities/Google-webMail-Filer-for-Thunderbird.shtml
You are using Google. They limit your bandwidth use on a daily basis. to 500mb. IMAP uploads. That is not GB it is Megabytes. Depending on the size of attachments that may well be less than 500 emails a day.
To quote from the Google link below Don't perform migrations or bulk operations using IMAP Avoid large copy/paste or drag/drop operations when using IMAP.
Thanks a lot for the reply! That's very interesting and I wish I knew that before I started the migration, but I'm almost done doing it manually. The main problem with the timeouts seemed to be larger attachments.
What I ended up doing was sorting mails by size and I would send a few thousand smaller emails at a time and they usually went through no problem, once or twice it failed (one time I tried sending 15,000 in one go another time about 5,000 I think).
So if you're patient and just send a few thousand at a time and then send less as the mails get bigger, you should be ok?
Instead of copying them, I moved them, if the transfer didn't complete properly, all the mails, even the ones that sent ok, would remain in the sending folder. If the transfer went ok, all mails would be removed from the sending folder, so if you left your pc for a while and came back and the transferring stopped but there were still the mails you sent in the folder you were sending them from, you would know not all of them went, so you had to try again, maybe this time with less mails.
I also think gmail keeps only one copy, so if you send 500 and it fails after 400, and you send 500 again and it succeeds, you will have only 500 in gmail, not 900 with 400 of these being duplicates. I didn't investigate this fully, but I think it works this way.
The limits in your link do seem pretty restrictive on a mass migration. I can't be 100% sure, but there's a chance I may have exceeded them and didn't run into any problems sending more or having my account suspended?
It's a very long process but I'm almost through it. It seems all the reply statuses, notes I had with mails using the xnotes addon and everything else seems to still be intact with the mails on imap as they were on POP.
I just hope I don't discover down the line that loads have been corrupted or missing or anything, but it's looking good so far!