Search Support

Avoid support scams. We will never ask you to call or text a phone number or share personal information. Please report suspicious activity using the “Report Abuse” option.

Learn More

Duplicate emails being downloaded

  • 4 cavab
  • 2 have this problem
  • 1 view
  • Last reply by Matt

more options

I'm using Tbird 68.12.1 with Windows 10. I have several email accounts, all POP. I've noticed occasionally that I have collections of duplicate emails in some local folders and I've never known what caused them. But today I saw it happening. I'd been doing a bit of exploration/housekeeping/deleting on Gmail webmail (which I rarely look at and never use) and a few minutes later, hundreds of duplicate emails flooded into my Gmail Inbox in Tbird. I though Gmail was the culprit, but then realised that the same thing was happening in another Inbox from a different ISP.

Some of the duplicates are copies of those already in one of the Inboxes, but most are copies of those filed away in local folders. I've seen several contributors asking about similar occurrences, but have not yet seen a solution to what is happening on my computer.

I'd be really grateful if someone could explain:

  • why this is happening; and
  • how I can prevent it happening in the future;
  • whether there is an easy way of identifying and deleting all the duplicates (short of locating and selecting them all individually).
I'm using Tbird 68.12.1 with Windows 10. I have several email accounts, all POP. I've noticed occasionally that I have collections of duplicate emails in some local folders and I've never known what caused them. But today I saw it happening. I'd been doing a bit of exploration/housekeeping/deleting on Gmail webmail (which I rarely look at and never use) and a few minutes later, hundreds of duplicate emails flooded into my Gmail Inbox in Tbird. I though Gmail was the culprit, but then realised that the same thing was happening in another Inbox from a different ISP. Some of the duplicates are copies of those already in one of the Inboxes, but most are copies of those filed away in local folders. I've seen several contributors asking about similar occurrences, but have not yet seen a solution to what is happening on my computer. I'd be really grateful if someone could explain: * why this is happening; and * how I can prevent it happening in the future; * whether there is an easy way of identifying and deleting all the duplicates (short of locating and selecting them all individually).

All Replies (4)

more options

The best anyone can do is guess. V68 has been out of support for years now and basically will probably stop working with gmail soon.

The control of POP mail is somewhat complex as most folk leave mail on the server, and gmail in particular will not delete it when asked to unless you have already modified their default settings on the web site, the same for hotmail/MSN/Outlook. This is further complicated by folk having anti virus products that purport to scan mail. Generally they do this by placing themselves in the middle of the data stream and pretend to be Thunderbird to the server, and the server to Thunderbird. In hacking circles this is called a man in the middle hack. If you care to look it up. The result is the same, if the AV coughs or dies completely (relatively common when the mail is malformed spam unfortunately) the issue is reported as a mail issue not defective security software.

Then we come to the second issue. Because mail is left on the server (something the original mail planners never thought server would have the space for) Thunderbird has to maintain a file with the what is happening status for all messages downloaded, but not removed from the server. Complicated by the server ignoring deletion requests, so not all mail on the server is actually managed in the file. Thunderbird expects the server to do what it is told. However Thunderbird's store of statuses is only as good as the mail ID it uses. It gets this from the se4rver initially, but certain server changes change the id, with the result the message is not in the file any longer, at least not recognizably so. It is also only as good as it's ability to update the file used with the newest information. This is all good on a system where file access is local and not inhibited by other things, but unfortunately these days those systems are rare. Especially those running older operating systems. Things like streaming backups, the use of cloud synchronized storage locations, NAS storage, limited disk space and anti virus scanning of changed files can see updates to these files fail silently in the background.

Generally a look at the actual profile on disk will show many popstate.dat files with numbers appended to the names in this circumstance. But while the data is saved, it is not accessed, so those mails get downloaded again.

Fundamentally what I am trying to say is anything that gets between Thunderbird and it's profile and Thunderbird and the server is going to be an issue. Which of those things is your problem is anyone's guess.

For your last point there is this addon https://addons.thunderbird.net/en-us/thunderbird/addon/removedupes/

more options

Thank you very much, Matt -- I saw somewhere else that popstate.dat files could be a problem, but have not been sure where to find them. I'll study your reply, look at the popstate.dat files and get back to you.

In the meantime, however, my problem is getting worse. Current emails are still being downloaded to my Samsung phone, but no current emails to my Win 10 computer for about 24 hours. But old emails (currently June 2022) are still being downloaded to the computer, initially in quite large batches, but now in very small batches of about 2 per time -- but only when I click on 'Get Messages'. Does any of that ring any bells with you?

I stuck with v. 68 because, at that time, the later versions did not have the add-ons that I wanted to continue using. But I'll look through the add-ons and upgrade as soon as I can do that. Is there any particular recent stable version that people recommend?

more options

POP mail is downloaded from the earliest to the latest dates. That is how it works. It sounds like something may have upset things and the entire account is being redownloaded. I would guess anti virus, but only because they have traditionally been the cause of about 90% of getting mail issues on windows. They are updated sometime many times a day and get it all wrong far to frequently for supposed security software.

If you have email scanning enabled disable it. This will at least make the epic redownload go faster. Probably twice as fast or better, it might also remove the get mail need.

If your account is Google, there is a setting to force only download of pop mail from today going forward on this page https://mail.google.com/mail/u/0/#settings/fwdandpop

Other providers have similar options, although Outlook actually move the mail to a pop folder so redownload is not a real issue there. Yahoo, well they just tinker round the corners and sink further into obscurity.

more options

Oh, I feel obligated to recommend the newest version 102. Like all new versions it has had some issues, but the release of a point 1 version sees most of them fixed. Like windows and Microsoft generally used to be, you skip the version updates and buy in on the point one update (In Thunderbird's case a month or two latter). If you allow automatic updates, the update is generally not offered until the point release or even the second if bug that are found on new installs warrant delay.

Note add-ons will only get better going forward. Thunderbird core on which Firefox is also built was changed by Mozilla to not allow old style add-ons. V68 was the first real pinch that came from that change with 100% of add-ons needing update to work with the new version. V78 again had similar requirements. Since then the pool of add-ons that are compatible is again filling out. Prior to V68, it was a case of add-ons might work for a decade without attention, so many of the add-ons people used were just not supported by the folk that had originally written them many years before.

There has been a change in the way add-ons are managed on the web site as well which makes it easier for a new person to take over maintaining an add-on. A result of the changes however is that sometimes your old add-on is not updated, a new one is released. For instance the import export tools add-on was one of the top ten prior to V68. https://addons.thunderbird.net/en-US/thunderbird/addon/importexporttools/?src=ss The author withdrew all of his add-ons in protest, but a new add-on importexporttoolsNG was released. https://addons.thunderbird.net/en-US/thunderbird/addon/importexporttools-ng/ so checking for newer versions is a bit problematic and the site search is your friend if you do not see an update to the exact add-on.