Firefox 3.0.10 doesn't have a Private Browsing mode, that was added to Firefox in version 3.5.

It's not version 3.0.10. Like the question states it's Firefox 4

You have a corrupted user agent the identifies you as Firefox/3.0.10

• Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.10; .NET CLR 3.0; ffco7) Gecko/2009042316 Firefox/3.0.10

See also:

• Web sites or add-ons incorrectly report incompatible browser

• You enter Private Browsing mode if you select: Tools > Options > Privacy > History: Firefox will: "Never Remember History"
• To see all History and Cookie settings, choose: Tools > Options > Privacy, choose the setting Firefox will: Use custom settings for history
• Uncheck: [ ] "Automatically start Firefox in a private browsing session"

See also:

• How to fix preferences that won't save

Start Firefox in Diagnose Firefox issues using Troubleshoot Mode to check if one of the add-ons is causing the problem (switch to the DEFAULT theme: Tools > Add-ons > Themes).

• Don't make any changes on the Safe mode start window.

See:

• Troubleshoot extensions, themes and hardware acceleration issues to solve common Firefox problems

Actually it seems to be something with the FireGestures. I had uninstalled FF and deleted the preferences and such. So I had a fresh copy of the browser. It still did the same thing. Disabled FireGestures and it stopped doing it.

Now I need to find a good mouse gesture addon

Actually this wasn't solved. It's not really Private Browsing as much as history just stops being added. It's not an add-on, it's something with FireFox 4. It seems that https triggers it.