There are two prefs that control Tracking Protection:

• privacy.trackingprotection.enabled (false)
• privacy.trackingprotection.pbmode.enabled (true)

It sounds that both are true in your case or you haven't disabled private browsing mode.

You can open the about:config page via the location/address bar. You can accept the warning and click "I'll be careful" to continue.

• http://kb.mozillazine.org/about:config

That's not my point.

I did turn off private browsing.

After I close off private browsing - even after restarting Firefox - privacy.trackingprotection.enabled remains true.

I have tried this on multiple computers.

I know how to use about:config to toggle prefs but I shouldn't have to.

Surely, it should revert to false.

Turning off private browsing mode doesn't affect the two prefs I posted above. The prefs only determine when Tracking Protection is active. By default only privacy.trackingprotection.pbmode.enabled is set to true to enable Tracking Protection in private browsing mode. If privacy.trackingprotection.enabled is set to true then this pref likely shows as user set (bold) and and you have made this change manually and this means that Tracking Protection is active in normal mode.