That option should turn on automatic private browsing, but perhaps there is a glitch in the new "in tab" options page. Try checking the box for "Always use private browsing mode". Firefox should say that it needs to restart. Does that work?

In case this is a problem with the new "in tab" version of the Options dialog, do you want to try using the old one? Here is how you switch back:

(1) In a new tab, type or paste about:config in the address bar and press Enter/Return. Click the button promising to be careful.

(2) In the search box above the list, type or paste pref and pause while the list is filtered

(3) Double-click the browser.preferences.inContent preference to switch it from true to false

Does that make any difference to whether the checkbox is enabled?

(Note: I don't know whether the old dialog will be in Firefox forever or is a transitional feature. As you've probably noticed, the long-term trend is to remove things that pop up.)

Firefox shows "Use custom settings for history" as an indication that at least one of the history and cookie settings is not the default to make you aware that changes were made. If all History settings are default then the custom settings are hidden and you see "Firefox will: (Never) Remember History". "Never Remember History" means that Private Browsing is active and "Always use private browsing mode" gets a checkmark. "Use custom settings for history" stays selected if at least one of the History or Cookie settings is not the default to make you aware that changes from the default setting have been made.