https://support.mozilla.org/en-US/kb/how-to-fix-preferences-wont-save

Note: Some software, like Advanced SystemCare with Surfing Protection, can protect files in the Firefox profile folder against changes. If you have such software then check the settings or uninstall this software.

It is quite normal to see "Use custom settings for history" selected, so it is likely that nothing is wrong. You only need to understand how showing custom settings works. You probably made some changes to the default history settings if you get the custom setting selected. "Use custom settings for history" allows to see the current history settings, but selecting this choice doesn't make changes to the history settings.

The "Use custom settings..." choice will be selected when one or more history settings are not default to make you aware that you made changes. Only when all History settings are default then the custom settings are hidden and you see "Firefox will: (Never) Remember History". "Never Remember History" means that Private Browsing is active and "Always use private browsing mode" has a checkmark.