There might be a corresponding extension, which is the kind of add-on that usually affects your toolbars.

orange Firefox button (or Tools menu) > Add-ons > Extensions category

Can you find a logical candidate there and disable it?

Jscher, no; there are no extensions either (its so new I havent added anything). The Add-ons manager, the internet options and even the simplepass options do not have options for me to remove the icon.

Firefox stores toolbar customizations in a file that sometimes becomes corrupted. You might try renaming that file so that Firefox returns to the default settings for toolbar layout. Here's how:

(1) Open your personal settings (AKA Firefox profile) folder

Help (or Alt+h) > Troubleshooting Information > "Show Folder" button

(2) Switch back to Firefox and exit the program (e.g., File > Exit)

(3) Wait a few moments for Firefox to finish updating files in your settings folder, then rename localstore.rdf to something else, like localstore-bad.rdf or localstore.old. Alternately, you can delete the file.

(4) You're done with your profile folder and you can restart Firefox now.

Does HP manage to reinsert itself?

Yes. I've checked to make sure both firefox and HP are updated. I have a simple touch on my work PC and that does not have the same issue

I solved it by following the steps provided by @jscher2000 and then removing C:\Program Files (x86)\Mozilla Firefox\distribution\bundles\websitelogon@truesuite.com