It sounds like this is an issue with the Cisco Anywhere Connect VPN client. If you do use that software, then you will need to stop it before installing Firefox 4.

According to SiliconChaos Blog: Firefox 4: libsmime3.dylib in use error during Mac OS X install you can accomplish that by:

1. Open a terminal
2. Type sudo killall vpnagentd
3. Press enter

That should successfully stop the software and you may then proceed to install Firefox 4.

If you are not using that software, you might also try removing the existing installation of Firefox (by dragging it to Trash) and then installing. Your profile is stored in a separate location, so it will not be removed.

I am not using that software and when I trashed Firefox and emptied the trash I get "The operation can’t be completed because the item “libsmime3.dylib” is in use."

Trying again. After trashing Firefox and getting it gone from the trash, I get "An older item named “Firefox” already exists in this location. Do you want to replace it with the newer one you’re moving?" and the trash is empty. Then when I say OK I still get the above message. And now I have no Firefox.

I have the same problem trying to update Firefox from 4 to 5.0.1. I contacted AppleCare and was told that I should absolutely NOT go to the terminal because it is dangerous! I was advised to contact Firefox online support. So here I am and I don't know what I can do about this and whether I will ever be able to update Firefox!