The emoticons had appeared in facebook chatting for a long time. It may be because you are running a legacy browser before so that no emoticon in textbox was supported. It was unable to disable displaying emoticons from IM chat because it was designed by facebook webpage (not through Firefox) and even if you use scripts to disable emoticons, anyone that you are chatting with will still be able to see your emoticons.

Not a solution, but leaving a space between the colon and the parenthesis seems to prevent smileys in the latest version of FB Messages.  : )