Firefox should not be opening New Windows unless you tell it to. Or are meaning tabs ? As there are Extensions that can be used to limit the number of tabs that can be opened. https://addons.mozilla.org/firefox/extensions/

Hmm, what do you mean by instance?

Normally if you were to use your Firefox desktop shortcut when Firefox is already running, Firefox would open a new tab or new window in the same instance of Firefox -- meaning it is in the same session and the two tabs or windows will share history, cookies, etc.

If firefox is already running, and I click its icon in the startmenu, or on the desktop, I want it to focus on the already open instance, not open a new window, as it currently does.

Unfortunately that is not what the icon is for at this time. Have you checked out what happens when you right click it when it is running ?