I'm not sure I understand what's happening. Let's say you just started Firefox, or you open a new window using Ctrl+n. If you start opening new tabs using Ctrl+t or the new tab button, after some number of tabs Firefox pops up a new window instead of just squeezing them into that window?

If the problem is related to links, there could be special reasons for the new window opening. For example, scripts can specify particular window dimensions or open the window as a "dialog" (dialogs can only have one tab). Some of Firefox's behavior here can be modified with settings.

If the problem is related to bookmarks, hmmm, not sure why that would be a problem.

If the problem arises using the Windows 7 Task Bar, please describe the steps to re-create the issue.

Do you mean that additional windows open in some cases?

You can try to keep open only the (one) window that works properly (if necessary open a new window) and close and restart Firefox to see if that makes new windows work properly as well.
If you are in a window that is opened via JavaScript as a pop-up window then such a window will behave differently.
This can also happen if a new tab is opened in the current window via JavaScript and the code tries to disable some features in that window (tab).
You may need to clear the cache to reset currently stored data and possibly delete the sessionstore.js file in the Firefox profile folder.

• http://kb.mozillazine.org/Prevent_websites_from_disabling_new_window_features