The application to handle isn't determined by the file extension, it's determined by the server content-type header. If your server is always sending application/octet-stream then by DEFINITION that means that it's an unknown filetype and that Firefox should prompt you.

Most other browsers don't respect this part of the spec but that's what they're supposed to do.

You can read more about it here

Firefox's automatically opening file behavior depends on the site sending the correct mime type with the file. We don't/should not trust the file extension.

Well, the content-type headers don't appear wrong:

Content-Type:application/x-nzb

FYI, it was chunked encoded, other than that, the transaction looked fine.

Any other ideas?

Attachment headers also cause this.

Yep, showing as attachment:

Content-Disposition:attachment; filename="Jamies Dream School S01E01 WS PDTV XviD FTP.nzb"

Any way to work-around this issue, perhaps by site?

I finally found a solution to this problem. Simply install the InlineDisposition extension: https://addons.mozilla.org/en-US/firefox/addon/inlinedisposition/

Sites like Binsearch will now load nzb files into Newsbin (or any other usenet reader) automatically without the user getting the "What should Firefox do with this file?" prompt every time after you've already checked "Do this automatically for files like this from now on"