You can only do this if you use "Clear history when Firefox closes" to clear the Download history. Otherwise you need to clear this folder manually.

Firefox manages downloads in the Downloads folder in the Library (History > Show All History)

• Downloads are treated as history items: clearing history removes included download items and vice verse.

Note that Firefox may still be saving the downloads in the classic Download Manager that stores its data in the downloads.sqlite file (the new Download Manager uses places.sqlite).

I remember being able to Not keep the download history without having to clear the browsing history. Guess I'll be looking for a new browser. This is becoming too much like Facebook. To do one thing, you're forced to do another.