Tools > Options > Advanced > General tab, click Config Editor there. Copy-paste mail.tabs.drawInTitlebar into the search bar and double-click on it to set it to "false" and the title bar should be controlled by the OS again.

Wow! That's a very well-hidden setting. May I suggest that before the final release it could be made a little more accessible?

I sure agree that the setting isn't easily discoverable, but that's too late for the 38.0 release given that string freezes went in effect already (the reason being that localizers have to translate all UI elements for their respective language).

The respective bug report has been up for more than two years, but thus far not much progress in terms of a solution acceptable by the developers. I don't know if the extension proposed in the last comment was ever developed and posted.

Well, all right. But there will probably soon be a lot more people asking this question. Hopefully the solution will be in a future release not too far away.

It appears that this "feature" and the respective default have been in there since Thunderbird 17.0 already, thus you shouldn't have seen the problem using an existing profile with 38.0; however, if you invoke it with a new profile for testing (as recommended for pre-release versions), the default kicks in.