Is your back button missing from the main toolbar? By default it appears immediately to the left of the address.

By the way, another shortcut is to hold down the Alt key and press the left arrow key.

Open you history and look for what you want.

Now that I read your question again, is the problem that there are extra entries on this history menu when you right-click the back button? Some sites create extra entries by redirecting or update the address bar. However, I haven't tried that site to see whether that might be what it's doing.

You can middle-click a link to open it in a new tab.
Then you can close the tab (e.g. with a middle-click on the tab or Ctrl+W) to return to the tab with the Drudge website.