Best way to handle browser "back" functionality for a voting webapp

The context is a single page webapp with a section that allows users to vote on content. The votable content is loaded via ajax.

The webapp itself implements the back button using the HTML5 history api. But since it is preferable to not have people vote multiple times on the same item, the back functionality is not implemented for the content voting part. To explain better, clicking the back button while on a voting page will not take the user back to the previous votable content, but rather to whatever other page (like contact/submit/profile/etc.) the user was on before coming to the voting part.

Does this seem reasonable? Is there a better way of going about this?

Edit: clarification with example flow.

Suppose the user views the webapp's pages in the following sequence: Home > About > Contact us > Voting page (item 1) > Voting page (item 2) > Voting page (item 3)

Now when the user clicks on Back while they're on Voting page (item 3) will take the user back to the Contact us page, and not to Voting page (item 2).

My dilemma is that whether the back button should take back to the last votable item (Voting page (item 2)), or to the last item before the user got on to the voting screen (Contact us).