Should I call my button "Unjoin" or "Leave"

I'm creating a web app where people can organise football games, played in the real world (screenshot below to give some context).

I'm struggling to choose a good name for the button, which allows you to leave/stop/unjoin/cancel the current game you have joined. I choose unjoin because it's opposite state is called join. However I've had users complain that they think it's not good english.

What do you guys think? Is there any research on button naming conventions?

Thanks :)