Delete value in a modal

I have a page that contains a section that has a button for the user to assign a contact. This button will bring up a modal that has a field where the user can input the contact's name.

Adding a contact is not required, but in this modal, the field becomes required because we don't want the user submitting empty values from the modal.

But once the user has assigned a contact, what would be the best way to go about delete it?

In the mockup below, once the user hits delete, the assigned value will be delete, and the modal will disappear.

We decided that for now, no deletion confirmation is needed.

Another idea is to add a delete button from the main page (inline with the saved value), but that might be too easy a mistake to make.