Should a delete action’s "undo" be underneath the "delete"?

Take for example this button:

                 _________
                /         \
               |   Trash   |
                \_________/

Which when clicked, becomes this:

   _________     _________
  /         \   /         \
 |  Restore  | |  Destroy  |
  \_________/   \_________/

Which means that when clicking on the delete button, your mouse is already on the confirm button, which allows a quick double click to delete. This is good for deleting many items quickly. But a naive user might accidentally double click on the button, resulting in a permanent delete rather than a prompt to confirm.

The action I'm implementing is a Trash / Restore / Destroy pattern, where you have the option to either restore or destory an item that has been trashed.

What's better - protecting the user from an accidental destroy or allow faster deletes by having the confirm to destory action "underneath" the trash button?