Only button linked or whole item linked?

A site I'm working on has summary panels, with an action button to continue or view more details on that item.

I'd like to make the whole panel clickable not only the button. Are there any negatives to this? Which is preferred?

There are no other actions within the panel nor should there be. I imagine having the whole panel as an action cloud lead to miss clicks or fat fingering on mobile.

The circle in this example is an icon, not a secondary action.