Design line breaks on hover, or button breaks

A common problem I experience is the one below.

enter image description here

  • Everything is in line in the first state.
  • On hover, the vertical line is still in place.
  • On hover, the button space is not equal on both sides which is bad.
  • If adding missing space to the left inside the button, the horizontal line will break.

Conclusion

Leaving it like it is will break the button on hover. Adding space to the button will break the rest of the layout.

A fix could be adding space and a visible background to fix it, but maybe a visible background is not what we want.

Question

Is there a good way to fix this common issue?