How to better understand and test the impact of change in UX design – case studies or examples?

I'm trying to explain why one of tests is causing a negative reaction when the UX of the page has been improved. My instant thought was that the change we have made is too extreme and our core audience doesn't like it.

In your experience, has anyone come across examples of this behavior? Will be helpful when trying to explain why a test that on the surface looks like it should win, when in-fact it has returned a negative result.