Page Load vs Fancy Design

The more complex/fancy looking a website is the greater the page load - we know this.

My question is, does a slow website have more of a detrimental effect to the experience of a user then a hard to use, plain website?

I suspect a balance between the two is the ideal but I have come across a case where in order to make the user experience better the website will suffer slow load times. If anyone could point me in the direction a of study around page load frustrations that would be great.