Should wireframes be tested?

I've seen some mentioning in recent times that user testing wireframes is best practice.

Obviously testing as early as possible is something to pursue and a wire frame is a lot easier to make than a fleshed out design with realistic ui and visual elements.

However the argument being presented is not to test early but that even if you have a fully fleshed out design you should strip it all back to a black and white wire frame before testing.

The explanation for this is that it means users don't get distracted by by the ui and will focus purely on the ia and functionality.

However... I'm not so sure this is automatically the best thing to do. It strikes me that if in testing users constantly comment on visuals then it may come time to strip things back but if you have a hi res version available to you then you should at first test with that. Testing something as close to the finished product as possible gives you the most accurate results.

Has there been any decent research into the pros and cons of stripping everything back to pure wireframes?

Is this considered universal best practice now or are there only certain times it should be followed?