Is a repetitive, three seconds response time the absolute worst?

My very first IT manager back in the 90s once stated that the absolute worst repetitive response time an application can have is three seconds. His argument was that this is long enough to be a significant annoyance, and too short to get a (cognitive) break. So if you want to design a system for maximum frustration, make every action have an average response time of three seconds (and allow for some variation too, just to make it unpredictable as well).

This was my managers anecdotal input, but I have always thought of it as valid. And it seems to make sense given Jakob Nielsen's thoughts on the matter.

Is there any research to back up (or invalidate) the claim?