Should alert boxes be avoided at any cost?

Thanks to Gabriel Svennerberg and Sam K for this one - raised in a comment here.

I was fairly casual about alert boxes until a run of user tests where an alert box was put up to warn learners that if they left an online course at a particular point they would lose work they'd already done. Leaving aside the fact that that situation should probably never arise in the first place (! no poka-yoke there), we were very alarmed by what we saw.

Users basically clicked on that alert box in a completely random way, based on their prior experience of alert boxes in other software.

Only a smallish minority of users read the text of the alert box. Most just clicked something: either the button that kept them safely on the page, or the one that deleted their work, or the 'X' to close the dialogue. Most were bemused by whatever came next. Digging deeper, it seemed that many people just had a set action for 'error' boxes - click something to make it go away.

On that basis, how far do we go to avoid alert boxes?