Validate on keypress after an incorrect submit attempt?

I understand that validating on keypress is not recommended because it's annoying to get errors thrown in your face when you haven't finished filling out the field yet.

However, what about the scenario where you make a mistake, move to the next form field, return to the form field with the mistake, correct the mistake, and haven't yet changed focus away from the input field? Doesn't it make sense to remove the error message? Ie. doesn't it make sense to validate on keypress after an incorrect attempt to fill in a form field?

As a concrete example, consider:

After I correct the error by entering my name, doesn't it make sense to remove the error message, even though I haven't moved focus away from the form field yet?