Empty input fields and functional design

Let's say my application has two input fields. Here is the situation:

• When starting the application, both fields are empty.
• When entering a value into one of the fields, it will be parsed to a number (Only after the value changes!).
• If the parse succeeds for both fields, a sum will be displayed.
• If the parse does not succeed, the field will change to display a red border, the sum will disappear. This also includes empty values.

The problem: At the beginning, the empty fields do not indicate an error, but when entering and then deleting something (The field returning to the old state) an error will be indicated.

My question: When should an error be displayed and how should an initial and a non-inital empty field be treated by the application?