Undo-Redo for sections not visible to the user

In a multi-step task-based UI, imagine the flow below -

  1. User adds a task and a panel opens up to fill in the task details
  2. User fills the details and saves the task
  3. User closes this add-on panel
  4. User performs an undo for some reason
  5. The undo operation is applied to the closed task panel as that was the last action

However, this does not seem right as the action taken by undo isn't in the user's view. Thoughts?