Simplify an interface for flexibly applying rules to periods of time

We have a module in an application where users define time periods for different criteria that will be used later to filter some lists.

By default each criterion applies any time. Its validity can be limited by both restricting the time periods where it applies, and adding exclusion time periods. Already-defined periods of applying and excluding can be edited or deleted.

The current interface is as follow:

^{download bmml source – Wireframes created with Balsamiq Mockups}

Users find it difficult to understand the logic of "application" vs "exclusion" time periods, which I totally understand. I can't find a solution to simplify the interface while keeping all the existing possibilities.

What can be done to improve the user's experience here?