When to use a filter that shows all by default when no filter is applied?

This has always been strange to me. On most ecommerce websites, they usually have a filter for something like colors. When the user sees the page for the first time, it shows all the colors. But none of the colors are included in the filter.

This has never bothered me as a user, but after thinking a bit more about it, it seems inconsistent with the filter model. From colors to prices, they should really be applied from the first page load. The only problem with this method is that the user is very likely to only want a couple of filters applied, and they would force them to uncheck more options.

Is what I'm talking about even a filter? Is it a different concept or is "show all" simply implied for this sort of a filter?