Is it ok to allow Steppers or incremental buttons to remove items?

I read sometime ago that it is bad UX if the system allows the user to remove an item (for example, form their cart) if he uses numeric steppers down to 0. Should the system allow this or stop at 1 as the minimum amount and have another button to remove the item?

EDIT:

This is a mobile app so it would be within user's expectations that the items should update automatically. Expecting the user to refresh their cart every time doesn't feel that right to me... Also my users are mostly elder and/or sick people.

This stepper function is being implemented in a product carousel in the main page to add items to the cart AND in the cart. Below every item in the carousel there's an "Add to cart" button which after being pressed turns into the incremental steppers. Some users didn't have a problem to add items, set an amount and then going to the cart to make further changes if later on they decided to do so. However someone mentioned that they might have added the item by mistake while viewing the carousel and wanted it removed without having to go to their cart to do so. That raised the question whether changing the quantity to zero should also remove the item. In the cart there's also a "remove item" button but I'm unsure on how to proceed with the carousel without adding more complexity.