Carousel to display colour variants: should the colour selected always be visible?

I'm working on a product page for an e-commerce website, and I have a carousel component to display different colour variants for the product itself. When you select a colour, the product picture updates accordingly to offer a preview.

My question is: if you have a particular colour selected, and then you use the arrows to navigate the carousel and display other colours, once the selected colour is hidden should the selection change?

Example: I have a carousel with 3 colours displayed at a time. I select the first colour, green. I use the right arrow to see other colours, thus the green moves to the left and I can't see it anymore. Should I change the selection automatically so there's always a correspondence between a visible colour in the carousel and the product picture displayed above? Or is it too frustrating that the selection changes automatically?