Why online chat is so invasive?

I analysed the statistics to remove chat from the e-commerce but the online chat has a lot of conversions. On the other side, it is so invasive and absolute boring for customers that don't need or won't like to use the chat.

What's the best solution to improve the experience for users (most of them) that would not like to use or see chat?

Ps.: At the moment, we are showing on the whole website, we are tracking in which page they use more, the chat are located on the right on the bottom (like the most websites) we are using custom pop-up messages for each page and after the user close the pop-up we will not show again. BUT I'm still feeling invasive, mainly in mobile.