Is an English only aria-label better than no aria-label at all for non-English speakers?

We have a set of buttons that have a single icon for the content and no text content. One example, a button in the top left of a modal that has an 'X' icon which is an image file. To better communicate the button's action to those with using screen reading technology we are looking to add an aria-label attribute, in this case, one that has a value of Close.

Our application supports multiple language, but, for technical reasons, these buttons would not get their aria-labels translated.

So the question: If the aria-labels were English only, and using simple language like 'Close' and 'Go back', would they provide any value to non-English users, most likely be ignored, or cause confusion?

Thanks! And please let me know if you need extra details about the scenario.