What are the downsides of asking your current users to participate in a study (e.g. for new functionality)?
What I mean is, you rarely see companies put some form of banner out there on their homepage saying:
"Hey, we're developing some new stuff and would like our customer's opinions on it. Take a quick test!"
Usually, they do those tests behind closed veils and with specifically filtered testers (e.g. in remote testing).
So my question is, what is the primary reason for that?
These thoughts come to mind as arguments against it:
- Only specific users would participate and skew the perspective. The average user, who is the main buyer, will ignore it; while the power or frequent users will use the chance to complain or inject their very specific wishes.
- Your users are also accustomed to the site by now and will most probably view any new design changes as "bad" and will vote for keeping what they already know.
On the other hand, these are the things that could be said in favor of it:
- You can get insight into problems that only long-term users might face, while new users do not even know they exist.
- You build more loyalty due to the trust you display for their opinion.
I guess I kind of answered my own question, as it usually depends on context. You use foreign, new users to test things that are supposed to attract new paying customers, while you can ask existing users when you want to improve deeper & more complex functionality.
But it seems you never see the second case, but most always the first one.
Does anyone have further points to add to the two lists or other thoughts related to that?