How to Move Toward a Dedicated Usability Testing Budget

How do you know that you or your team are doing your best work? One way is to examine how you do the work.

Development teams may hire third-party auditors to examine their compliance with maturity models, or they may run retrospective meetings after each sprint to assess their work less formally. The lessons learned would help them avoid making the same mistakes repeatedly. As a result, they would correct their course in future releases, which would free them up to produce higher-quality products.

Metrics also exist for measuring your product’s user experience. Similarly, several leading UX practitioners have formulated maturity models, which describe how well organizations truly care about their users.

One of the most well-known UX maturity models comes from Jakob Nielsen. Organizations progress through his model’s 8 stages in sequence. Once teams have started to care about users, the most major hurdles they need to clear involve money. First, they need to start spending money on usability studies. Second, they need to build a group of UX professionals in their company. The latest stages of the model deal with expanding this UX group’s influence.

Usability studies involve putting your product in front of its current or potential users in order to find out how you can improve it. They can occur in a wide range of formats, such as the following:

Card sorting and tree testing exercises measure the effectiveness of navigation within your product.
Five-second tests measure users’ first impressions of a product and their recollections from seeing it briefly.
Click tests measure where people would click first, and preference tests let them help designers choose between several design concepts.
Think-aloud studies instruct users to speak their thoughts out loud as they complete a series of tasks in a product.

Traditionally, human factors professionals performed usability studies on site in dedicated lab space. Lab studies’ cost and overhead provided a high barrier of entry for companies who wanted to embrace usability testing for the first time. In the late 2000s, usability testing panel companies, such as UserTesting, began making significant headway on remote usability studies.

Benefits of Usability Studies

Even if you only run a usability study on your live product, the studies allow you to both watch users’ behavior and find out their problems in their own words. This can show you how you can improve your product’s user experience, user interface, and even its copy to keep users in your product and productive.

But starting here requires you to spend significant time and effort building a live product. You can actually start usability studies before you have written any code. Usability studies also help you catch design flaws in interactive prototypes, wireframes, and paper prototypes. For new products, you can run usability studies even before you start design; use competitors’ products to hear about their weaknesses from your potential users.

Usability studies also have more hidden advantages for your team:

More accurate bug reports: You don’t have to make assumptions about bug repro steps or get customers on the phone or live chat in order to figure out what is happening. Test videos are faster and more accurate.
Another source of expert input: You can outsource usability studies and usability evaluations to specialists. Whole professional organizations focus on this.
A corrected focus: Studies help you see how many people actually run into an edge case that your development team is probably spending a lot of time discussing.
Development cost savings: It saves you money on development because it can keep you from going down the wrong path.
Support cost savings: It saves you money on support because it can keep you from shipping features or interfaces that your users don’t understand.
Competitive intelligence: It gives you a structured way to hear from your users about competing products that they might use instead of yours.

Common Objections to Usability Studies

The first financial hurdle in Nielsen’s UX maturity model involves spending money on studies. Why would an organization avoid a method that can have such a strong ROI? Here are some common reasons:

Cost: Third-party panels can charge up to the low hundreds of dollars per participant, and they may require enterprise clients to sign up for very high-priced plans. Even moderated studies with free screen-sharing tools can require expensive participant incentives. However, studies still cost much less than trying to fix an already-built, faulty design.
Time and overhead: Even organizations who hire outside help to run a study will still need to invest some time on their side to ensure the best possible results from it. But they are spending significant developer time anyway – possibly on building products that will not convert their target audiences. In effect, their team may just be looking busy.
Competitors or confidentiality: This objection might have the most teeth to it because of legal reasons. But there are ways to protect your idea in testing that should make your lawyers happy in most cases.
Rework: Occasionally, usability studies can identify key features as useless or a product’s key assumptions as flawed. Throwing away extensive work on an already-built product or an interactive prototype requires swallowing pride. This objection is valid. But shipping a product that confuses users or does not convert may result in a redesign or other extensive rework later.
Fear of criticism: Software developers are known as highly logical thinkers. But that doesn’t stop you from fear about whether or not you’re producing good work. If anything, it might lead you into hyper-mindfulness of failure and a pervasive fear of it. Not allowing criticism of your ideas from your users now is just a stall tactic that sets you up for a potentially huge failure later – whether you’ve changed jobs by then or not.

Navigating Budget Objections

Teams may face resistance to usability studies for any of the reasons above. However, the real reason may be described as “You can’t have the budget for this.”

Instead of taking it at face value, ask yourself these questions:

Does our company really have no money to do a study, or are they resisting doing the study for this price?
What is the minimum level of involvement from me for a study to give me the feedback I need? Can someone else in my organization or a contractor manage the overhead?
Will the legal department (and/or upper management) be happy if we have our participants sign an NDA and record uninstalling our product at the end of their test session? Or could we recruit participants who are in our company but not on our team, just so that we can hear from people who do not have our own team’s deep knowledge of our product?
How can I make a business case for fixing a bad design before that bad design ships and the cost of rework becomes prohibitive?
How can I make my team more aware of what users really think about our product – both positive and negative? How can we get better at identifying root causes behind users’ complaints?

A valuable rule can help usability studies to start gaining traction in your organization, and even some major usability testing panels have set themselves up to let you take advantage of it.

The Company Credit Card Rule

At a business conference I attended in 2016, one speaker gave a talk on going from being a consultant to being a client. He explained that many freelance designers and developers approach potential clients expecting to land tens of thousands of dollars’ worth of work immediately.

Meanwhile, the clients need to stay within their budgets, and they need to make sure that their products ship successfully. Moreover, their projects need to produce a solid ROI. Clients who fail to do this can lose their jobs.

So, naturally, consultants have to go through many approvals before they can work with a big organization on a large project. They might even require approvals from C-level executives, which can take a lot of time and effort. Clients who do not already know, like, and trust a consultant are unlikely to go through these approval processes.

In contrast, complicated approvals rarely make as much sense for smaller projects and products. Clients can become more familiar with consultants by buying their e-books, SaaS subscriptions, and other less expensive products. And these can be impulse buys, using a company credit card. Hence, the speaker called this the Company Credit Card Rule.

So, too, with a usability study. Find out if your manager or your client has a budget limit, above which they have to go through an approval cycle before they can spend the money. If there is a number like that, all you have to do is get the usability study to cost less and have this person approve the time you want to spend on that study.

Lower-Cost Studies

An outsourced, lab-based usability study might cost U.S. $12,000-18,000, and moderating studies in a usability lab yourself would take significant time. Balancing time and financial cost, remote unmoderated studies from panel sites are your most ideal option for your first usability studies.

Not every panel site requires larger businesses to purchase expensive enterprise plans. UserTesting currently offers introductory pricing for individuals; WhatUsersDo and TryMyUI offers free trials. Lower prices get people used to the idea of running usability studies before they commit to making them a regular part of their process.

What if the introductory prices are still expensive enough to require approvals?

Lower-cost panels have launched, allowing more organizations to benefit from usability studies and allowing testers to participate in more studies.

Feedback Army, which launched in 2008, was one of the first to do this. They used a panel of Amazon Mechanical Turk workers to review websites and apps. Feedback Army shut down in 2016, but there are several newer alternatives. They also gave instructions on how to set up a usability testing Human Intelligence Task (HIT) yourself.
UserBob uses a prescreened panel of Amazon Mechanical Turk workers, who have already completed a sample test, to test websites and mobile apps.
UserInput.io is another Mechanical Turk-based panel which lets their customers ask users up to 7 questions about their website or business idea.
Usability Hub allows people to complete a 5-second test, a click test (“where would you click first?”), a preference test on a range of design ideas, a navigation test, or a question test (survey about a design). Their pricing works in terms of credits per respondent.

“Zero-budget” Usability Testing

Panel sites give part of the cost of their studies to their participants. This not only pays them for their time but also encourages them to prioritize completing the studies.

Even if you moderate studies yourself instead of ordering them from a panel, plan to offer an incentive for participation. Studies without incentives tend to attract only your most engaged fans, who might not represent your whole user base accurately.

Because people will only complete a study for free when they feel like it, expect such studies to both take longer, result in more administrative overhead, and still have fewer participants than they need.

Using Your Own Participants

Recruiting on websites like Craigslist typically does require incentives. If you cannot give incentives at all, you can ask your family and friends to test your product. Or you could use your company’s newsletter or an email to a list of your product’s users to get participants.

If you can give a minimal incentive, you can ask random strangers at a coffee shop or a restaurant to test your product. You will need to buy them coffee and be clear (and accurate) about the duration of the study.

Incentives do not always need to be in cash, but they should reflect the time and value that you expect from a participant. For example, you can offer a discount on a participant’s most recent or next purchase, a free membership, a free temporary upgrade to their existing membership, or a gift card that you already have.

Letting Your Colleagues Participate

Internally in larger companies, you can ask colleagues to participate in your study. Choose co-workers who are not on your product team, because members of your team would know more about your product than any of your end users would.

Similarly, Steve Krug identified a great source of test participants for internal products: new hires. They have enough domain knowledge to use your product but low familiarity with internal products. Since not everyone who will use your product works for your organization now, making your product easy for new hires to use will eventually increase its adoption.

Recruiting in the field

You can go to where potential users are and recruit them to spend some time walking through using your product.

If you have an app for city tourists, find tourists in that city.

If you have an education technology product, spend some time in a teachers’ break room.

Wherever you go, make sure it is legal for you to be there. Ask for permission from the owners. Many malls and other public spaces have rules against solicitation.

Recruiting at Trade Shows

If your company exhibits at trade shows, you might be able to run tests at your company’s booth – with the permission of your salespeople.

Note: So that they can stay in business, trade shows commonly prohibit two activities. Outboarding is hosting competing events for a trade show’s attendees during official trade show time, without the trade show organizer’s consent. Suitcasing is selling your product on a trade show’s grounds without buying exhibit space. Trade show organizers see both of these activities as theft, and they may ban violators from their future events. They may interpret usability studies as an attempt to sell your product – and therefore suitcasing.

Usability testing at trade shows is not technically zero-budget. Trade shows typically require attendees to pay, and they charge exhibitors significantly more. And exhibiting at major trade shows requires months of preparation.

Therefore, if your company is not already exhibiting at a trade show, the cost and risk make running studies there much more expensive than ordering a panel study.

Moving Toward a Dedicated Usability Testing Budget

If you avail yourself of free trials or introductory pricing, you will only be able to run a few usability studies at most. Similarly, low-cost panels may provide you some valuable insight, but you will likely find them limiting.

For these reasons, Nielsen’s maturity model emphasizes managing your organization’s UX effort. This includes running studies regularly and keeping an archive of your findings to track your progress, which helps justify to the rest of your organization that the studies are valuable. But that requires a dedicated budget.

Why Dedicate a Budget to Usability Studies?

There are some caveats to running usability studies with a zero budget:

Logistics: Assume the studies will take longer. You will need time away from your other work to get the studies done. (In that case, the study would cost your company however many days of your time. It wouldn’t actually be “free”. Still, explaining that trade-off will probably mean spending more time at your desk and no time doing the study.)
Task quality: If you do not have budget to hire outside help and do not have a UX specialist on staff, you might not know the best practices for setting up an effective study. While you can still find out usability problems in your product, a specialist will help you gain the kinds of insight you need from your study.
Feedback quality: There’s no fair value exchange when people are giving you feedback without getting anything from you in return. You’re less likely to get good feedback or as much feedback.
No-shows: No incentive means that people are less likely to make showing up for their session a high priority. User research professionals generally recommend overbooking your study by several spots to account for no-shows, but you’ll need to do this even more with no budget.
Skewed samples: Testing with your family, friends, or most enthusiastic users skews the sample. You would be hearing from people who will probably not want to offend you, so they might not say anything bad about your product. But if you have to go this route, you can use their social networks to your advantage. Ask your friends and family if they know other people who could participate.
Less (or no) screening: You may lack access to the participants you really need – and instead have to settle for any willing participant. In contrast, panel sites, recruitment services, and lab studies often allow you to set up screeners so that you can learn how your product needs to improve in order to attract and retain its intended audience. Higher-end panels provide more effective ways to screen out “professional testers”.