The following indicators can help determine if a usability test will produce useful results. This is not an exhaustive list, but it should be helpful in planning and assessing usability tests — additions welcome!

Indicator Good sign Bad sign
Study design
Study purpose Researcher clearly articulates the purpose of the usability test (for example, as a specific question to answer, an area of inquiry, etc.). Researcher does not specify a purpose for the test, or the purpose specified is very broad, like “testing the app” or “finding problems.”
Number of participants Tests include a sufficient number of participants necessary to see patterns, and there’s only one participant per session. Studies includes only one participant, or studies include multiple participants tested simultaneously (like a focus group 😱 ).
Sampling bias Participants are people who actually use the application (or do the task the application is supposed to support). Participants are experts (vs. average users), friends, family, or close colleagues of the product team.
Moderator style
Question style Researcher asks follow up questions that are open-ended and non-leading. Questions are leading or subtly suggest potential solutions or hypotheses.
Task variability Researcher asks participants to complete similar tasks related to the study’s purpose. Researcher asks participants to complete different tasks, or tasks unrelated to the study’s purpose.
Framing Researcher asks participants to complete tasks with the product or service that align with their work-related goals. Researcher asks participants to complete tasks unrelated to their work-related goals. For example, asking a participant how they might send a fax when their job doesn’t call for that.
Priming Moderator asks participants to complete tasks without indicating how that might be done. For example, “How would you view the status of your application?” Researcher guides participants in completing tasks. For example, “Which of the links in the header would you click to login?”
Team participation
Number of teammates The team designates a single moderator for the test, and at least one member of the product team observes the usability test. A single person from the product team participates in and leads the test.
Observer role Observers do not moderate. They are generally quiet, and ask open-ended questions after the test has concluded. Observers interrupt the participant, or attempt to sell or explain the product. Observers debate the participant’s actions in their presence.
Sensemaking
Notetaking Tests are recorded or notes are taken for sharing with absent stakeholders. Tests are not recorded, or test results are not documented.
Synthesis Moderator debriefs with teammates immediately after each interview. Researcher looks for patterns across multiple participants and surfaces problems that affected several people. Moderator reports the most memorable problems without conducting affinity mapping or some other analysis technique.
Presentation of findings Researcher reports findings to team and stakeholders in an easy to follow, well prioritized way. Researcher presents team a “basket of issues” or an unprioritized laundry list of potential changes.
Incorporation of findings Product team translates findings into future user stories or product refinements. Researcher reports do not affect the product backlog or ongoing development work.