Skip to main content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Bias

All research is subject to bias, whether in our choice of who participates, which pieces of information we collect, or how we interpret what we’ve collected. Proactively engaging with bias helps us improve the credibility of our research. The following list of biases is our starting point.

Research design bias

What it is: When the team doesn’t acknowledge bias or designs their research to advance their existing beliefs. For example, if an agency executive believes that they already understand user needs, that executive might discourage the team from speaking directly to users—why learn what the executive already knows?

How to counter:

  • Be mindful of the kinds of bias that can affect design research
  • Surface the team’s assumptions
  • Clearly identify the team’s research objectives (which should center around people’s needs)
  • Note the research’s inherent limitations

Sampling bias

What it is: When some members of the target population are less likely to be included in the study. For example, if a team leans too heavily on digital-first participant recruiting processes, it risks excluding members of the public who don’t interact with government online.

How to counter:

  • Discuss research participant recruitment strategies
  • Clarify the target population, which can involve clarifying the difference between stakeholders (usually public servants) and users (usually the public)
  • Look for ways to encourage diversity or representativeness in the sample by asking, “Who haven’t we talked to yet?” For example, people who access this service via screen reader or people who have limited access to technology are groups to consider
  • Document the plausible shortcomings of your participant recruitment strategy
  • Be careful of the conclusions drawn from any one study

Interviewer bias

What it is: When the interviewer’s own beliefs or assumptions influence how they lead a session. This can be especially apparent at the start of an interview — for example, if the interviewer expresses excitement about a particular aspect of the product or service (“Our team is really proud of the new search feature!”).

How to counter:

  • Be mindful of your ability to prime participants
  • Refrain from asking close-ended questions
  • Practice interviewing beforehand
  • Periodically echo what you’ve heard during the interview back to interviewees (“Just to be sure I heard you correctly, you said…”)
  • Conduct post-interview debriefs

Social desirability bias

What it is: The tendency for people to respond in ways that paint themselves in the best possible light, regardless of whether or not that bears any resemblance to reality.

How to counter:

  • Build rapport with participants
  • Emphasize the goals of the research, and how honest feedback is the best way to meet those goals
  • Distance yourself from any proposed design solutions (“these are just a few ideas the team came up with, but you’re the expert here,” “I didn’t design this, so if you don’t like it, you won’t hurt my feelings”)
  • Consider changing the research mode. For example, some research has shown that social desirability bias may be less likely when interviews are conducted over email compared to face-to-face

Confirmation bias

What it is: When you (or your team) interpret research in a way that conforms with your own beliefs or values.

How to counter:

  • Bring attention to the team’s shared values and beliefs—for example, by conducting a hopes and fears exercise before the research begins
  • Emphasize diversity when recruiting research participants
  • Invite the team to observe research in action, and hold post-interview debriefs
  • Collaboratively analyze data before synthesizing it, and note any plausible alternative interpretations
  • Consider different perspectives that challenge your beliefs
  • Use a variety of research methods when collecting data to triangulate your findings

The observer effect (the Hawthorne effect)

What it is: When the people who participate in research modify their behavior simply because they’re being observed. An example is when an office becomes unusually quiet while an interviewer conducts on-site interviews.

How to counter:

  • Build rapport
  • Ask for introductions from key stakeholders, and be mindful of the ability to prime participants while informing their consent
  • Blend in by noticing and following participants’ social and cultural norms
  • Pay attention to what people say they do, as well as what they actually do (you might ask participants to teach you their process)
  • Use mixed methods as well as unmoderated research modes, like monitoring forum posts or web analytics (while being mindful of privacy)
  • Be careful to avoid over-interpreting what you see or hear

Avoiding bias

In general, you can avoid bias and arrive at better solutions by intentionally including people throughout the design process. Help your team see research as a team activity, and understand why it’s better to talk to a few users throughout the design process than none at all (as Erika Hall says, “The most expensive [usability testing] of all is the kind your customers do for you after launch by way of customer service.”).

Bias is a starting point for improving the team’s research practice—everyone benefits when we share a commitment to asking better questions.

U.S. General Services Administration Logo 18F Logo

18F UX Guide is a product of GSA’s Technology Transformation Services, and is managed by 18F.

Looking for U.S. government information and services?

Visit USA.gov