Skip to main content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Do

Planning research helps you identify the team’s goals, methods, etc. for its research. Doing research involves engaging with previously agreed upon knowledge, participants, and data. This article covers tasks agreed to and implied during research planning such as:

  • Reviewing what’s already known
  • Designing your research sessions
  • Running a pilot
  • Corresponding with participants
  • Facilitating research sessions
  • Debriefing

Reviewing what’s already known

Reviewing what’s already known helps your team draw connections and contribute new, original thought. The most fundamental technique for identifying what’s already known (such as relevant business, technology, user, and content considerations) is in-depth interviews with stakeholders. It’s also generally helpful to review the current product or service, including its existing content or training materials. A knowledge inventory session helps capture what your team, including your agency partners, already knows about your research questions. To facilitate this session, you might bring the team together and ask everyone to independently make a mind map of what they know relative to the team’s research questions. Alternatively, you might ask the team to collaboratively create things like a swimlane diagram or a journey map.

Finally, ask your agency partners for any trusted sources, or proprietary information they’ve collected about your area of focus. You might also review:

  • Academic and statistical research — Specialized academic search engines can help you find peer-reviewed publications and articles on your area of focus
  • Internal benchmarking — If your research applies to existing programs, internal benchmarking can help you better understand which metrics matter to stakeholders
  • External benchmarking — Identify any non-profits, journalists, etc. talking about or benchmarking programs related to your area of focus

Designing your research sessions

Research sessions enable you to directly engage people in facilitated conversations about your area of focus. The design of your research sessions will need to account for participant context and encourage the broader team’s involvement.

Describe participant context

Before designing your sessions, consider the context(s) of your target groups. You may be able to rely on your team, including your agency partners, or you may need to ask participants themselves during recruitment. Consider especially:

  • Location(s) and norms
  • Roles in and (expected) understanding of existing business processes
  • Past participation in government-led outreach or educational efforts

For example, if your participants normally work in a sensitive compartmentalized information facility (SCIF) it’s logistically easier to not require the use of screen sharing technology.

Choose location(s) and times

Choose session location(s) and times that are most convenient for your participants. You can run research sessions in person (in labs, meeting rooms, public spaces, etc.) or remotely. Whatever location you choose, make sure it’s accessible so you don’t exclude participants with disabilities.

When choosing session times for lab-based and remote research:

  • Plan for no more than five one-hour sessions a day
  • Allow at least 15 minutes between sessions
  • Allow time for people to eat lunch

For on-site research at your participants’ work or home:

  • Plan for no more than four one-hour sessions a day
  • Allow at least 30 minutes between sessions, plus traveling time between them
  • Allow yourself a break for lunch

Clarify team roles

Design your sessions to encourage as much team involvement as possible without being disruptive.

For sessions where you’ll interact with participants one-at-a-time, such as in-depth interviews and usability tests, clarify which team members will moderate, take notes, and observe:

  • Moderators facilitate the session. Don’t let just anyone moderate; be honest about whether or not moderation will be difficult for someone. For example, the product owner may find it difficult to moderate without introducing interviewer bias
  • Notetakers document, usually word-for-word, what happens during the session. If you’re fortunate enough to have multiple notetakers, you might ask each notetaker to make notes about specific aspects of the session. For example, one notetaker might capture verbatim notes of what the participant says while another might capture what the participant does while interacting with a prototype.
  • Observers pay attention during the session, and ask questions when given time by the moderator

Make time to meet with any team members who were not involved with designing the sessions but will help with research in order to explain the objectives of the research and review the structure of the sessions, session materials, etc. This helps observers contribute to the comfort level of the session and to correctly interpret interactions between moderators and participants. Consider passing along this one-pager about assisting in research to people who are brand new to research.

Preparing a workshop

When conducting small group activities, prepare an agenda and confirm access to any materials you will need, such as sticky notes, printed facilitation aids, etc.

Time moves fast in workshops, so be clear about what you want to accomplish and what you are willing to cut. Make sure to include time for tech checks and have a backup plan. The U.K. Government Digital Service (GDS) has some great additional tips for conducting research in small group workshops.

Preparing a wireframe or prototype

Identify in your research plan if your sessions will require the creation or use of wireframes or prototypes. This helps manage the team’s expectations, and gives the people creating those artifacts time to prepare them.

In evaluative research such as usability testing, prototypes often suggest how users might complete specific tasks. In this case:

  1. Clarify the tasks your sessions will investigate. Tasks are often included in artifacts such as personas or user stories. If your team doesn’t yet have those artifacts, ask them to identify “the most essential things that people need to do” relative to your research area of focus. Then pick the top two or three tasks the prototype will need to depict.
  2. Prepare a scenario for each task. Scenarios help your team create a more believable prototype; they also help moderators prepare research participants during the sessions themselves. If applicable, incorporate your scenarios into the guide you’ll use to moderate the session. Here’s a boilerplate usability test guide (Google Docs version).
  3. Discuss with your team how and when you’ll share the prototype (for example, sharing a link to an online prototype or sending a document).

Prototypes create dependencies in your research. When doing research involving prototypes, you will need to:

Preparing an interview guide

An interview guide keeps sessions consistent, allows multiple people to moderate, and serves as an example for future research efforts.

Most 18F interview guides include these sections:

  • Introductory preamble
    • Remind participants that you are not testing them, and that there are no wrong answers
    • Remind participants that their participation does not entitle them to compensation, and they can decide at any point that they no longer want to participate
  • Consent for note-taking or recording
    • Tell participants how you’ll use the notes
    • Tell participants whether you’ll attribute any quotes, and if so, how (by name or by role, for example)
    • Confirm that participants have signed a consent form; get verbal consent to proceed
  • Priority questions
  • Potential follow-up questions
  • Concluding thanks, questions, and opportunities to follow up
  • Ask who else you should meet with (if appropriate)

For starting points, see this example interview guide (Google Doc version of interview guide).

When interviewing stakeholders, ask about their work as it relates to your research area of focus. Ask about business processes in which they play a role, the information and tools they use, the things they produce or make, and the decisions they’re responsible for. Ask about the people they work with on a regular basis and any challenges they face.

Doing a practice session

Practicing your session can help ensure everyone is on the same page and provide valuable feedback on:

  • The timing of the session
  • The order of the activities
  • The phrasing of your questions
  • The composition of your wireframes or prototypes (if applicable)

A practice session also allows you to confirm team members’ roles and test whether or not your documentation setup will provide the information you need for analysis. Be sure to send team members links to relevant documents (for example, notetaking docs for the note takers and a view-only version of the interview guide for observers) before the pilot.

It’s best to conduct a practice session a few days before your sessions with participants begin, because you will very likely find things you want to change.

Corresponding with participants

Once you’ve designed your sessions, meet with your team to discuss, decide, and agree how participants will learn about, sign up for, and contribute to those sessions.

Inviting participation

Screening is the activity of identifying or locating people who fit your target groups, while recruiting is the activity of establishing an agreement with the people you’ve screened about their participation in the research. Together, screening and recruiting invite people to participate in your research and collect their informed consent.

To invite people to participate, you will need to advertise your sessions in places where you are most likely to reach your desired participants. To do this:

  1. Identify where potential participants are likely to be
  2. Go to those places and get people interested in the research
  3. Identify interested people and determine if they meet your recruitment criteria

Make a list of places where participants are likely to be, both on- and offline. Consider:

  • Relevant community organizations
  • Government-run events
  • Government buildings (Doing pop-up research in GSA buildings)
  • Personal and professional networks, social media, mailing lists, etc.
  • The new or existing website

Next, go to those locations and get people interested in the research. This often involves the creation of an intercept (“research call-to-action”). For example, if you’re doing pop-up research in a GSA building, you might print your intercept on a poster; if you’re doing remote research, you might place a pop-up directly on the website itself.

Note that while it can be tempting to include additional form fields on your intercept to help with screening, this can introduce a delay due to The Paperwork Reduction Act of 1995 (PRA). See Legal for more information. All you really need to collect from people is their contact information, which isn’t subject to PRA (per this memo) if it’s collected on its own. A tweet-length intercept might read:

“Help us improve the design of OPM.gov! Sign up to participate in a remote feedback session by emailing participate-18F@gsa.gov. Learn more about how we’ll protect your information at https://www.gsa.gov/reference/gsa-privacy-program/Privacy-act-statement-for-design-research”

You can find out more about what makes a good tweet-length intercept here in our Legal article.

The third and final step is identifying interested people and determine if they meet your recruitment criteria. Once you’ve got a list of interested people and their contact information, you can easily screen them by having brief, unstructured conversations with them. Once you’ve determined that someone is qualified to participate in the session, you’ll need to get their informed consent.

Incentives

We now have the ability to compensate user testing participants so long as they are public participants, not government employees. This major advancement for research and human-centered design makes it possible for us to get input from more representative groups of people in order to better deliver products and services. Our ability to compensate users for testing depends on both obtaining authorization from the agency partner with the appropriate level of authority and meeting the following criteria:

  1. The benefit to the government in obtaining the feedback outweighs the benefit to the participant (i.e., amount paid to the user)
  2. Compensating user testing participants must directly advance the client’s statutory mission and objectives

The process of paying participants for research is similar to paying for a contractor or vendor. TTS will continue to iterate on this approach in partnership with OGC. We are currently collecting value statements that might be useful in demonstrating eligibility for the “benefit to government” portion of the criteria.

More detailed information can be found in the Compensating research participants document. In order to add compensation for participants to your in-flight engagement or new IAA, you can use the above resource, which includes the User Testing Authorization, Attachment 2 - CoE - document.

Everyone who participates in 18F research must do so willingly and with an informed understanding of what their participation means. It’s your responsibility to inform the people who participate in your research and get a record of them giving their express consent.

Getting informed consent ensures that:

  • Participants understand what they’re signing up for, making your sessions more effective
  • The research is ethically sound (see Ethics)
  • The research complies with The Privacy Act of 1974 (see Privacy)
  • The research complies with The Antideficiency Act (see Legal)

Copy and customize this example design research participant agreement (Google Docs version of template or Google form version) to reflect your research design. This is the agreement you’ll ask participants to sign before they participate.

Correspond with participants ahead of time to explain things that might otherwise get lost in the agreement-signing process. In the event that a participant has not signed an agreement ahead of the interview, obtain their verbal consent. In order to give their informed consent, participants need to understand:

  • Who is doing the research
  • The purpose/goals of the research
  • The duration of the research session
  • What data you’ll collect, and how long it will be kept
  • What will happen during the research, including any reasonably foreseeable risks or discomforts to the participant
  • How you will use the results of the research, and who you’ll share them with
  • That they may discontinue participation at any time without penalty
  • What their rights are and how they can complain

You must also let participants know:

  • Whether the session is being observed, and who’s watching
  • Whether and how the session is being recorded
  • If you will ask participants to share their screen (some organizations lack the technology or may have policies preventing screen sharing).

There are two email templates available to help with participant correspondence:

  1. Template for introducing yourself
    (Google Docs version of email introducing yourself)
  2. Template for passing along a participant agreement
    (Google Docs version of email to accompany participant agreement).

Remember to cover the information listed above. Ask participants if they have any questions. Ensure they don’t feel pressure to participate in ways that make them uncomfortable.

If your plans for keeping research-related records changes later, it’s your responsibility to communicate to participants about any changes in the use of their data.

Scheduling

Aim to help participants arrive at your sessions well prepared, but with an open mind; be especially mindful of your ability to prime participants and introduce bias.

Research participants are doing us a favor, so you should do everything you can to ensure that they have a good experience during their session. Before scheduling them for a session, make sure that you inform participants:

  • How to get to the session itself (if it’s in-person)
  • If they need to prepare anything ahead of time
  • How they will access any session materials (for example, a link to a prototype)
  • If you’ll need them to share their screen (some organizations lack the technology or may have policies preventing screen sharing)
  • If they’ll need to use any software during the session that may be unfamiliar to them (such as Mural)

Also remember to:

  • Make sure calendar invites are clear, both to participants and to your team. For PII protection, it can sometimes make sense to use separate team-facing and participant-facing invites and set appointments to “private” visibility. Google Suite instructions for how to do change privacy settings.
  • Schedule a little more time than you think you’ll need — it can take a few minutes to get everyone on the line, or you may have to work through technical issues. An extra 15 minutes can keep it from feeling rushed.
  • Your calendar’s appointment feature may help you cut down on the email correspondence when scheduling sessions with multiple participants. Google calendar instructions on appointment slots](https://support.google.com/calendar/answer/190998?hl=en). If you use this feature, note that you can’t set the appointments to be private beforehand; you’ll need to change the privacy after participants have signed up for a slot.
  • Confirm with participants that they’ll be able to use conferencing, video, or screen sharing tools and test the setup ahead of your session; create a backup plan if they can’t.
  • Consider including a “tech check” list of tasks directly in the meeting invite asking participants to:
    • Try connecting to the remote meeting
    • Make sure that their video and sound works
    • Download and install any necessary software (eg. Zoom if you are using Zoom for your sessions)
  • If you conduct research sessions via Google Meet and initiate a screen recording, the video will be saved to the Google Drive of the person who created the meeting. A link to the video will be added to the calendar event after the meeting.

Your agency partners can often assist in the scheduling process, particularly when setting up sessions with stakeholders. Don’t hesitate to ask them for help! You can provide them with a tailored version of this email where stakeholder introduces researcher. (Google docs version of email).

Moderating research sessions

Before each session, you should double-check:

  • The date, time, and location of the session
  • How you’ll get to and from the session
  • How you’ll need to dress
  • That you have a signed participant agreement or a plan for obtaining participants’ verbal consent
  • Your primary documentation method
  • That you have a backup documentation method (such as a notepad and a pen)

Moderating a research session can be nuanced, but we encourage all team members to be involved. Make sure participants are comfortable and that facilitators, notetakers, and other team members are prepared. We created a checklist of best practices for interviews. (Google docs version of the checklist). The checklist includes:

  • Keep an eye on the time. Decide beforehand which questions or activities you must cover and which ones you can cut if you run out of time.
  • Be curious. Ask clarifying questions about the stories behind answers.
  • Adapt your script as needed to make the participant comfortable and to be respectful of their time.
  • Allow time for notetakers or observers to ask follow-up questions. Leave time for the interviewee to ask questions of the team as well.
  • If appropriate, ask if they know others who would be good for your team to talk to.

While UX Designers heavily utilize interviews, we also use a variety of other methods. Tips for moderating these can be found in the 18F Method Cards. We’ve also created a breakdown of usability testing quality heuristics. If you’re interested in developing more guides, the research guild is a great place to find collaborators.

Tips for remote research

Because our partner organizations often have national audiences and our teams are distributed, a large portion of the research we conduct is remote. Technology challenges can be frustrating for both the participant and the facilitator. Here are a few things to keep in mind when facilitating remote research:

  • Make sure you’re familiar with the tools ahead of time. Test your call-in line, screen sharing tools, or video conference before your first session. If possible, have your participant(s) test that they’re able to join the video conference tool before the session as well.
  • In the morning on the day of the interview, share with each scheduled participant any links or files they’ll need to participate, and reiterate any logistics their participation will require, such as screen sharing.
  • Allow for buffer time for people to dial in and get comfortable.
  • Introduce everyone on the call and their roles, e.g. lead interviewer, notetaker, observer, etc. This makes it less intimidating for the participant and less surprising if new voices speak up to ask questions.
  • Have notetakers and observers introduce themselves and then mute their audio so the interviewee knows who is on the line, but isn’t overwhelmed.
  • If you are asking participants to share their screen, double-check to make sure they still are able to and are okay with it.
  • If the participant consents to sharing their screen, remind them to close, minimize or hide any windows or information they do not want to share.

Note: Remote-only research can exclude some people with visual or hearing impairments. Try to use a variety of approaches to make sure your research is inclusive.

Tips for in-person research

In-person research happens most often when we go on-site with an agency partner or meet participants in the field. At on-sites we often conduct workshops, but they are a great time to conduct interviews as well. Regardless of the method, here a few things to keep in mind when facilitating in-person research:

  • Try to get as much information about the room(s) you will be in and arrive early. Security at partner buildings can take time.
  • Make sure the room is accessible for all of your participants.
  • Take note of how the environment affects your participants. This may mean limiting the number of team members in the session.
  • Spend less effort on note taking and more effort on connecting with participants. Utilize photography and voice/video recording with permission. If it makes sense, try to involve participants and/or partners in capturing information.
  • Make sure to document your thoughts as a team shortly after the session to fill in any gaps in your notes (see debriefing, below).

Taking notes

Make sure that participants have explicitly consented to you taking notes before you write things down.

The most common type of notes taken by 18F’s UX team are verbatim. Write down everything participants say — to the best of your ability — during each session. The goal is to capture as much as possible during the precious time we have with our participants, and avoid introducing the cognitive biases that come into play when we are selective about what we write.

Taking verbatim notes also curbs the natural tendency to try to understand and analyze what’s being said in the moment. If you’re having trouble writing everything down, focus on capturing what participants say; you or the interviewer can always go back and clarify what questions were asked.

Another style of note taking we commonly use is interaction notes: when we write down all of the actions people take and the reactions they have. For example, capturing a note such as “scrolled to the top of the page, re-read the instructions, scrolled back down to the input field and typed in name” would be sufficient. If conducting usability testing, consider flagging bugs or usability issues.

Note: If there are two notetakers available for a session, consider having one person take verbatim notes, and the other take interaction notes. In this case, it’s best to work in separate documents, as working too close to each other in the same file can be distracting.

Content audits frequently use spreadsheet notes: these are used to track insights and quality of existing content.

Debriefing

Debriefing is a useful way to discuss and capture what everyone took away from a session. It is important to debrief shortly after each session, while the details are still fresh in everyone’s minds. Keep in mind, debriefing is not synthesis — unless you’re doing continuous synthesis, it’s good to keep it descriptive only and move assessment until later in the research process.

Here are a few high-level things to capture in a debrief:

  • How did this session answer our research questions?
  • What were the main themes of this session?
  • Which questions or activities did the participant(s) have a lot (or very little) to say about?
  • What new questions do we have?
  • Are there any new people we should talk to?

You may want to use this example interview debrief worksheet; feel free to modify it to your team’s needs. When determining which team members to include, default to a more diverse group. If a team member wasn’t part of the session, their role can be focused on asking questions and documenting.

Additional reading

U.S. General Services Administration Logo 18F Logo

18F UX Guide is a product of GSA’s Technology Transformation Services, and is managed by 18F.

Looking for U.S. government information and services?

Visit USA.gov