Planning research helps you identify the team’s goals, methods, etc. for its research. Doing research involves engaging with previously agreed upon knowledge, participants, and data. This article covers tasks agreed to and implied during research planning such as:
- Reviewing what’s already known
- Designing your research sessions
- Running a pilot
- Corresponding with participants
- Facilitating research sessions
Reviewing what’s already known
Reviewing what’s already known helps your team draw connections and contribute new, original thought. The most fundamental technique for identifying what’s already known (such as relevant business, technology, user, and content considerations) is in-depth interviews with stakeholders. It’s also generally helpful to review the current product or service, including its existing content or training materials. A knowledge inventory session helps capture what your team, including your agency partners, already knows about your research questions. To facilitate this session, you might bring the team together and ask everyone to independently make a mind map of what they know relative to the team’s research questions. Alternatively, you might ask the team to collaboratively create things like a swimlane diagram or a journey map.
Finally, ask your agency partners for any trusted sources, or proprietary information they’ve collected about your area of focus. You might also review:
Designing your research sessions
Research sessions enable you to directly engage people in facilitated conversations about your area of focus. The design of your research sessions will need to account for participant context and encourage the broader team’s involvement.
Describe participant context
Before designing your sessions, consider the context(s) of your target groups. You may be able to rely on your team, including your agency partners, or you may need to ask participants themselves during recruitment. Consider especially:
- Location(s) and norms
- Roles in and (expected) understanding of existing business processes
- Past participation in government-led outreach or educational efforts
For example, if your participants normally work in a sensitive compartmentalized information facility (SCIF) it’s logistically easier to not require the use of screen sharing technology.
Choose location(s) and times
Choose session location(s) and times that are most convenient for your participants. You can run research sessions in person (in labs, meeting rooms, public spaces, etc.) or remotely. Whatever location you choose, make sure it’s accessible so you don’t exclude participants with disabilities.
When choosing session times for lab-based and remote research:
- Plan for no more than five one-hour sessions a day
- Allow at least 15 minutes between sessions
- Allow time for people to eat lunch
For on-site research at your participants’ work or home:
- Plan for no more than four one-hour sessions a day
- Allow at least 30 minutes between sessions, plus traveling time between them
- Allow yourself a break for lunch
Clarify team roles
Design your sessions to encourage as much team involvement as possible without being disruptive.
For sessions where you’ll interact with participants one-at-a-time, such as in-depth interviews and usability tests, clarify which team members will moderate, take notes, and observe:
Make time to meet with any team members who were not involved with designing the sessions but will help with research in order to explain the objectives of the research and review the structure of the sessions, session materials, etc. This helps observers contribute to the comfort level of the session and to correctly interpret interactions between moderators and participants. Consider passing along this one-pager about assisting in research to people who are brand new to research.
Preparing a workshop
When conducting small group activities, prepare an agenda and confirm access to any materials you will need, such as sticky notes, printed facilitation aids, etc.
Time moves fast in workshops, so be clear about what you want to accomplish and what you are willing to cut. Make sure to include time for tech checks and have a backup plan. The U.K. Government Digital Service (GDS) has some great additional tips for conducting research in small group workshops.
Preparing a wireframe or prototype
Identify in your research plan if your sessions will require the creation or use of wireframes or prototypes. This helps manage the team’s expectations, and gives the people creating those artifacts time to prepare them.
In evaluative research such as usability testing, prototypes often suggest how users might complete specific tasks. In this case:
Prototypes create dependencies in your research. When doing research involving prototypes, you will need to:
Preparing an interview guide
An interview guide keeps sessions consistent, allows multiple people to moderate, and serves as an example for future research efforts.
Most 18F interview guides include these sections:
When interviewing stakeholders, ask about their work as it relates to your research area of focus. Ask about business processes in which they play a role, the information and tools they use, the things they produce or make, and the decisions they’re responsible for. Ask about the people they work with on a regular basis and any challenges they face.
Doing a practice session
Practicing your session can help ensure everyone is on the same page and provide valuable feedback on:
- The timing of the session
- The order of the activities
- The phrasing of your questions
- The composition of your wireframes or prototypes (if applicable)
A practice session also allows you to confirm team members’ roles and test whether or not your documentation setup will provide the information you need for analysis. Be sure to send team members links to relevant documents (for example, notetaking docs for the note takers and a view-only version of the interview guide for observers) before the pilot.
It’s best to conduct a practice session a few days before your sessions with participants begin, because you will very likely find things you want to change.
Corresponding with participants
Once you’ve designed your sessions, meet with your team to discuss, decide, and agree how participants will learn about, sign up for, and contribute to those sessions.
Screening is the activity of identifying or locating people who fit your target groups, while recruiting is the activity of establishing an agreement with the people you’ve screened about their participation in the research. Together, screening and recruiting invite people to participate in your research and collect their informed consent.
To invite people to participate, you will need to advertise your sessions in places where you are most likely to reach your desired participants. To do this:
- Identify where potential participants are likely to be
- Go to those places and get people interested in the research
- Identify interested people and determine if they meet your recruitment criteria
Make a list of places where participants are likely to be, both on- and offline. Consider:
- Relevant community organizations
- Government-run events
- Government buildings (Doing pop-up research in GSA buildings)
- Personal and professional networks, social media, mailing lists, etc.
- The new or existing website
Next, go to those locations and get people interested in the research. This often involves the creation of an intercept (“research call-to-action”). For example, if you’re doing pop-up research in a GSA building, you might print your intercept on a poster; if you’re doing remote research, you might place a pop-up directly on the website itself.
Note that while it can be tempting to include additional form fields on your intercept to help with screening, this can introduce a delay due to The Paperwork Reduction Act of 1995 (PRA). See Legal for more information. All you really need to collect from people is their contact information, which isn’t subject to PRA (per this memo) if it’s collected on its own. A tweet-length intercept might read:
“Help us improve the design of OPM.gov! Sign up to participate in a remote feedback session by emailing participate-18F@gsa.gov. Learn more about how we’ll protect your information at https://www.gsa.gov/reference/gsa-privacy-program/Privacy-act-statement-for-design-research”
You can find out more about what makes a good tweet-length intercept here in our Legal article.
The third and final step is identifying interested people and determine if they meet your recruitment criteria. Once you’ve got a list of interested people and their contact information, you can easily screen them by having brief, unstructured conversations with them. Once you’ve determined that someone is qualified to participate in the session, you’ll need to get their informed consent.
We now have the ability to compensate user testing participants so long as they are public participants, not government employees. This major advancement for research and human-centered design makes it possible for us to get input from more representative groups of people in order to better deliver products and services. Our ability to compensate users for testing depends on both obtaining authorization from the agency partner with the appropriate level of authority and meeting the following criteria:
- The benefit to the government in obtaining the feedback outweighs the benefit to the participant (i.e., amount paid to the user)
- Compensating user testing participants must directly advance the client’s statutory mission and objectives
The process of paying participants for research is similar to paying for a contractor or vendor. TTS will continue to iterate on this approach in partnership with OGC. We are currently collecting value statements that might be useful in demonstrating eligibility for the “benefit to government” portion of the criteria.
More detailed information can be found in the Compensating research participants document. In order to add compensation for participants to your in-flight engagement or new IAA, you can use the above resource, which includes the User Testing Authorization, Attachment 2 - CoE - document.
Getting informed consent
Everyone who participates in 18F research must do so willingly and with an informed understanding of what their participation means. It’s your responsibility to inform the people who participate in your research and get a record of them giving their express consent.
Getting informed consent ensures that:
- Participants understand what they’re signing up for, making your sessions more effective
- The research is ethically sound (see Ethics)
- The research complies with The Privacy Act of 1974 (see Privacy)
- The research complies with The Antideficiency Act (see Legal)
Copy and customize this example design research participant agreement (Google Docs version of template or Google form version) to reflect your research design. This is the agreement you’ll ask participants to sign before they participate.
Correspond with participants ahead of time to explain things that might otherwise get lost in the agreement-signing process. In the event that a participant has not signed an agreement ahead of the interview, obtain their verbal consent. In order to give their informed consent, participants need to understand:
- Who is doing the research
- The purpose/goals of the research
- The duration of the research session
- What data you’ll collect, and how long it will be kept
- What will happen during the research, including any reasonably foreseeable risks or discomforts to the participant
- How you will use the results of the research, and who you’ll share them with
- That they may discontinue participation at any time without penalty
- What their rights are and how they can complain
You must also let participants know:
- Whether the session is being observed, and who’s watching
- Whether and how the session is being recorded
- If you will ask participants to share their screen (some organizations lack the technology or may have policies preventing screen sharing).
There are two email templates available to help with participant correspondence:
- Template for introducing yourself
(Google Docs version of email introducing yourself)
- Template for passing along a participant agreement
(Google Docs version of email to accompany participant agreement).
Remember to cover the information listed above. Ask participants if they have any questions. Ensure they don’t feel pressure to participate in ways that make them uncomfortable.
If your plans for keeping research-related records changes later, it’s your responsibility to communicate to participants about any changes in the use of their data.
Aim to help participants arrive at your sessions well prepared, but with an open mind; be especially mindful of your ability to prime participants and introduce bias.
Research participants are doing us a favor, so you should do everything you can to ensure that they have a good experience during their session. Before scheduling them for a session, make sure that you inform participants:
Also remember to:
Your agency partners can often assist in the scheduling process, particularly when setting up sessions with stakeholders. Don’t hesitate to ask them for help! You can provide them with a tailored version of this email where stakeholder introduces researcher. (Google docs version of email).
Moderating research sessions
Before each session, you should double-check:
- The date, time, and location of the session
- How you’ll get to and from the session
- How you’ll need to dress
- That you have a signed participant agreement or a plan for obtaining participants’ verbal consent
- Your primary documentation method
- That you have a backup documentation method (such as a notepad and a pen)
Moderating a research session can be nuanced, but we encourage all team members to be involved. Make sure participants are comfortable and that facilitators, notetakers, and other team members are prepared. We created a checklist of best practices for interviews. (Google docs version of the checklist). The checklist includes:
While UX Designers heavily utilize interviews, we also use a variety of other methods. Tips for moderating these can be found in the 18F Method Cards. We’ve also created a breakdown of usability testing quality heuristics. If you’re interested in developing more guides, the research guild is a great place to find collaborators.
Tips for remote research
Because our partner organizations often have national audiences and our teams are distributed, a large portion of the research we conduct is remote. Technology challenges can be frustrating for both the participant and the facilitator. Here are a few things to keep in mind when facilitating remote research:
Note: Remote-only research can exclude some people with visual or hearing impairments. Try to use a variety of approaches to make sure your research is inclusive.
Tips for in-person research
In-person research happens most often when we go on-site with an agency partner or meet participants in the field. At on-sites we often conduct workshops, but they are a great time to conduct interviews as well. Regardless of the method, here a few things to keep in mind when facilitating in-person research:
Make sure that participants have explicitly consented to you taking notes before you write things down.
The most common type of notes taken by 18F’s UX team are verbatim. Write down everything participants say — to the best of your ability — during each session. The goal is to capture as much as possible during the precious time we have with our participants, and avoid introducing the cognitive biases that come into play when we are selective about what we write.
Taking verbatim notes also curbs the natural tendency to try to understand and analyze what’s being said in the moment. If you’re having trouble writing everything down, focus on capturing what participants say; you or the interviewer can always go back and clarify what questions were asked.
Another style of note taking we commonly use is interaction notes: when we write down all of the actions people take and the reactions they have. For example, capturing a note such as “scrolled to the top of the page, re-read the instructions, scrolled back down to the input field and typed in name” would be sufficient. If conducting usability testing, consider flagging bugs or usability issues.
Note: If there are two notetakers available for a session, consider having one person take verbatim notes, and the other take interaction notes. In this case, it’s best to work in separate documents, as working too close to each other in the same file can be distracting.
Content audits frequently use spreadsheet notes: these are used to track insights and quality of existing content.
Debriefing is a useful way to discuss and capture what everyone took away from a session. It is important to debrief shortly after each session, while the details are still fresh in everyone’s minds. Keep in mind, debriefing is not synthesis — unless you’re doing continuous synthesis, it’s good to keep it descriptive only and move assessment until later in the research process.
Here are a few high-level things to capture in a debrief:
- How did this session answer our research questions?
- What were the main themes of this session?
- Which questions or activities did the participant(s) have a lot (or very little) to say about?
- What new questions do we have?
- Are there any new people we should talk to?
You may want to use this example interview debrief worksheet; feel free to modify it to your team’s needs. When determining which team members to include, default to a more diverse group. If a team member wasn’t part of the session, their role can be focused on asking questions and documenting.