Planning ensures that everyone’s time is respected throughout the research process, and helps the team adapt its approach in response to the real world.
Writing a research plan
A research plan (sometimes also called a research protocol), describes the design of your research. Typical 18F research plans include:
- Research questions
- Team participation
- Participants and recruiting
- Ethics considerations
- Outputs and outcomes
18F maintains a research plan template on GitHub. (18F/GSA access only). Your research plans do not have to follow this template. What’s important is that you create a plan at all. Research planning helps you and your team:
- Openly commit to learning more about the problem(s) at hand
- Agree on which information is most useful for informing future decisions
- Learn about design research itself
- Encourage reflective practice (for example by reviewing how well the plan matched reality)
Describe factors that the research will need to account for, including any shared beliefs or forces motivating the research itself. Review and summarize any relevant secondary sources (like websites, reports, case studies, presentations); or link to prior research plans or earlier versions of the concepts you’re testing.
Design research is fundamentally about reducing risk and informing decisions. When writing your goals, use reality-oriented words like “describe,” “evaluate,” “quantify,” or “identify” and avoid vague words like “understand” or “explore.” Example goals could include: “describe users’ goals and pain points,” or “identify and evaluate the hypothesis behind our proposed design.”
Research can also have subgoals. For example, some agencies choose to work with 18F to learn more about our approach. Explicitly stating these kinds of subgoals helps provide an honest account of the coaching work that the team will undertake alongside the research itself.
Everyone on the team should agree on the research goals. A useful starting point for this conversation is clarifying research types.
What do you want to learn to make better evidence-based decisions? Research questions should be relevant, actionable, and practical. They should also be ethical; consider whether answering your research questions would put participants in a compromising position. For example, studying the degree to which participants adhere to a law or policy enforced by the researcher’s own office or institution could jeopardize participants’ careers and/or pose authority and coercion issues.
Consider holding a research alignment workshop to help stakeholders share and discuss what they’re interested in learning. Regardless of how you build alignment, focus on the value of obtaining useful information.
- Research alignment workshop plan. (18F/GSA access only version)
- Research alignment presentation slides (18F/GSA access only)
Choose one or more methods appropriate for meeting your goals and answering your research questions. Multiple methods can help you challenge or verify information collected from a single source and create a more complete understanding. 18F’s Methods provide an overview of our preferred research and design methods. Use these as a starting point, not as a list of constraints.
- Stakeholder and user interviews
- Contextual inquiry
- Content audit
- User interviews
- Design studio
- KJ Method
- Usability testing
- Cognitive walkthrough
- Card sorting
Good research is collaborative. People who help accomplish the research are more likely to agree with its outputs.
When planning your research, clarify with your partners the typical activities involved in 18F’s research, and determine which members of your partner agency’s team will help at each stage of the research process (that is, plan; do; analyze, synthesize, and share). Including partners in this process helps meet our team’s principles of designing together and training advocates.
Your timeline should provide a useful estimate of how your research process will unfold. Remind everyone that the timeline is just an estimate, and that the actual timeline will depend on a few things outside of your control, like your partners’ ability to participate, your participants’ availability (if applicable), etc.
Plan more time than you think you need, and consider especially:
Here’s a sample timeline for a contextual inquiry (on site) followed by eight 1-1 interviews (remote) with stakeholders:
|Research activity||Estimated time to complete|
|Initial meeting||1 day|
|Research design (research planning)||1 day|
|Contextual inquiry||1 day|
|Session design||0.5 day|
|Recruiting and scheduling||1 week|
|In-depth interviews (remote)||1 week|
|Initial analysis||4 days|
|Collaborative analysis||2 days|
|Communicating the results||2 days|
Participants and recruiting
Most of 18F’s design research depends on you directly interacting with people. Who those people are matters. Participants are the people you’ll recruit to take part in your research. For planning purposes, recruiting involves identifying target groups and defining your recruitment criteria relative to your research question.
Identifying target groups
Because of the time-limited nature of 18F engagements, target groups can depend on the type of research you’re doing and where you’re at in the overall design process. For example, if you’re doing stakeholder interviews as part of a Path Analysis project, you’re likely to learn more about who you need to talk to with each interview you do. As a result, you can often rely on opportunistic tactics like “snowball sampling” (asking interviewees “who should I talk to next?”) instead of explicitly identified target groups.
Once you’ve framed a problem or have otherwise centered your research around an existing service, though, it’s important that your target groups include all of the different kinds of people who may experience that problem or need to use that service. User profiles and personas are a good place to start; create provisional personas (that is, personas not based on research) if you need to.
- People who have disabilities or use assistive technologies
- People who have limited digital skills or low literacy
- People who may need help using the service in question
Defining recruitment criteria
Recruitment criteria specify the people you want to participate in your research. This depends on your research questions.
Example criteria might include:
- A particular demographic (for example, young people aged 16 to 24)
- A specific target group (for example, small business owners)
- A particular experience (for example, veterans who have recently moved home)
- A problematic situation (for example, people who suffer from opioid abuse)
- Particular ways of accessing your service (for example, people who rely on a screen reader, use speech recognition software, or who only access the internet at a library or day center)
If you’re doing usability testing, consider the following questions as well:
- What are the specific behaviors we’re looking for from participants?
- What level of tool knowledge do participants need?
- What level of domain knowledge do participants need?
Regardless of how you arrive at them, review your recruitment criteria with your team. Make sure you’re planning to recruit the right people to help answer your research questions.
Compensating research participants
GSA can compensate members of the public for participating in user research. We can not compensate government employees. We must do research with people who will actually use our services. Many government services support people through stressful situations. We learn the usability limitations of our designs when we get input from people with lived experience and diverse perspectives.
Section 508 standards require that our designs are accessible to people with disabilities. The best way to make sure our products and services are accessible is to design for these users from the start. Include people with disabilities in your user research and usability testing. To learn more about inclusive design, visit Accessibility for Teams, 18F’s Accessibility Guide, or the TTS Accessibility guild #g-accessibility (18F/GSA access only).
Why do we offer compensation?
We compensate participants for more than just the time they spend speaking with us. There can be additional costs like transportation, time off from work, and child care. We also compensate to show we value their lived experiences and expertise. Sometimes we ask participants to imagine or recall a painful personal experience, including previous difficulties that resulted from interactions with government services.
In addition to paying participants, there are other ways to recognize the value of participants’ knowledge and experience. These can include:
- Opportunity to provide feedback on design over time
- Resources for additional information on the topic
- Offer to find out answers to questions they may have
- Whenever possible, share the outcomes of the research
To compensate people for participation in design research, we must obtain authorization from the agency partner with the appropriate level of authority, and we must meet the following criteria:
- The benefit to the government in obtaining the feedback outweighs the benefit to the participant (i.e., amount paid to the user)
- Compensating user research participants must directly advance the client’s statutory mission and objectives
The process of paying participants for research is similar to paying for a contractor or vendor. In order to add compensation for participants to your in-flight engagement or new IAA, you can use the 18F User Research Authorization - Part B, Attachment 2 (18F/GSA access only).
Initial OGC guidance is to work with a recruiting agency to make payments to participants. In many cases, we’ll be more successful at achieving diverse and representative participation if we work with trusted community organizations, including from tribal nations. TTS will continue to iterate on this approach in partnership with OGC. More detailed information can be found in the doc: Compensating research participants document (18F/GSA access only).
Research affords your team powerful opportunities to interact with people and to explore what’s possible. While 18F’s UX team agrees on our own ethical principles for design research, these are just our own. Discuss and clarify ethical principles with your team and your partners. Note any ethical dilemmas or concerns.
Next, engage your team in a conversation about bias. Bias is always present in research, but you can help mitigate it by discussing the types of bias we actively work to mitigate. Power dynamics are always at play when people interact with government. As a researcher in the federal government, be aware that people’s willingness to share may change depending on their level of trust in government.
Outputs and outcomes
Before you get started, discuss with your team (including your agency partners) the desired outputs and outcomes of the research.
We follow a lean, iterative process, which allows the team to be more responsive and flexible to redefine outputs based on what the process finds. Avoid overspecifying your outputs, because you have no way of knowing what you’ll find until the research is underway. For example, it’s safer to say “We’ll produce a persona” (a type of artifact) than it is to commit to “We’ll provide 10 useful insights,” because it’s difficult to know how many useful insights the research will produce. That said, discussing possible outputs is useful because it can directly affect how you choose to document the research.
Involving partners in research planning
Hold a meeting to bring the team — including your agency partners — together to agree on the research plan. Tailor the agenda to your project’s history and your partner’s design maturity. For example, if your partner doesn’t yet have personas, you might create provisional personas before the planning meeting; if your partner hasn’t ever planned research before, you might draft a plan for them to respond to. Be ready to educate your partners on the methods you chose and why you chose them, provide example outputs from prior research, etc.
Create an agenda and invite anyone who has an interest in the team’s research. Depending on where you’re at in the design process, you might begin the meeting with level-setting excercises such as:
- Hopes and fears exercise
- Research-alignment exercise
- Creating provisional personas (that is, personas not based on research)
Next, review and confirm elements listed in the research plan. It’s especially important to confirm:
- The timeline
- What you hope to learn or do (outcomes)
- What you plan to produce (outputs)
- How the team will participate in the research
An example agenda for a research planning meeting might include:
|Hopes and fears||9:30|
|Discuss research goals||10:30|
|Review (or co-create) research plan||11:00|
|Discuss participants and recruiting||12:00pm|
|Review (or co-create) session materials (such as interview guides, wireframes, or prototypes)||1:30|
|Discuss desired outputs and outcomes||2:30|
Set up a roster
A roster is a spreadsheet to collect participants’ names, titles, contact information, and to track whether they’ve been contacted, interviewed, thanked, etc. A roster should note if specific people have opted out of the research.
Create a folder to contain your roster, interview guides, session recordings and notes, etc. This folder should also be accessible only to the core team, as it will likely contain personally identifiable information (PII); see Privacy. A good way to share interview notes without jeopardizing PII is to assign each participant a codename or participant number and refer to those in calendar invitations and notes documents. Destroy this roster at the end of the engagement.
Documenting the sessions
Session documentation can take many forms. We often conduct research that may cover sensitive topics or information. Consider the following as you decide how you will document your sessions:
Regardless of the method you choose, it’s good to keep in mind the overall reasons why you should document research as you proceed. We document research so that:
- Team members who can’t attend the sessions can look at the notes and get a very clear sense of what the user said and did;
- When even attendees’ memories eventually fade, we can refer back to the notes; and
- We create a starting point for analysis and synthesis.