Background
Over the past seven years, I’ve helped teams in a range of sizes and locations, levels of maturity for design and research practice, and levels of support for research activities, build research programs that effectively inform their design decisions and product strategy. This experience has lead me to develop a perspective on the shared needs and qualities of successful research programs. The following case study shares my experience on what has worked well and how to face common challenges that may affect research teams.
OBJECTIVE
Share recommendations on building Research Programs that support Design, Product Management, Engineering, and Strategy teams in taking on more frequent, effective research and building more user-centered products and services.
ROLE
Lead Research Program development, working cross-functionally to better understand the capacities, interests, and needs of multidisciplinary teams as they pertain to research execution and collaboration.
Research Program Requirements
In my experience, the vast majority of research programs require a similar set of core features:
There are any number of approaches to building the above, and each should be created in a way that suits the team’s structure, needs, timelines, resource availability, and contexts. The following sections highlight a few examples for establishing the above, and a few things to keep in mind as some common challenges that may be faced.
Access to Appropriate Users
Naturally the first requirement for a successful Research Program is defining and gaining access to the right individuals to participate in that research. While this may be one of the biggest and most common challenges for research teams, there are a number of different user types that can be considered to make accessing users a bit easier:
1 | External Active Users
In the best case scenario, the product or service will have an active and engaged community to recruit participants from. These individuals can be sourced directly from their engagement with the product or service or through other means (e.g. email newsletters, app notifications, external recruitment services, in-platform recruitment tools). They may or may not need an incentive to participate. This route can be quite straightforward and may be easily accessible.
Potential Recruitment Tools: Pendo, Ethnio, Calendly, YouCanBookMe
2 | Internal Users
When external users are either difficult to access or difficult to incentivize and recruit, leaning on internal users or proxy-users can be effective. First, identify internal Subject Matter Experts, who are frequently members of the Sales or Client Services teams, who have direct access to users. Build strong relationships of collaboration and feedback with those individuals and engage them in the research activities on behalf of their clients. The challenge to this route is being aware of the potential bias some internal roles may unintentionally introduce to the feedback they’re providing. This can be mitigated by balancing internal feedback with external feedback if possible, or by the research team becoming a “fly on the wall” for the conversations those SMEs are then leading with clients.
Potential Recruitment Tools: Calendly, YouCanBookMe, Slack, Internal Newsletters
3 | External Proxy Users
A proxy user is anyone who fits the essential characteristics of your user base without actually being a user of the product or service itself. Whether that means they’re in the right industry, have similar experiences or interests, or fit some demographics key to the persona, they can represent the general sentiment of the user group if given enough context. This is the biggest challenge for proxy users, as providing “enough context” without slowing down or overcomplicating the research request may be difficult. Leveraging personal networks, friends, and colleagues to review research prompts ahead of sharing them with external proxy users can help in estimating the best level of context to include.
Potential Recruitment Tools: User Interviews, User Testing, Respondent, Testing Time
3 | Custom Panels
If finding an appropriate volume of participants is the biggest challenge faced by the research team, establishing a research panel could potentially be the best route. This can be a small group of highly dedicated individuals who may come from any of the above groups. The expectation to be set with this panel is that they will be the go-to resource for all research participation for an extended period of time. The challenge for this route is the possibility of missing significant portions of the user base if the panel is not a representative sample or introducing bias through repeatedly returning to the same set of participants. In order to alleviate some of this tension, identify which characteristics most strongly associate with the motivations and behaviors you’re hoping to influence through the design and research efforts and be sure to find a representative for each. Also, continue to seek out additional research participants from the above categories to introduce to and diversify the participant pool.
Potential Recruitment Tools: User Interviews, Optimal Workshop, Calendly, YouCanBookMe,
Once the audience has been defined, keeping these individuals engaged could become a full-time job. The next section will speak to some strategies for maintaining engagement and managing facilitation to learn as much as possible from each participant.
Note: I have chosen to skip the intricacies of the recruitment process itself as the creation and execution of any recruitment process will be defined heavily by the tools and audiences selected. These tools will be discussed in greater depth in the next section.
Recruitment & Facilitation Tools
Both recruitment and facilitation can be highly influenced by the tools and resources that are available, the level of time commitment expected from both the research team and the participants, and the types of research activities that should be leveraged. There are a large number of tools available with a vast range of features and flexibility levels depending on what the research team needs:
All inclusive
The most expensive, and potentially the simplest toolset to onboard, is an all-inclusive platform that covers everything from building screeners, to managing incentives, setting up prompts, and potentially even synthesizing results. Some tools in this area may offer research consulting services that will deliver insights to the team following the research activity. The biggest challenges for using these tools include an extremely high cost and potential being complex to integrate with an existing systems or process the team already relies on.
Examples: User Testing, Usability Hub, Schlesinger Group
If the all inclusive model doesn’t fit, a recruitment tool can be combined with facilitation tools to create a strong and more flexible toolset for the team:
Recruitment
Effective recruitment tools will alleviate some of the stress of identifying and managing the right participants. I recommend looking for systems that offer: recruitment both from the organization’s proprietary panel as well as an internal or custom panel, provide some level of incentive and NDA management, and some level of scheduling and management of research sessions. There are bound to be cancellations, logistical and technical issues, and mismatches of participants to screener questions, so finding a company that provides quick and effective customer service is a plus.
Examples: User Interviews, Ethnio, Respondent, Userlytics
Facilitation
Facilitation tools will have the biggest impact on the variety of research activities that can be leveraged by the team. If resources or time are limited, a basic screen-share and recording system can meet a teams’ needs and allow for remote interviews and some testing; however, as the interest in and requests for more complex, long-term, and in-depth research grows, the need to expand outside of just remote user interviews will become apparent. Systems that allow for both quantitative and qualitative studies as well as organization of projects with multiple types of research prompts will be especially valuable.
Examples: Zoom, Optimal Workshop, Useberry, Maze
A note on research methods:
Selecting an appropriate research method, whether qualitative or quantitative, evaluative or generative, is extremely important; however, that process will not be covered in-depth in this case study. I do want to address one common challenge briefly. It's important for teams to avoid the tendency of leveraging only one research approach, or failing to try more effective, or potentially less familiar, approaches. Below are a few questions to ask yourself if you’ve found your team in this scenario:
How would I answer this research question if my participants only had 5-10 minutes? How about 2 hours? How about 2 days?
How might my observation of this activity affect the participants’ responses? Is there a way to plan this activity both with and without observation? Which feels more effective? How will that affect what you learn?
- Are there activities that the participant may already be doing that may naturally answer this question? How might I observe, track, or measure those activities and/or capture feedback before, during, or after those activities?
- What additional existing data, analytics, market or secondary research can answer or inform this question?
Knowledge Management
A research team’s goal is to gather knowledge and share insight in order to influence decision making, so naturally the management of that knowledge will be integral to the team’s success. There are a few topics to consider within the activity of knowledge management:
1 | Archival
A well organized and maintained research archive will give the team significant levels of knowledge to pull from for future research as well as clear and disciplined ways to organize active projects. The system should fit the team, company, and project, but as long as it’s well maintained and adhered to, it will be extremely valuable in the long term. One challenge I recommend keeping in mind is having the ability to search, filter, and reference information across the system, as it’s nearly impossible to know when and what old data will become relevant in the future.
2 | Levels of granularity
Imagine a team member is returning to find a research insight from a project that was completed two years ago. What information would they need to pick the right project out from a list? After selecting it, what additional context would they need to know to confirm that they’re heading in the right direction? Once confirmed, what details do they need to have about the process, participants, outcomes, insights, etc. to understand what happened in that project? Lastly, how can they access the primary information and potentially uncover new insights that weren’t seen on first pass? Asking these questions will help to structure a knowledge base in a way that is easy to skim, search, and explore at a variety of levels of granularity. This will make it more approachable for research team members and other stakeholders, giving the knowledge gained more value and a longer lifespan.
3 | Access
In a world where information privacy is increasingly important, knowing what information you’re capturing from each individual participant as well as who has access to that information both immediately and in long-term must be top of mind. Pay close attention to any personal information you may be collecting and build systems where anonymity can be created and maintained as needed.
Synthesis & Communication
I firmly believe that the synthesis and communication tools needed to best share the information gathered through research must match the insights themselves, as well as the audiences’ needs and preferences around receiving that information. Failing to do either will severely impact the effectiveness of the research overall and the team’s ability to influence strategy and decision making. Highlighted below are a variety of synthesis types that can be leveraged depending on the context and needs of the teams and projects:
1 | Individual Artifacts
All researchers should be familiar with a variety of artifacts that can be leveraged as needed to communicate various types of insights. The ability to identify which artifacts fit the information gathered is vital to the artifact successfully communicating that insight. I would recommend against any research team requiring too many specific synthesis artifacts or limiting the types of artifacts a researcher can build or leverage.
Examples: Personas, Workflows, Journey Maps, Mental Models, Decision Trees, Flow Diagrams
2 | Collective Artifacts
For the purpose of maintaining a more visual or scannable knowledge base, a team may require certain types of artifacts be created or updated for each individual research activity or project. While these can become redundant over time, there are some benefits to building consistency and connections across research projects as a whole.
Examples: User Baseball Cards, Pull Quotes, Ecosystem Maps
3 | Communication Artifacts
All of the artifacts listed above are used for some type communication; however, teams may also leverage lightweight artifacts specifically for the purpose of updating and sharing insights continuously during the research project to keep relevant stakeholders informed. This helps not only improve alignment throughout the project, but also builds the influence of the research team within the organization.
Examples: Newsletters, Slack Posts, One-Pagers, Lean Learning Documents
Collaboration & Prioritization
Any team looking to improve their product or service with user research is likely to leverage one of the many existing models for doing so. Whether that be the Double Diamond, The Loop, Design Thinking, Lean Startup, Agile, or some combination, there are many well established discovery and development processes to pull from that each have a relatively consistent core process:
From the perspective of the research team, it’s vitally important to be involved in every stage of this process. A few recommendations that I follow when establishing how a research team may integrate cross-functionally throughout any of the above models are:
1 | SPEAK THE SAME LANGUAGE
It’s extremely valuable for research teams to work in tandem with the development, product management, data science, marketing, or other teams that they are most frequently delivering insights to. This could mean working in two-week versus six-week sprints, communicating through Slack versus email, writing OKRs and/or building roadmaps, the list goes on. Whatever method the team uses for goal setting, tracking, and communication, do the best you can to integrate with and reflect that system.
2 | educate and evangelize
Often, non-research teams are unsure of what a research team may do or the value they provide. It’s important from the outset that the researchers educate stakeholders on the breadth of researchers’ roles, reasonable delivery timelines, levels of communication before, during, and after research projects, outcomes that can be expected, and levels of context the team will need. The best model for this education, is being as inclusive as possible and allowing individuals in any role to see and participate in the research as much as appropriate. Collaboration between teams is absolutely not a one-size-fits-all solution and will likely evolve as the teams grow and learn more about how they prefer to work together.
3 | Avoid the “validation” trap
A significant challenge I’ve seen research teams face is the “validation trap,” or being consistently asked to confirm the assumptions, directions, and decisions that other teams have made with little to no involvement of research in developing those ideas. While this model may work for some teams, in my personal experience this has caused extreme frustration to those who are carrying out the research and design work. Setting expectations for what type of work researchers do, including both generative and evaluative, quantitative and qualitative research, and that the “validation” a stakeholder is looking for may not exist, is important in facing this challenge head on.
Final Thoughts
Lastly, the most important thing to keep in mind is that research is a highly flexible, constantly evolving practice. None of the above recommendations are rigid nor can they be actioned on in a vacuum. Research Programs must serve their users, teams, and organizations in ways that are meaningful to the teams themselves. I hope that the above can be helpful in establishing Research Programs that are affective and insightful. I look forward to evolving my own perspective as well and working closely with teams invested in the process of improving research practices for all.
