Engaging stakeholders in evaluation design improves the likelihood that the findings and recommendations are relevant, credible and useful. What are the reasons for engaging stakeholders, what practical steps can evaluators take to ensure that the full range of stakeholder perspectives are represented, and how can evaluators engage with stakeholders to develop consensus around evaluations questions that are fit for purpose?
The following is a summary of A Practical Guide for Engaging Stakeholders in Developing Evaluation Questions.
- Evaluation questions set the scope of an evaluation and communicate what the evaluation will and won’t address. It is important to involve stakeholders from this stage of the evaluation process because they are the intended users of an evaluation’s findings.
- Stakeholder engagement helps to ensure that the evaluation will address their needs. Engaging stakeholders from diverse backgrounds can help to determine if the evaluation will be relevant to a range of stakeholders. In this current context of development, engaging a wide range of perspectives can also improve the evaluations credibility.
- Engaging stakeholders in developing the evaluation questions, brings in different perspectives. Drawing on the stakeholders’ knowledge, interests and experiences can improve the depth and quality of the evaluation questions. It also improves transparency by providing stakeholders a chance to raise issues from the beginning.
- Engaging stakeholders at this early stage also facilitates the rest of the evaluation. Developing stakeholder trust, can improve access for data collection, and the increased awareness of the evaluation builds the audience for its findings.
- Involving a wide range of stakeholders acknowledges that the evaluation takes place in a political context. It communicates the evaluation’s commitment to be inclusive, outward looking and expansive. Additionally, stakeholders’ engagement in developing the evaluation questions can build their evaluation capacity, and foster relationships and collaborations with one another.
- To prepare for stakeholder engagement, begin by understanding the program being evaluated. This includes when it began, who funds and conducts it, why it exists, its expected outputs and outcomes, and what has influenced its design and implementation. Information can be gathered from organizational documents and research, or conversations with individuals responsible or knowledgeable about the program.
- It is important to already start to develop relationships with a wide variety of stakeholders early. Systematically collect a breadth of contacts, and note their affiliations, expertise, location, and what groups, experiences and perspectives they represent. Extend this list, focusing on those you do not know yet. This can be done through membership networks and snowball sampling. Enough stakeholders are considered once they are representing similar feedback or a pattern of ideas.
- The nature of the program helps determine who to engage. However, types of stakeholders include experts in the area; those who represent diverse perspectives including beneficiaries, underrepresented groups and critics; managers, funders and others responsible for the program; policy makers and community power brokers and others in positions of influence; those who are interested in what the program wants to address and want to help; as well as proponents of evaluation who can build buy-in and support for the process itself.
- Time, feasibility and finances also limit the number of stakeholders that will be engaged in the evaluation question processes. Consequently, it’s important to prioritize stakeholders who are essential for the success of the evaluation process, and those who are important for practical or political reasons. Ensure that beneficiaries, and those whose points of view you are least familiar with are included. You may need to seek input from people with a different perspective on the program to make this assessment.
- Once the list of who to invite has been determined, consider what will motivate them to participate in the process. Some may be committed to the goals of the program or have a sense of community responsibility. Others may have a personal stake in the program, because of its implications on their work or reputations. For others, involvement may provide opportunities for personal development. Stipends may be expected or required by stakeholders who would not otherwise participate. These motivations will inform what engagement strategy to use in the development of the evaluation question process.
- The engagement strategy is how stakeholders will be involved in developing the evaluation questions. This includes whether you will interact with them in person or virtually, individually or in groups, through meetings or surveys. As well as the stakeholders’ motivations, factors that influence your choice include: the evaluator’s available time, budget and skills, the number of stakeholders, their geographic location and availability; and the stakeholders’ relationships with one another, familiarity with evaluation and the complexity of the program.
- For instance, one on one meetings may be more suitable when there is a short evaluation timeline and conducting it virtually can overcome the challenges of dispersed stakeholders, but it is less suited to engaging many stakeholders. Surveys on the other hand, can engage many stakeholders, but are not suitable for complex programs, and may not create the opportunities to interact that some participants seek in the way in person group meetings can.
- There are also a range of facilitation techniques that can be used during group meetings. They may be more suited to different types of groups and surfacing different aspects of the program.
-
- Developing a theory of change and/or a log framework can be helpful if the program’s goals are not clear and can inform evaluation questions about whether or how a program produces these outcomes.
- Mind-mapping brings together views about activities, actors and purpose and how they interact. It helps to develop questions about the components, elements of relationships.
- Appreciative inquiry involves stakeholders sharing stories of what works well and using the themes that emerge to determine the evaluation questions, such as why this works, and how to enhance it further.
- Roleplaying has group members assume different roles using information provided and debrief questions to understand different perspectives. This informs discussion about the evaluation questions.
- Brainstorming/nominal group technique allows individuals to generate a wide range of evaluation questions which the group can discuss and select from.
- Focus group discussions are suited to exploring attitudes and feelings. It helps to bring out issues that may have not been brought up as the subject of evaluation.
- Discussion of an article or presentation can focus and inform the stakeholders, leading to more insightful questions.
- Whichever engagement strategy and facilitation technique is chosen, evaluators generate evaluation questions by embedding key questions in the discussions. These include what success looks like, what is known and important to know, what questions about impact come up repeatedly, and what those in other roles want to know.
- While these steps help to identify and engage stakeholders, doing so is nevertheless challenging. When time is very limited, notes that previous stakeholders gathered can inform evaluation questions. It is also helpful to incrementally gather contacts and perspectives in anticipation of future evaluations.
- Engaging a wide range of stakeholders, may also necessitate managing differing opinions and priorities. To address this, establish decision making roles, and use appropriate conflict management methods. Prioritize evaluation questions according to who needs the information, when and why, and note questions that could be addressed in future evaluations or research. It is also important to facilitate dialogue between evaluators and stakeholders and recognize stakeholder contributions.
- Stakeholder tensions may also affect the process. This may require planning to create a more equitable balance of power, and facilitation processes and/or engagement strategies that give everyone the opportunity to be heard.
- Some stakeholders that are important to the process may not be motivated to participate. It can help to be clear about the importance of the evaluation, what you think they can contribute and gain from the process, or other support that can be provided.
- Finally, it is also important to engage stakeholders in the various phases of the evaluation and program development process, and in the research it spur.
Read the full guide:
A Practical Guide for Engaging Stakeholders in Developing Evaluation Questions (Guide, PDF 1001 KB, 90 mins)
Photo by Jason Goodman on Unsplash
