1. Engage Key Partners
This set of questions is designed to guide the development of a comprehensive and effective policy or program evaluation. The first section focuses on building a well-rounded evaluation team by engaging key partners with diverse expertise, access to data, and perspectives of those impacted by the policy. The second section helps clarify the goals of the evaluation—whether it’s assessing implementation (process) or outcomes (impact)—and ensures alignment with broader state initiatives and the scope of the policy. Finally, the third section emphasizes the importance of generating support and buy-in by identifying stakeholders, communicating benefits and challenges, and evaluating feasibility based on available resources and institutional dynamics.
This text outlines how to create a thoughtful and effective data collection plan for a policy or program evaluation. It emphasizes identifying existing data sources and determining what new data needs to be collected, including the tools and responsibilities involved. The plan should reflect the evaluation’s goals, ensure data quality and consistency, and consider diverse populations and contexts. It also encourages balancing quantitative and qualitative data to provide a fuller picture of the program’s implementation and impact.
1.1. Establish an Evaluation Team
1.1.1. Who has knowledge of the program/policy design?
1.1.2. Does the evaluation team include partners who can speak to the different forms of data that may be used for a project? Does the evaluation team include partners who can help the team access these data?
| Example | The Tennessee team tapped a colleague at the Tennessee Department of Education who was completing their doctorate and could help with the development and distribution of their survey. |
|---|
1.1.3. Does the evaluation team include partners with expertise to analyze different types of data? If not, how might we reach out to researchers at district/state research departments, Regional Education Laboratories (RELs), or individual researchers to expand the team’s research expertise?
| Example | The Virginia team initiated an institutional colleague to assist with data analysis. |
|---|
1.1.4. Does the evaluation team ensure a balanced composition of members—those with the expertise needed to conduct the evaluation alongside individuals who have the time, access to data, and key positions within the state, as well as research expertise—to effectively advance the work? Does the evaluation team include members who can speak to the perspectives of those most directly affected by policy?
| Example | The Virginia team balanced important stakeholder groups to carry out their Collaboratory work. They had members on the team with expertise in new teacher mentoring and induction as well as Virginia Department of Education staff who could facilitate access to state data. |
|---|
1.2. Determine Goals of Policy Evaluation
1.2.1. Are you evaluating a program or policy?
1.2.2. Does the program/policy have an existing theory of action or logic model?
1.2.3. How is the program/policy related to other efforts or goals in the state?
| Example | All states connected the program or policy to larger efforts in the state. For instance, Virginia’s efforts to better understand how new teachers were mentored tied to larger efforts to improve teacher retention. |
|---|
1.2.4. What scope of the program/policy being implemented (e.g., statewide, pilot districts, etc.)? Does the scope of the evaluation match the scope of the program/policy?
| Example | To understand the context around the statewide law that put teachers’ tenure clock on hold if they switched from general to special education, the New York team took steps to access state data that could help speak to this issue. |
|---|
1.2.5. Do you plan to document the process or impact of the program/policy (i.e., how a program/policy is being implemented vs. its outcomes)?
1.2.6. For a process evaluation, is the evaluation meant to…
1.2.6.1. assess whether the program/policy can be improved?
1.2.6.2. determine barriers and facilitators related to successful implementation?
1.2.6.3. establish implementer attitudes towards the program/policy…
1.2.6.4. describe why a program/policy worked, or didn’t?
| Example | The Ohio team conducted a mixed methods study to understand how beginning administrators enacted leadership practices that supported the needs of all students. Participants described the leadership practices they adopted as aligned to Standard Six of the state leadership standards. They then summarized challenges in enacting these practices. |
|---|
1.2.7. For an impact evaluation, is the evaluation meant to…
1.2.7.1. determine whether or not the program/policy had the desired outcome?
1.2.7.2. impact some groups more than others, including disparate impacts on groups historically marginalized by the education system?
1.2.7.3. have unintended consequences?
| Example | Part of the Virginia team’s work with the Collaboratory involved a quantitative impact study designed to identify the degree to which school divisions that received more state resources for mentoring and induction had lower beginning teacher turnover rates. |
|---|
1.2.8. What are the goals of the evaluation (e.g., short- and long-term goals are met per theory of action/logic model, Causal relationships)?
1.2.9. What “counts” as evidence to guide different decision-making? How does the team understand “data”?
| Example | In recognition that different types of data accomplish different aims for their team, all teams collected a mix of quantitative and qualitative data to inform their understanding of how the policy or program was implemented, how it may or may not have benefited its intended audience, and how it could be improved in the future. |
|---|
1.2.10. What would make our evaluation relevant and convincing to those outside the team?
1.2.10.1. How rigorous of an evaluation is needed to achieve this goal?
1.3. Generate Support and Buy-In
1.3.1. How does the program/policy align with broader goals of educational partners in the state?
1.3.1.1. Are there requirements or a plan to evaluate the program/policy?
| Example | The Tennessee team focused on teacher endorsements given ongoing policy interest in the state about using endorsements as a tool to reduce out-of-field teaching. |
|---|
1.3.2. How does the program/policy address educational challenges within the state?
1.3.3. What is the rationale or documented need for the program/policy?
1.3.4. Who will be affected by the program/policy, either positively or negatively?
1.3.5. How do you communicate the benefits and possible downside of the program/policy?
1.3.6. Who stands to benefit from evidence of the program/policy? Who might be harmed?
1.3.7. How can relationships with different partners be leveraged to support a successful evaluation?
| Example | The Virginia team dropped part of their evaluation to avoid duplicating efforts with another research center also studying new teacher mentoring. |
|---|
1.3.8. What institutional partners may prevent a successful evaluation?
| Example | The Tennessee team forwent more distal outcome measures, such as teacher shortages, that would have taken more time and resources. The team instead focused on a targeted survey that was more feasible to carry out. |
|---|
1.3.9. Will it be feasible to carry out an evaluation based on current constraints (e.g., time, resources, logistical, personnel, political,)?
