3. Formalize Measures
This set of questions provide guidance on formalizing measures for evaluating a program or policy, depending on whether the focus is on process or impact. For process evaluations, it emphasizes identifying specific implementation activities, resources, timing, and potential barriers, and turning these into measurable benchmarks. For impact evaluations, it focuses on defining clear, observable outcome measures aligned with the program’s objectives, including attention to equity and differential impacts across groups.
The overall goal is to ensure that evaluation measures are realistic, meaningful, and aligned with broader questions and state-level goals.
If conducting a process evaluation: Get specific about what it would take to implement the program/policy well and turn these into process measures (i.e., outputs).
3.1.1. What activities will be adopted as part of the program/policy, if any? Will the activities supplement or supplant current practices?
3.1.2. When will these activities occur? How frequently? For how long?
| Example | The Virginia team conducted a survey of school division staff to describe the landscape of mentoring and induction in the state. By understanding program structures and approaches used and how funding was used, the team could identify best practices in the state. |
|---|
3.1.3. Who will implement the activities? What training or support will they receive to support implementation? What are the attitudes towards this policy? How might attitudes shape implementation? What might prevent them from implementing the activities well? How might they adapt activities?
| Example | The Tennessee team surveyed educators to understand their decision to complete (or not) an endorsement program for high-needs teaching fields. The survey was designed to understand how licensure endorsements could be more effectively designed in the future. |
|---|
3.1.4. Where will implementation occur? Does program implementation have a disparate impact on nondominant communities?
3.1.5. What resources are available to support implementation (e.g., space, time, scheduling, materials, communications, other financial considerations). What are the opportunity costs (i.e., what might not be done as a result of these new activities)?
3.1.6. Turn process measures into process benchmarks (e.g., what will happen, when will these activities occur, who will do it, and when).
3.1.7. Align measures to evaluation questions/broader efforts or goals in the state.
If conducting an impact evaluation: Get specific about what the program/policy is trying to change and turn these into outcome measures.
| Example | The Virginia team supplemented their process evaluation with an impact evaluation designed to estimate how different levels of state-provided mentoring funding impacted early career teacher retention, both overall and among special education teachers. |
|---|
3.2.1. What are the intended objectives of the program/policy? If it works as intended, what will we see differently in school systems?
3.2.2. What outcome measures are most relevant to the program/policy? What should be excluded?
3.2.3. What are the specific, observable, and measurable indicators used to measure each outcome?
3.1.3.1. Are these short-term, intermediate-term, or long-term outcomes?
3.2.4. What is the unit of analysis for these indicators (e.g., individual student; school)?
3.2.5. Does the evaluation plan attend to the ways in which the program/policy was more/less beneficial to some groups more than others (e.g., geography, race/ethnicity, etc.)?
| Example | The Virginia team supplemented their overall analysis with a subgroup analysis by urbanicity to see if rural and urban districts in the state had different levels of success with mentoring and induction. |
|---|
3.2.6. Establish a combination of realistic and ambitious but attainable goals.
3.2.7. What would the team count as success?
3.2.8. Align measures to evaluation questions/broader efforts or goals in the state.
