5. Develop a Data Analysis Plan
This set of questions is intended to help teams create a clear and actionable data analysis plan for evaluating a program or policy. It guides teams to identify the type of analysis needed to answer their research questions, including simplifying data, identifying patterns, and comparing results to expected goals. It also encourages consideration of comparison groups, potential confounding factors, and disaggregation to understand impacts on different subpopulations. Finally, it clarifies roles and timelines for analysis, ensuring the plan supports broader state-level goals and provides meaningful insights.
5.1. Define the analysis approach.
5.1.1. What analysis is needed to answer the team’s evaluation question?
| Example | Teams relied on a number of research designs related to their research questions. Teams in Virginia and New York conducted quantitative analysis. In Ohio and Tennessee, teams supplemented quantitative analyses of survey data with coding of qualitative participant responses. |
|---|
5.1.2. How can the data be simplified to answer the evaluation question (e.g., averages)?
5.1.3. Will the analysis plan enable the team to address associated other goals or efforts in the state?
5.1.4. Who is responsible for data analysis? When will they analyze the data?
| Example | Teams were strategic in who would conduct study analyses. In Ohio, members of the Impact Evaluation committee from the Dean’s Compact on Exceptional Children conducted a comprehensive analysis with support from several team members. The Virginia team collaborated with researchers across the state to conduct the two phase of their quantitative analysis. |
|---|
5.2. Understand general patterns in the data.
5.2.1. What descriptive patterns does the team see in the data?
5.2.2. For process evaluations, how did implementation depart from the best case scenario?
5.2.3. For impact evaluations, how do the results compare to both the realistic and ambitious but attainable goals?
5.2.4. Are there outliers in the data? What can be learned from exceptional success or a less-than-to-be-desired outputs or outcomes?
5.3. Attend to context when analyzing study data.
5.3.1. Have the data been disaggregated (i.e., broken down) for relevant subpopulations?
5.3.2. What other programs/policies might this group be exposed to at the same time? Can we account for this in any way?
5.3.3. What comparison group(s) can be integrated into the analysis, including observations for groups over time and/or with similar others?
| Example | The Virginia team built upon the descriptive study with a regression discontinuity research design to estimate whether school divisions that received more funding for mentoring and induction had higher new teacher retention rates than school divisions that received less funding. |
|---|
