4. Develop a Data Collection Plan
Next, the roadmap guides teams in creating a thoughtful and effective data collection plan for a policy or program evaluation. It emphasizes identifying existing data sources and determining what new data needs to be collected, including the tools and responsibilities involved. The plan should reflect the evaluation’s goals, ensure data quality and consistency, and varied contexts. It also encourages balancing quantitative and qualitative data to provide a fuller picture of the program’s implementation and impact.
4.1. Identify existing and needed data.
4.1.1. What data are already available?
4.1.2. How can the team get access to this data?
| Example | The New York team took steps to access state data on dual licensed teachers in this state. |
|---|
4.1.3. What new data needs to be collected?
4.1.4. Are valid and reliable data collection tools available?
4.1.5. Has the team considered benefits and tradeoffs of collecting different types of data?
| Example | The Tennessee team initially planned to use state administrative data but shifted towards survey data after determining that evidence from a survey was more actionable for improving the design and delivery of state licensure policies. |
|---|
4.1.6. Are there issues with the chosen measures?
4.1.7. Does the data the team is collecting align with the evaluation questions?
4.1.8. If the team is primarily collecting quantitative data? Are there ways qualitative data could help tell the story of their work?
4.2. Plan for collecting and managing study data.
4.2.1. Who is responsible for designing new data collection instruments?
4.2.2. Has the data collection plan been recorded on the Outcomes Documentation?
| Example | The Virginia team began their process of studying mentoring and induction by examining an existing survey of school division directors leading these efforts. They used gaps in this survey as the basis for a revised survey that collected more actionable data. Stakeholders at the state department of education, school districts, and educator preparation programs were all involved in the development of this survey. |
|---|
4.2.3. Does the sampling plan reflect what you are hoping to learn (e.g., implementation in an “average” school/district, positive outliers, varied school/district settings).
4.2.4. Does the data collection plan reflect the voices, experiences, and/or outcomes of diverse populations, broadly defined (e.g., geographically, racially, etc.)?
4.2.5. Who is responsible for data collection?
4.2.5.1. When will they collect the data?
4.2.5.2. From whom will they collect data?
4.2.5.3. Will participants need to be sampled? If so, how? How will they ensure access to participants?
4.2.5.4. How will they ensure the data is collected consistently and accurately?
| Example | Teams established detailed data collection plans, which indicated who took the lead in collecting study data and safely storing data. Plans also indicated internal deadlines to ensure study findings could inform ongoing decision-making in the state. (also 4.7) |
|---|
4.3. Who is responsible for storing and cleaning all data?
| Example | Teams established detailed data collection plans, which indicated who took the lead in collecting study data and safely storing data. Plans also indicated internal deadlines to ensure study findings could inform ongoing decision-making in the state. |
|---|
