An Introduction to Program Evaluation
The full report can be found here: http://www.edu.gov.on.ca/kindergarten/FDELK_ReportFall2012.pdf
Model
Before beginning with the evaluation process, the team developed a logic model to guide their evaluation of the FDELK Program implementation. The model, with its clear visual representation of the objectives, actions and outcomes, sets out clear goals to define the primary processes used throughout the two year evaluation period. By generating the evaluation outcomes for immediate, intermediate and long-term impacts on the FDELK program, the evaluators give focus to their interpretation of the data and clearly identify their goal to support all stakeholders as the program evolves. The model also supports the collection of qualitative data from a 16 school case study and the interpretation of quantitative data provided by the Ontario government, all of which was used in drawing final conclusions and providing recommendations.
Strengths
Early Involvement
I found it interesting that the evaluation team was brought in only one month after the first stage of FDELK program was implemented. The report gives the impression that the team was able to develop their methods in tandem with the program, offering opportunities for insights that could have been missed if the evaluation was conducted further into the implementation process. It also gives a positive impression of the commitment to the program by the Ontario government.
Details, Details, Details
Right from the first section, the reader can appreciate the level of detail that is reported throughout the final report. The authors clearly communicate specific and relevant information for each major section, including a dense section of quantitative data, and do not shy away from discussing the challenges that were faced during the evaluation process. It is clear the team did not rely on one method or source of information. This, in my opinion, gives the reader a sense of the evaluator’s credibility and accountability.
Multiple Modes of Representation
This final evaluation report presents the authors’ procedures, findings and recommendations in a variety of ways. Using text, charts, data tables and other representation tools, the document is accessible to variety of readers. As someone new to these reports, I felt that it did not take me long to become familiar with the document as there were many tools to support my understanding.
Case Study Narratives
While I was unsure about what I would find as I delved into this lengthy document (225 pages), I was not anticipating the three narratives which represent different levels of fidelity* and were created through the synthesis of the case study findings.
The narratives act to clearly summarize the successes and challenges of various FDELK programs within the 16 schools included in the case study. This method of writing makes the findings accessible for a variety of readers and provide other educators with a familiar starting point for comparing their own implementation to the various success levels that were documented. They also serve to frame the recommendations found later in the report.
* A term I had to look up to properly understand - I found the brief definition from Wikipedia to be the most approachable: http://en.wikipedia.org/wiki/Fidelity#Program_evaluation
Successes, Challenges, Recommendations
Found within the appendices, although referred to throughout the report, the authors have created detailed charts that outline the successes, challenges and specific recommendations found in different Program Areas as identified by various stakeholders. This section is rich with information and provides recommendations that are relevant to certain areas that impact the broader program.
Weakness
A Blend of Formal and Informal
The final report seems to be a blend of highly accessible ‘informal’ writing (although at no point is the style truly casual) and highly formal reporting of data through the use of specific terminology. I found the sections containing quantitative data caused me to skim through to get back to a point that I was more comfortable reading. While I understand that statistical analysis and reporting requires certain details to be included, these sections impacted the overall readability of the document.
A Lack of Diversity
One of the first things that I noticed after my initial review of this document was the similarity between all of the schools within the case study. All were from relatively major cities and the students with the groups were similar in their lack of diversity. I could not help but wonder about the impact that including schools from smaller communities, or those with more diverse ethnic populations, would have had on the final findings and recommendations of the evaluation. With over 600 schools available for study, why not strive for more variety within the case study?
Final Thoughts
In all, the completed evaluation of the FDELK program implementation is rich with data, clear conclusions and specific recommendations. It reflects the obvious commitment of the evaluation team to provide a clear and detailed report without obvious bias. While dense at times, the authors do a good job of communicating the results of their evaluation in a variety of ways without shying away from the challenges that were faced by all stakeholders.
An Aside
As an aside, I think it would be interesting to see if a logic model was created during the initial development of the FDELK program as, according to McCawley (n.d.), it is a model that can be easily applied for both program planning and evaluation purposes. In looking for more information on the Logic Model, I found the outline produced by McCawley to be at the ‘just right’ level for my current understanding.
You can find McCawley’s document here: http://www.uiweb.uidaho.edu/extension/LogicModel.pdf
McCawley, P. F. (n.d.). The logic model for program planning and evaluation. Retrieved from http://www.uiweb.uidaho.edu/extension/LogicModel.pdf
Vanderlee, M., Youmans, S., Peters, R., & Eastabrook, J. (2012). Final report: Evaluation of the implementation of the ontario full-day early learning kindergarten program. Retrieved from http://www.edu.gov.on.ca/kindergarten/FDELK_ReportFall2012.pdf