Friday, 16 March 2012

ECUR 809 - Assignment #5

For this assignment we were asked to create a survey for our test program evaluation.  Once we created the survey, we were to test it with four individuals and then re-work the survey based on the results we found from the testing.

            The survey that I created for this assignment was for the students that are participants in the CAP program.

            In creating the survey I needed to first reflect on what information I wanted the survey to give me.  In doing this I decided to look back at the logic model that myself and the stake holders had developed for the program.  My thinking in doing this was that I could see the ideal for the program.  I wanted to see what the program was supposed to be doing, and then use the survey to help find out if it was doing those specific things.

            The first area that I wanted to look at was why the students are in the program.  I wanted to be able to determine if students were in the program because of a previous class that they had failed, if they are in the program for support in a class they are currently taking, or if they are in the program to work ahead.

            Secondly I wanted to get the student’s opinion on how the CAP program environment differs from the environment they experienced in the traditional classrooms.  I thought that this may give me some insight into why the Cap program is more or less successful to them than the traditional classroom is.

            Thirdly, I wanted to determine if the CAP program was a success for these specific students.  I wanted to know if their GPA had increased as a result of the program, or if they received more credits than they ultimately would have.  And I wanted to see if the program had any effects on work habits.  These are all outcomes that the stakes holders hope for, and that could be found in the logic model we created.

            I have posted below this post both the original version of the survey as well as the completed version that was re-worked after the testing.  The test subjects completed the first version of the survey and then I evaluated the results with the stake holders.  In evaluating the responses from the first version of the survey, there were a few areas that we felt we should address.

            The first area was that we felt we did not know enough about the student participants.  We felt to really evaluate more accurately the effectiveness of the CAP program; we need to know more about the participants who entered the program.  This prompted us to add the questions asking how many classes they were participating in CAP for, as well as asking for a GPA prior to entering CAP.

            We also felt that we were not truly having the students rate the success of the CAP program because we were only asking about the GPA after the program.  The GPA increase was not the only factor that impacts the success of the program; we also wanted to know if they received credits they would not otherwise have received.  A question was added to determine if students felt they received more credits than they would have without CAP.  We also added the opportunity for the students to list some skills the CAP program helped them develop.

            The wording of a couple questions also gave the students some difficulty.  They were asked in the original version about “support” they received from teachers, and they seemed to misunderstand the term “support”.  We decided to change the wording to the “help / attention” they received from teachers.  This seemed to convey are meaning better to the students.

            Finally, the question about attendance was another area where we were not sure of the accuracy of the responses.  We were not sure that the students were clear about the options for the frequency of attendance, so we changed them to be clearer.

            This process was certainly a beneficial one, for two reasons.  The first was just the overall process of creating a survey.  Developing the questions gave me the opportunity to really examine the program and understand further what I wanted to learn about it.  But beyond the experience of developing the questions, the experience of testing a survey to see if the expected results are returned was beneficial as well.  It was good to see that what you expect to hear back is not necessarily what you will hear.

This is the draft version of the survey...

This is the Completed version of the survey...



Friday, 10 February 2012

ECUR 809 - Assignment #4

For this assignment we were asked to map out the program we are evaluating using a logic model, and then to describe the model that we created.
This Logic Model starts by looking at the Inputs of the program.  The Inputs are the things that the program needs before it can begin if it is going to reach its outcomes.  In the case of the Vanier CAP program, we determined the Inputs as; Classroom Teachers – those teachers that teach the regular classroom courses, CAP Teachers – The teachers that actually run the CAP program and work with the students within that program, The Administrators – those who make the decisions regarding how the program should run, and The Students – those students who are participating in the CAP Program.
Then the Logic Model looks at the Outputs of the CAP program.  The Outputs are the things that the program actually does during the program.  In the case of the CAP program, we determined the Outputs as… Developing lessons that meet the students’ missing objectives – this is done by the CAP Teachers in coordination with the Students.  Providing support for the students while they work towards completing the missing objectives – this is done by the CAP teachers in coordination with the Students.  Contacting students who have not received credits in previous semesters to enroll them in the program – this is done by the CAP teachers working with the Students.  And finally, recommending current students who should participate in the CAP program – this is done by the Classroom teacher in coordination with the CAP teacher.
Finally, the Logic Model looks at the Planned Outcomes of the CAP Program.  The Logic Model breaks the Outcomes into Short, Medium, and Long Range.  The Short Term Outcomes are… At risk students receive more one on one support from teachers.  Students are completing course work that could not otherwise be completed in traditional classroom.  Students who need help are identified during the semester before they fail.  And, communication between teachers regarding at risk students improves.  The Medium Term Outcomes are…Students develop work habits that allow them to be productive in traditional classroom and, students receive credits they wouldn't have otherwise received.  And the Long Term Outcome is that more students are graduating.
Mapping out the program in this format was very beneficial.  It helped me to get a better understanding of how the program is supposed to be working.  Having a model like this will be helpful when actually performing the evaluation because I can compare what is actually happening to what the program is supposed to be doing according to the original plans.
I have attached an image of the completed Logic Model in this post.

ECUR 809 - Assignment #3

For this assignment we were asked to complete an evaluation assessment for our chosen program in order to examine the feasibility of the evaluation, as well as to understand more about the path that the evaluation will take.
I found this exercise to be extremely useful as it really made me sit down and think about what I would like the evaluation to accomplish, and what the best route would be to help reach that goal.
The program that I will be evaluating is the Credit Acquisition Program at Vanier Collegiate.  This program is used to help students who cannot receive credits in the traditional classroom work towards receiving their credits so that they can graduate.
In the first section of the assessment I look at the stake holders of the program and how to engage them in the process.  I decided that the stake holders need to look at this as a positive experience, so I will pitch it to them as that.  We are not here to scrutinize the program, but rather to see what is working and how we can adapt the things that aren’t working to make the program work to its fullest potential.
The next section of the assessment looks into Focusing the Evaluation.  It examines what types of questions this evaluation will look to answer.  Some of the questions are, “Is this program addressing a real problem?”, “What does the program do?”, “What are the desired outcomes of the program?”, etc. 
The next section of the assessment discusses the collection of the data.  I decided that I will collect using a variety of methods including; Survey, Interview, Observation, Document Review, and Testimonials.  I decided to use these methods based on the nature of the program and how it runs.  I wanted to obtain data in the least intrusive possible way, and I felt that I will receive the biggest picture of the program by looking at it from all of these different angles.  I will have surveys during and immediately after the program.  I will collect observations during the program.  I will review documents before and immediately after the program.  I will interview participants during the program, and receive testimonials following the program.
And finally the assessment looks at how the evaluation will be shared with the stakeholders.  I decided that the administration will receive a copy of the final evaluation, and we will meet one on one as well.  The CAP teachers and other teachers will also receive a copy of the final document, and the information will be shared to them during a staff meeting.
I have attached the completed worksheet below this blog post.


Friday, 20 January 2012

ECUR 809 - Assignment #2

For this assignment we were asked to review a program outline provided to the class, decide which model of Program Evaluation we would use to evaluate the program, and explain why we chose the model we did.  The program outline that we were given was a Prenatal Exercise Program for Aboriginal Women in Saskatoon.
In reading over the outline of the program it seemed to be separated into three main sections; the description of the problems that were occurring before the program took place, the events that should happen during the program, and then the outcomes that this program hopes to experience.
Because I feel that this program outline was separated like this, I would use Stake’s Countenance Model to evaluate the program.  I would choose this model because it examines the congruence between what was intended by the program and what actually occurred in three areas; the “Antecedents”, the “Transactions”, and the “Outcomes”.  These three areas in Stake’s Countenance Model can be mapped to the three main sections that I found to be in the program outline.
In examining the Antecedents I would look at the items listed in the “Introduction” and “Background and Preparation” sections of the program outline.  These sections outline the conditions that are taking place among Aboriginal women before the program began.  I would look at the congruence between how this program outline document outlines the conditions prior to the program and how others may have explained the conditions prior to the program.
In examining the Transactions I could look at how well the observed program actions are congruent with what the program had planned to take place.  Did what the program outline said should be done in order to resolve these problems actually take place when the program was running?
And in examining the Outcomes I would look at if the observed program outcomes were congruent with what the program had intended to be for the outcomes after participants had taken part in the program.  
I think by using this model, in this case, the effectiveness of the program could be thoroughly evaluated in a meaningful way

Friday, 13 January 2012

ECUR 809 - Assignment #1

          For this assignment we were asked to pick a completed program evaluation, explain the model used, and determine the strengths and weaknesses.  The completed Program Evaluation which I will be examining is the Saskatchewan KidsFirst Program Evaluation which was completed in 2010 by the Saskatchewan Population Health and Evaluation Research Unit (SPHERU).  In completing this evaluation several reports were published, however the bulk of the findings can be found in the following three documents; The Report of the Qualitative Study http://www.kidskan.ca/node/279, the Report of the Quantitative Study http://www.kidskan.ca/node/280, and the Summary of Findings and Recommendations http://www.kidskan.ca/node/281. 

            The KidsFirst program is a program provided by the government of Saskatchewan to at risk families with young children in certain areas of the province.  The goal of the KidsFirst Program Evaluation was to, “assess the program’s effectiveness in helping participating families and communities make positive changes” (Summary of Findings and Recommendations).

            The Saskatchewan KidsFirst Program Evaluation does not employ one pure model of program evaluation.  However, you can clearly see the strands of Stufflebeam’s CIPP model being weaved throughout the evaluation.  The CIPP model looks into four main areas of the program when doing the evaluation.  These areas are the Context, the Input, the Process, and the Product. 

Context - The authors evaluate the context by clearly outlining the four main goals of the KidsFirst program, and the objectives underneath all four of those goals, and then each one of these objectives is evaluated.  This process is clearly evaluating the context of the KidsFirst program.

Input – The KidsFirst Program Evaluation also examines the way in which the strategies of the KidsFirst program are linked to the goals and outcomes that the program wants to reach.  It examines this mainly in the qualitative study by interviewing the program participants.  This helps the evaluation team examine how well the program strategies and the goals are linked.

Process - Evaluating the process is looking into the strategies, techniques, or daily activities that are done in order to reach the goals of the program.  In this specific program evaluation, the process is evaluated mainly by interviewing those who take part in the process.  The participants interviewed included the parents, those who conduct the home visits, the home visitor supervisors, and the program managers.  Each participant was asked questions regarding how the process currently works, and was asked if they had anything that they would change if they had the opportunity.

Product – The product of the program is evaluated by determining how well the program has reached its main goals and their sub objectives.  In order to do this it attempts to answer a variety of questions including, “Did families in KidsFirst develop and maintain a safe and secure home environment?” (Quantitative Study).  All of the questions that the evaluation asks, and ultimately answers, are tied to the objectives that the program has set out to accomplish.  By answering these questions, the Product of the program is being evaluated.

            The main strength of this evaluation, I believe, is its thoroughness.  In conducting this evaluation, the team published six different reports which focused on a specific area, and then finally submitted a report which was a summary of their findings and recommendations.  Dividing the report into sections with related material made it easy to read and find the required information which is definitely a strength.  And another point that I found to be positive was that the evaluation conducted both a quantitative and a qualitative study.  This fact allowed the team to make their arguments based not only on the data, but also on feedback from the participants of the program.

            I found the main weakness of the evaluation to be in its recommendations.  Maybe if I was someone they were directing them to, the meaning may have stood out a bit more.  However, I felt that some recommendations were fairly vague.  The recommendations were listed, but no real reason as to why this is being recommended, or how it may be implemented was discussed.

            I really enjoyed the opportunity to study this program evaluation.  It gave me the opportunity to take a first look at a completed program evaluation, and helped me to start to wrap my mind around the process involved in conducting such an evaluation.