PAF 7510
Program Evaluation in Public Affairs
Fall 2005

Dr. Brandon Applegate
HPA I 360 Telephone: 407.823.3739
Office Hours:
M 2:30 - 5:30 pm & W 8:30-10:30 am
bapplega@mail.ucf.edu

Course Description

This elective course is intended for students who have completed the Foundations of Public Affairs, Advanced Research Methods, and Advanced Quantitative Research Methods courses. The course will prepare students to be educated consumers of program evaluation research and to conduct beginning evaluations independently. It also provides the foundation necessary for students to undertake more advanced evaluations with the input of an experienced evaluator.

Course Objectives

Students in this course should master (1) the conceptual knowledge of the theories and concepts essential to conducting and using evaluation research; (2) the technical skills necessary to conduct evaluation activities; (3) the interpersonal and communication skills that are required for effective dissemination and use of evaluation results; and (4) administrative skills for planning, scheduling, and managing all aspects of an evaluation.

Text and Materials

Rossi, Peter H., Howard E. Freeman, and Mark W. Lipsey. (2003). Evaluation: A Systematic Approach (7th edition). Thousand Oaks, CA: Sage.

Additional readings of journal articles and individual chapters will also be assigned.

Assignments and Activities

The course will include an examination on November 14 (25% of grade).
Also required will be an individual student or small group program evaluation project, including written report and oral presentation to the class (50% of grade). For the program evaluation project, students will design and conduct a need, process, or impact evaluation of a local social service program. During the class periods that are devoted to the program evaluation project, the instructor and students will meet to discuss conceptual and technical issues. These meetings are intended to facilitate sharing and discussion of difficulties and successes among all students; everyone is expected to be an active contributor. Grades for group evaluations will take into consideration each student’s contributions.
A portion of each student’s grade (25%) also will depend on the quality and quantity of his or her participation during each class. It should go without saying that attendance at every class is expected.

Schedule

Date Issues/Topics Readings

Aug 22 Purpose, History & Concepts of Program Evaluation
Overview of Evaluation Types Ch 1

Aug 29 Politics, Stakeholders, & Ethics
Theory-Driven Evaluation
Identifying Evaluation Questions Ch 2, 3, 5; Weiss

Sep 5 Labor Day -- no class

Sep 12 Need Assessment
Implementation Evaluation
Program Monitoring Ch 4, 6; Lewis; Bouffard et al.

Sep 19 Program Theory/Evaluation Questions -- students will construct a logic model of their program's theory and develop the questions to be addressed by the evaluation. Students should have these materials prepared before class and be ready to share, discuss, and revise them.

Sep 26 Impact Assessment I: Designs Ch 7, 8, 9, 10
Aiken et al.; Miller et al.; Weisburd

Oct 3 Need Assessment -- Students will be prepared to discuss how the need for their program could be documented. A plan should be prepared before class (which may also include rudimentary data supporting program need).

Oct 10 Impact Assessment Design -- Students will be prepared to share and discuss their plan for assessing the processes of the program and its success at achieving proximate and distal outcomes. A formal plan should be prepared and should include tentative identification of data sources and data collection techniques.

Oct 17 Impact Assessment II: Data Types,
Sources & Collection Procedures Miller; Hatry; Mitra: Gilham et al.

Oct 24 Data Collection -- students will discuss data collection plans and issues in their evaluations; particular attention will be given to obstacles in data collection.

Oct 31 Statistics Overview Newcomer; Cohen; Gartin; Foster et al.

Nov 7 Cost Benefit Evaluation
Reporting Results
Maximizing Evaluation Utility Ch 11; Hendricks; Carter; Birkeland et al.

Nov 14 Exam

Nov 21 Discussion of Evaluation Report – students will compile their findings for an evaluation report. In discussions, particular attention will be devoted to how the report might be received by stakeholders and how the findings may be presented to maximize the likelihood that they will be used. To receive specific feedback, a preliminary draft may be turned in to me no later than 5:00 pm Thursday, November 17.

Nov 28 Presentation of Evaluation Findings – Individual students (or groups) will present the results of their evaluations to the class and to invited program stakeholders. These presentations will include summaries of program need, program theory, evaluation methodology, and assessment of program process and impact. The final draft of the written evaluation report is to be turned in at the time of the presentation.

Dec 5 Presentation of Evaluation Findings – continued

Minimal Citations for Readings Listed Above

Aiken et al., Evaluation Review 22:207-244.

Birkeland et al., Evaluation and Program Planning 28:247-256.

Bouffard et al., Evaluation and Program Planning 26:149-161.

Carter, pp. 576-589 in Handbook of Practical Program Evaluation, Wholey et al. (eds.). 1994.

Cohen, Psychological Bulletin 112:155-159.

Foster et al., Evaluation Review 28:434-464.

Gartin, Journal of Research in Crime and Delinquency 32:425-445.

Gilham et al., Evaluation Review 21:589-613.

Hatry, pp. 374-385 in Handbook...

Hendricks, pp. 549-575 in Handbook...

Lewis, Evaluation and Program Planning 20:269-277.

Miller, pp. 271-292 in Handbook...

Miller et al., Evaluation of the Juvenile Drug Court Diversion Program. Dover, DE: Statistical Analysis Center.

Mitra, Evaluation and Program Planning 17:133-140.

Newcomer, pp. 389-416 in Handbook...

Weisburd, Evaluation Review 27:336-354.

Weiss, Evaluation Review 21:501-524.

“Ch” followed by a number (or numbers) refers to a chapter (or chapters) in Rossi, Freeman, and Lipsey.

Back to Courses

Date	Issues/Topics	Readings
Aug 22	Purpose, History & Concepts of Program Evaluation Overview of Evaluation Types	Ch 1
Aug 29	Politics, Stakeholders, & Ethics Theory-Driven Evaluation Identifying Evaluation Questions	Ch 2, 3, 5; Weiss
Sep 5	Labor Day -- no class
Sep 12	Need Assessment Implementation Evaluation Program Monitoring	Ch 4, 6; Lewis; Bouffard et al.
Sep 19	Program Theory/Evaluation Questions -- students will construct a logic model of their program's theory and develop the questions to be addressed by the evaluation. Students should have these materials prepared before class and be ready to share, discuss, and revise them.
Sep 26	Impact Assessment I: Designs	Ch 7, 8, 9, 10 Aiken et al.; Miller et al.; Weisburd
Oct 3	Need Assessment -- Students will be prepared to discuss how the need for their program could be documented. A plan should be prepared before class (which may also include rudimentary data supporting program need).
Oct 10	Impact Assessment Design -- Students will be prepared to share and discuss their plan for assessing the processes of the program and its success at achieving proximate and distal outcomes. A formal plan should be prepared and should include tentative identification of data sources and data collection techniques.
Oct 17	Impact Assessment II: Data Types, Sources & Collection Procedures	Miller; Hatry; Mitra: Gilham et al.
Oct 24	Data Collection -- students will discuss data collection plans and issues in their evaluations; particular attention will be given to obstacles in data collection.
Oct 31	Statistics Overview	Newcomer; Cohen; Gartin; Foster et al.
Nov 7	Cost Benefit Evaluation Reporting Results Maximizing Evaluation Utility	Ch 11; Hendricks; Carter; Birkeland et al.
Nov 14	Exam
Nov 21	Discussion of Evaluation Report – students will compile their findings for an evaluation report. In discussions, particular attention will be devoted to how the report might be received by stakeholders and how the findings may be presented to maximize the likelihood that they will be used. To receive specific feedback, a preliminary draft may be turned in to me no later than 5:00 pm Thursday, November 17.
Nov 28	Presentation of Evaluation Findings – Individual students (or groups) will present the results of their evaluations to the class and to invited program stakeholders. These presentations will include summaries of program need, program theory, evaluation methodology, and assessment of program process and impact. The final draft of the written evaluation report is to be turned in at the time of the presentation.
Dec 5	Presentation of Evaluation Findings – continued