NPL 473 Nonprofit Leadership

Outcome Assessment and Program Evaluation

I. Planning the process

a. Chief executive should support the initiative

b. Specific individual assigned primary responsibility

c. Involve staff and clients

d. Link to organization’s information technology

II. Defining program goals

a. Types of Goals

i. Outcome goals—ultimate desired program impact

ii. Activity goals—internal mechanics of a program

iii. Bridging goals—connect activities to outcomes

b. Rules for formulating goals

i. Each goal should contain only one idea

ii. Each goal should be distinct from every other goal

iii. Use action verbs

c. Politics of goals definition

--Without clearly defined goals, “policy drift” results

d. Impact or ‘Logic’ model

i. Specifies how the various goals are expected to link to produce desired outcomes

1. abstraction

2. simplification

3. specifies significant relationships among its elements

4. formulates hypotheses

ii. Benefits

1. clarifies how program is expected to work

2. specifies questions about how program operates

3. how to use assessment data once available

III. Measuring goals

a. Concepts of measurement

i. Measurement validity—does it measure what it is supposed to measure?

ii. Measurement reliability—is measurement consistent from one application to another?

iii. Multiple measures—minimizes risks of reliability or fallibility of measurement

iv. Face validity—does measure appear valid to stakeholders?

b. Types of measures

i. Program records & statistics

1. staff are placed in untenable position if asked to provide principal measures of own effectiveness

2. staff who will record measures should be involved in defining measures

ii. Client questionnaire surveys

1. phone surveys

2. mail surveys

3. e-mail surveys

iii. Formal testing instruments

iv. Trainer observer ratings

v. Qualitative measures

IV. Data Collection, analysis and reporting

a. First, pilot-test the measures and data collection procedures

b. Establish schedule for regular reporting and review of data

c. Difference between outcome assessment & program evaluation

i. Outcome assessment data answer questions about progress on key objectives, but not the role of the agency in producing those changes

ii. Program evaluation adds comparison & control to assess that second question (role of agency)

V. Two approaches to program evaluation

a. Objective scientist approach

i. Critical distance

ii. Quantitative data

iii. Little interest in functioning of internal mechanics

b. Utilization-focused evaluation (Michael Patton)

i. Balance rather than objectivity

ii. Combine qualitative and quantitative

iii. Process as well as outcome assessment

VI. Who does the evaluation?

a. Internal evaluation performed by staff

b. External evaluation performed by outside consultants

c. Externally directed evaluation with extensive internal staff assistance

VII. Defining the purpose

a. Summative

b. Formative

c. Implementation assessment

d. Possibility of covert purposes

VIII. Outcome evaluation design

a. Causality

i. Must satisfy 3 conditions:

1. Covariation

2. Time order

3. Nonspuriousness

ii. Must satisfy

1. internal validity (design accurately describes what program achieved)

2. external validity (generalizes to contexts beyond the program being evaluated)

b. Threats to internal validity:

i. “Pre-experimental” designs

1. One-shot case study

2. Posttest only with comparison group

3. One-group pretest-posttest

ii. Threats to nonspuriousness:

1. maturation

2. regression

3. history

c. Experiments

i. Randomization

ii. control

d. Quasi-experiments

i. Non-equivalent control group (depends on quality of the match)

ii. Interrupted time series (history and obtaining multiple observations cause problems)

iii. Multiple interrupted time series

e. Other designs and controls (statistical controls)

IX. Process evaluation

X. Data development, report writing, and follow-up

a. Request opportunity to review & comment, while leaving final authority on substance in hands of external evaluator

b. Decide what final written products to request

i. Comprehensive report

ii. Executive summary

iii. Client-focused reports

c. Staff determine changes in programs in light of evaluation

MSU