What is Policy Evaluation?
'Evaluation is important for determining the extent to which a
policy has met or is meeting its objectives and that those intended to benefit
have done so'.
Modernising Policy Making: ensuring policies deliver value for money.
(134kb) National Audit Office, 2000
In order to know whether policies are working and why, we need to be able
to apply a range of research methods to evaluate the effectiveness of policy
interventions, implementations and processes.
This chapter describes the main types of policy evaluations and the variety
of contexts to which they can be applied.
More detailed guidance is contained in the Background
Document (pdf - 31kb)
What is Policy Evaluation?
- Policy evaluation uses a range of research methods to systematically
investigate the effectiveness of policy interventions, implementation
and processes, and to determine their merit, worth, or value in terms
of improving the social and economic conditions of different stakeholders.
- Policy evaluation uses quantitative and qualitative methods, experimental
and non-experimental designs, descriptive and experiential methods, theory
based approaches, research synthesis methods, and economic evaluation
methods.
- Policy evaluation for government privileges no single method of inquiry
and acknowledges the complementary potential of different research methods.
- The methods used in government evaluation and analysis are usually
driven by the substantive issues at hand rather than a priori preferences.
Summative and Formative Evaluation
- Two types of evaluation that are commonly used in policy evaluation
are summative and formative evaluation.
- Summative evaluation (sometimes called impact evaluation) asks questions
about the impact of a policy, programme or intervention
on specific outcomes and for different groups of people.
- Summative evaluation seeks estimates of the effects
of a policy either in terms of what was expected of it at the outset,
or compared with some other intervention, or with doing nothing at all
(i.e. the counterfactual).
- Formative evaluation (sometimes referred to as process evaluation),
asks how, why, and under what
conditions does a policy intervention work, or fail to work?
- Formative evaluations are important for determining the effective
implementation and delivery of policies, programmes or projects.
- Formative evaluation typically seeks information on the contextual
factors, mechanisms and processes underlying a policy's success or failure.
This often involves addressing questions such as for whom
a policy has worked (or not worked), and why.
- The distinction between summative and formative evaluations is not
always as rigid as the above characterisation might suggest. Determining
whether or not a policy has had an impact often involves asking questions
about how it has done so, for whom,
why, and under what conditions it has
had/not had the effect.
What is Theory-Based Evaluation?
- Theory-Based evaluation focus on unpacking the theoretical
or logical sequence by which a policy intervention is
expected to bring about its desired effects.
- For instance, a theory-based evaluation might ask about the steps that
are implicit between a policy initiative (e.g. a visit
to a prison by juveniles) and the policy outcome (reducing
crime and offending). Figure 1.1 below illustrates
what the implicit theory (of policy makers) might be:
Figure 1.1
| Visit to a Prison by juveniles |
|
First Hand Experience of Prison Life |
|
Exposure to Prison Life and Prisoners as Negative Role
Models |
|
Frightens or Scares Juveniles Away from Crime |
|
Reduces Crime and Offending |
- An alternative possible sequence of outcomes, which can be tested empirically,
is illustrated in Figure 1.2:
Figure 1.2
| Visit to a Prison by juveniles |
|
First Hand Experience of Prison Life |
|
Exposure to Prison Life and Prisoners as Positive Role
Models |
|
Stimulates or Attracts Juveniles Towards Crime |
|
Increases Crime and Offending |
- Failure to be clear about the causal sequence by which a policy is
expected to work can result in well intentioned policies being misplaced,
and outcomes that are contrary to those that were anticipated.
- Theory Based evaluation provides a number of ways of carrying out an
analysis of the logical or theoretical consequences of a policy, and can
increase the likelihood of the desired outcome being achieved.
- Theory Based initiatives will be considered in greater detail in a forthcoming
chapter of the Magenta Book.
Can Policies, Programmes or Projects be Evaluated?
- It is sometimes important to ask whether or not a policy, programme
or project can be evaluated at all.
- Some policy initiatives and programmes can be so complicated and diffuse
that they have little prospect of meeting the central requirements of
evaluability. These are:
- that the interventions, and the target population, are clear and
identifiable;
- that the outcomes are clear, specific and measurable;
- that an appropriate evaluation design can be implemented.
Have the Goals of a Policy, Programme or Project Been Achieved?
- This is one of the most frequently asked questions in policy evaluation,
and is sometimes referred to as Goals-Based evaluation.
- Before and after methods of evaluation (i.e. pre- and post- intervention
designs), using both quantitative and qualitative research methods, are
used to answer questions such as these. Comparisons between different
areas, units, or sub-groups of the population are also sometimes used
to establish variability of achievement.
- The evaluation of whether or not the targets set by Public Service
Agreements (PSA) and Service Delivery Agreements (SDA) have been met use
Goals-Based methods of evaluation.
- Goals Based evaluations do not necessarily assume that the chosen goals
are valid or appropriate measures of effectiveness. They simply measure
whether some goals or targets set by policy makers have been achieved.
- Even when the goals of a policy, programme or project have been achieved,
however, this does not necessarily mean that the policy in question has
been responsible for this outcome. Other factors may be responsible for
observed changes and alternative methods of evaluation (especially experimental
and quasi-experimental methods) are used to determine this.
How Do You Evaluate Unintended Outcomes?
- Policy makers and evaluators are often interested in the unintended
consequences or outcomes of a policy, programme or project. These unintended
outcomes may be beneficial or harmful.
- Goals-free methods of evaluation are used for this. They determine
the actual effects or outcomes of some policy, programme
or project, without necessarily knowing what the intended goals might
be.
- Goals-free policy evaluation is of interest to government social researchers
and other policy analysts in order to establish the balance between the
positive and negative consequences of policies. This is necessary to establish
the cost-benefit and cost-effectiveness of a policy or programme. Chapter
9 of the Magenta Book, ( Is it worth Spending Money on?), which is to
be published at a later date, will examine these issues in more detail.
- Goals-free policy evaluation uses similar methods as goals-based evaluation,
with a greater emphasis on qualitative methods of research (e.g. in-depth
interviews, participant-observation, focus groups).
What is Experimental and Quasi-Experimental Evaluation?
- Experimental and quasi-experimental research methods provide valid and
reliable evidence about the relative effectiveness of a policy intervention
compared with other policy interventions, or doing nothing at all (i.e.
the counterfactual).
- The purest form of experimental method is the randomised controlled
trial (RCT - sometimes called the random allocation method of
evaluation).
- Randomised control trials deal with the problem of other possible factors
influencing an outcome by exposing an experimental group of people, and
a non-experimental (i.e. control) group of people to exactly the same
factors except the policy, programme or project under investigation.
- The allocation of people to the experimental policy intervention, or
to the control (i.e. no intervention) situation, is done purely on the
basis of chance (i.e. randomisation). Randomisation may be of individuals
or of units (e.g. hospitals, schools), clusters, or whole areas.
- Randomisation does not guarantee that the experimental and control
groups will be identical, but it reduces the influence of extraneous factors
by ensuring that the only differences between the two groups will be those
that arise by chance.
Quasi-experimental methods refer to those research designs that compare
the outcomes of experimental and control groups by methods other than
randomisation. Figure 1.3 below, shows examples.
Figure 1.3
Examples of Quasi-Experimental Methods
- controlled before and after designs involving pre-test and post-test
comparisons using a single group of people (i.e.
where individuals or units are their own controls).
- controlled before and after designs in which pre-test and post-test
comparisons are made between two or more groups of people
(i.e. experimental and external controls).
- interrupted time series studies (based on repeated observations
over time of valid and reliable standardised measures of outcome).
- various types of matching designs using matched comparisons
of individuals or units before and after an intervention.
- regression discontinuity designs
|
- Experimental and quasi-experimental designs are discussed in greater
detail in the Magenta Book chapter Why
do social experiments.
What are Qualitative Evaluations?
- Qualitative evaluations are designed to 'permit the
evaluator to study selected issues in depth and detail' Patton (1990).1
- Such depth and detail is usually necessary to determine the appropriate
questions to ask in an evaluation, and to identify the situational and
contextual conditions under which a policy, programme or project works
or fails to work.
- Qualitative methods of evaluation are particularly important for formative
evaluation, though they are also used in summative evaluations.
- Qualitative evaluation uses a range of methods including in-depth interviews,
case studies, consultative methods, focus groups, ethnography, observational
and participant-observational studies, and conversation and discourse
analysis.
- These methods of qualitative evaluation are discussed in greater detail
in the Magenta Book chapter How
Do You Know Why (and How) Something Works?.
What is Economic Appraisal and Evaluation?
- Economic appraisal and evaluation uses the analytical methods of economics
to determine the cost, value and worth or a policy intervention. This
includes estimating the value of alternative uses of
a given resource (i.e. the opportunity cost of a policy,
programme or project).
- Economic appraisal is undertaken at the outset of
policy making (i.e. ex ante) to determine which of various policy options
is most likely to produce the desired outcomes, and at what cost.
- Economic evaluation is undertaken after (i.e. post
hoc) a chosen policy, programme and project has been running for some
time in order to determine whether or not the anticipated outcomes (or
other outcomes) have been achieved.
- Figure 1.4 illustrates the different types of economic analysis used
in economic evaluation.
Figure 1.4
Types of economic analysis used in economic evaluation
- Cost-analysis simply compares the costs of
different initiatives without considering the outcomes to be achieved
(or that have been achieved). The absence of information about
outcomes is a major limitation of cost appraisal and evaluation.
It cannot tells us much, or anything, about the relative
effectiveness or benefits of different interventions.
- Cost-effectiveness analysis compares the differential
costs involved in achieving a given objective or outcome. It provides
a measure of the relative effectiveness of different
interventions.
- Cost-benefit analysis considers the differential
benefits that can be gained by a given expenditure of resources.
Cost benefit analysis involves a consideration of alternative
uses of a given resource, or the opportunity cost of doing something
compared with doing something else.
- Cost-utility analysis evaluates the utility
of different outcomes for different users or consumers of a policy
or service. Cost-utility analysis typically involves subjective
evaluations of outcomes by those affected by a policy, programme
or project, using qualitative and quantitative data.
|
- Economic appraisal and evaluation uses a variety of other tools to
estimate the value of policy initiatives over time, such as the discount
rate for adjusting the value of outcomes that will occur in the
future.
- Detailed guidance on such tools, and on economic appraisal and evaluation
more generally, are provided by The
Green Book (published by HM Treasury) and will also be covered in
Chapter 9 of The Magenta Book, which is to be published at a later date.
Does Policy Evaluation deal with Ethical Issues?
- Policy making involves choices that are influenced by values and value
judgements. Political philosophy and ethics provide structured and systematic
procedures for evaluating the values and value judgements that are at
the heart of political decision making.
- Policy evaluation should include some consideration of the philosophical
and ethical dimensions of policy making and policy implementation. For
guidance on ethical assurance of government social research see Ethics
in Government Social Research.
How does Policy Evaluation relate to Project Management?
- Project and programme management provides structured and organised ways
of planning, implementing and concluding policy evaluations in an effective
and efficient manner.
- Chapter 11 of The Magenta Book will provide guidance on the main principles
and procedures of project and programme management, and of their relevance
to policy evaluation and analysis.
Reference
1 Patton, Michael Quinn. Practical
evaluation. Sage, 1992.
> Magenta
Book contents
