|

Search

National School of Government

Related Links

• Magenta Book
• Assessing quality of qualitative research
• How evidence contributes to policy making
• Research and evaluation evidence
• Evaluation guidance
• Researching Social Policy
• Training in evaluation
• ERA Demonstration Project
• Government Pilots Review
• Further Resources
• GSRU Publications

Feedback

• Request email bulletins
• Website feedback

Guidance and advice on evaluation methods

Introduction

This area of Policy Hub provides guidance on evaluation methods for both policy makers and analysts. It also includes information about other projects and activities that aim to ensure best practice in the use of evidence in policy making. It is divided into four sections.

• Magenta Book: Guidance Notes on Evaluation Methods
• DWP Evaluation Methods Papers
• Improving Standards of Qualitative Research Research
• Other Sources of Guidance
  
  Back to Evaluating Policy

The Magenta Book: Guidance Notes on Policy Evaluation and Analysis

Strategy Unit is preparing a set of guidance notes on policy evaluation and analysis (the Magenta Book) to help 'intelligent customers' and 'intelligent providers' determine what constitutes high quality work in the field on policy evaluation and analysis. The first two chapters ('What is Policy Evaluation' and 'Systematic Reviews') have been published.

These guidance notes will complement HM Treasury's Guide to Economic Appraisal, which is more commonly known as The Green Book.

The guidance notes will be written by Strategy Unit staff and by senior colleagues from the academic and wider research and evaluation community. These guidance notes will be published electronically and will be updated regularly to keep abreast of developments and innovations in the field of policy evaluation and analysis.

For further details about Strategy Unit Guidance Notes please contact Philip Davies .

Link to the Magenta Book.

Improving Standards of Qualitative Research

Assessing the Quality of Qualitative Research

Best practice in the use of evidence in policy making recognises that not all published, or unpublished, research meets the standards of validity, reliability and relevance that is needed for policy-making.

The Cabinet Office Strategy Unit in conjunction with the National Centre for Social Research have developed a framework for assessing the quality of research evidence.

Link to the Framework

Researching social policy: the uses of qualitative methods
The July/August 2002 of Local Government News included an article by Sue Duncan, Director - Policy Studies, Strategy Unit, Cabinet Office and Alan Hedges, independent research consultant and a spokesperson for the Association for Qualitative Research. This examined the social policy role of qualitative research, based mainly on group discussion techniques which is becoming a valuable tool to help local authorities and public bodies undertake public consultation and develop their policies.

We are grateful to the publishers of LGN for their permission to publish the article on Policy Hub. Follow this link to access the article.

ESRC Research Methods Programme
The Research Methods Programme forms part of the Economic and Social Research Council's strategy to improve the standards of research methods across the UK social science community.

Department for Work and Pensions (DWP) Evaluation Methods Papers

DWP publish papers on the methods that can be used in process and impact evaluations. Papers so far published of relevance include:-

• Research methods for policy evaluation. This paper gives a brief overview of some of the main evaluation methodologies used within the Department for Work and Pensions, especially the evaluation of active labour market policies and programmes. The main focus is on the questions that can be answered by process and impact evaluations - why they might be carried out and the possible research methods used.
• Estimating the impact of labour market programmes. This paper describes the main designs available for estimating the impact of policies, the assumptions needed for each design, and the circumstances under which each design might be appropriate. The report covers defining the counterfactual; randomized trials and quasi-experimental methods; and the sample size calculations. Examples of labour market policies and programmes that have used these methods are given at the back of the report.

Note this link takes you to the ASD Research Branch Working Papers 2001 page. You will need to navigate down the page to access the publication link.

Other Sources of Guidance

• The 'Green Book': appraisal and evaluation in Central Government: Treasury guidance. (HM Treasury, TSO, 2003. (This provides HM Treasury guidance on the economic appraisal of cost and benefits of policy options).
• Chapter 4 of the Better Policy Making report (pdf) covers examples of real policies that have used research, economic modelling and evaluation.

Seminars

The GCSRO collaborated with the ESRC Methodology programme to present two seminars in 2003 which showcased some of the most innovative work being carried out by Government Researchers. As part of the seminar series, GCSRO commissioned some eminent social scientists to offer a critique of the work presented, as a way of pushing forward our thinking in the areas under discussion. The topics covered are:

• The assessment of the quality of qualitative research
• The use of Propensity Score Matching in the evaluation of the Education Maintenance Allowance

The discussion papers are:

15 September 2003

• Quality in Qualitative Evaluation: A framework for assessing research evidence
  Liz Spencer, Jane Ritchie, Jane Lewis and Lucy Dillon, National Centre for Social Research
• First discussant, Mary Dixon-Woods, University of Leicester (82kb)
• Second discussant, Clive Seale, Department of Human Sciences, Brunel University (119kb)

12 November 2003

• The use of Propensity Score Matching in the evaluation of the Education Maintenance Allowance: thoughts by Alex Bryson, Principal Research Fellow, Policy Studies Institute