|

Search

National School of Government

Related Links

• Magenta Book
• Assessing quality of qualitative research
• How evidence contributes to policy making
• Research and evaluation evidence
• Evaluation guidance
• Researching Social Policy
• Training in evaluation
• ERA Demonstration Project
• Government Pilots Review
• Further Resources
• GSRU Publications

Feedback

• Request email bulletins
• Website feedback

What do we already know? Harnessing existing research
Systematic Review for Policy Evaluation

'Systematic reviews contribute to evidence-based policy and practice by identifying the accumulated research evidence on a topic or question'. Davies, P T (2003)

'Systematic reviews attempt to discover the consistencies and account for the variability in similar-appearing studies' Cooper and Hedges (1994)

Systematic Review provides a tool to help access, harness and use the best available research evidence for effective policy making. This chapter describes and explains the main principles.

More detailed guidance is contained in the Background Document (pdf - 231kb)

What are Systematic Reviews?

Systematic reviews

• are overviews of the existing research literature on a topic based upon comprehensive searching of print, electronic, and unpublished sources.
• Distinguish between high and low quality research studies.
• Have explicit inclusion and exclusion criteria for deciding which primary studies to review.
• Critically appraise studies for their validity, reliability and relevance.

Meta-analysis is one type of systematic review that aggregates the samples of different primary studies and provides a cumulative estimate of the likely effect size of a policy intervention. Meta-analysis can only be undertaken on primary studies that are genuinely comparable (see sections 4.0-5.6 of The Magenta Book background paper 'What Do We Already Know?')

Why do we Need Systematic Reviews?

Systematic reviews are necessary because:-

• Single studies can misrepresent the balance of available research evidence on a topic or question.
• Single studies are often also of variable quality in terms of their design, execution, analysis and reporting.
• Single studies can be biased.
• The sheer volume of new studies presents a major knowledge management problem for analysts and policy makers.

Common Principles of Systematic Reviews

There are some basic principles of systematic reviewing. These are presented in Figure 2.1 below.

Figure 2.1

Common Principles of Systematic Review Defining an Answerable Question Systematic Searching for Studies Obtaining and Reading Studies Organising/Mapping Studies by Methodology Critical Appraisal and Sifting of Studies Data Extraction and Organisation Analysis of Data from Sifted Studies Summary Answer(s) to the Question(s) Asked Implications for Policy and/or Practice Implication for Further Research Conclusions Date of Review/Latest Update

Defining an Answerable Question

A systematic review should address a question that has the following four components:

• A clear specification of the interventions, factors or processes in question
• A clear specification of the population and/or sub-groups in question
• A clear specification of the outcomes that are of interest to the user of the review
• A clear specification of the contexts in which the question is set.

An example of an answerable question about a policy intervention might be:

What is the effect of a personal adviser service (intervention) in terms of retaining (outcome 1) and advancing (outcome 2) lone parents (population) in the UK workforce (context)?

An example of an answerable question about the implementation of a policy might be:-

What are the barriers (factor/process 1) and facilitating factors (factor/process 2) to getting lone parents (population) to participate (outcome 1) and advance (outcome 2) in the UK workforce (context)?

• Questions such as these are preferable to general questions such as 'does a personal advisor service work?'. Such questions beg the further questions: for whom? (population), in what ways (outcomes), and under what circumstances? (context)
• The framing of a question in these ways also helps identify studies that have used the most appropriate methodology.
• Intervention questions are usually best answered using experimental or quasi-experimental designs. Implementation questions are usually best answered using social surveys, qualitative methods and consultative techniques
• All identified studies must be subjected to critical appraisal

Systematic Searching for Studies

• One of the ways in which systematic reviews are different from traditional reviews is the comprehensiveness of searching for all of the available research evidence
• Systematic searching helps avoid problems of selection bias that come from only identifying studies that are readily accessible, or that are already published and indexed in major databases
• Systematic searching also helps to deal with publication bias, which comes from the tendency of some journals to overly publish findings from studies that report positive (or in some cases negative) results
• Systematic searching involves electronic sources, print sources, and the 'grey' literature. Figure 2.2. presents what is involved in systematic searching
• Systematic searching of the existing literature is a highly skilled task for which qualified librarians and information specialists should always be consulted.

Electronic Searching

• Includes a wide range of databases, electronic libraries, and high quality internet sources, see Annex A
• A selective (i.e not comprehensive) list of electronic libraries that are relevant for policy evaluation purposes is listed at Annex B
• Searching internet sites requires a greater degree of caution and critical appraisal because of the lack of quality control mechanisms.

Print Sources

• Includes journals, textbooks, and published research reports (preferably peer reviewed)
• Handsearching of print sources can be useful for identifying relevant studies that are not indexed in electronic sources
• Guidance on handsearching can be found in the Cochrane Collaboration Reviewer's Handbook

Grey Literature

• Is work in progress or not yet published
• Can be identified by:
  
  
  • searching databases of unpublished research such as SIGLE and DISSABS (dissertation abstracts);
  • searching conference proceedings, abstracts and papers;
  • searching the registers of research funders and institutions that undertake research in the policy area in question.

Methods of Systematic Searching

• There are at least two methods of systematically searching for potential studies for a review:
  
  • by all types of methodology and research design;
  • by specific methodologies and research designs
• Searching by all types of methodology yields studies that are more sensitive to the overall literature on the topic in question. However, this method of searching may identify studies that have less relevance (i.e. low specificity).
• Searching by specific methodologies yields fewer studies but these may be more relevant. (i.e. less sensitivity).
• The danger of deciding a priori to search by specific methodologies only is that this introduces the chance of selection bias into the review.

Obtaining and Reading Primary Studies

• Obtaining and reading identified studies is essential. Relying on abstracts alone is unsatisfactory. The advice and assistance of a qualified information specialist is invaluable for obtaining primary studies.

Organising/Mapping Studies by Methodology

The literature on a topic needs to be mapped. Studies should be organised and critically appraised according to the criteria that are relevant to the methodology or research design that has been used.

Critical Appraisal

• is an essential part of a systematic review.
• uses explicit and transparent criteria to determine the quality and strength of the studies that have been identified.
• separates studies that are of sufficiently high quality from those that are not.
• An example of a set of criteria that are used to appraise studies using experimental designs is given in Figure 2.3
• Criteria for appraising studies that use other methodologies are currently being developed (See References).

Figure 2.3

Typical Criteria Used to Critically Appraise Experimental Studies Question Focus Was a clear and answerable question asked? Population/Groups Studied Were the populations and subgroups studied clearly reported? Sample Size Was the sample size adequate? Quality of Randomisation Was the randomisation undertaken properly? Selection Bias Was there any selection bias in the achieved sample? Performance Bias Were the trial and control groups treated similarly other than the intervention? Attrition Bias Was the sample lost-to-follow-up in the trial groups and control groups significant? Publication Bias Does the journal in which this study was published have any known publication bias? Statistics Used Were the statistical tests used appropriate to the questions beings asked? Statistical Reporting Were the statistics that were used adequately reported? Other Criteria? Are there other relevant criteria for assessing the quality of the primary studies?

Data Extraction and Organisation

• A data collection form should be developed recording how, and why, data are to be extracted from included studies. Figure 2.4 presents an example of what to record.
• The list in Figure 2.4 is neither prescriptive nor exhaustive. Anyone undertaking a systematic review must determine what data are relevant to the policy question being asked, and develop a data extraction and organisation form to capture this.

Figure 2.4

Details to be recorded on a Data Collection Form the studies' characteristics the methods used the participants (populations and sub-groups) included and excluded the nature of the interventions or processes studied the outcomes or processes measured/observed the main and subsidiary findings of the different studies

Analysis of Data from Sifted Studies

• The analysis of data from sifted studies will depend on the policy question(s) being asked, the type of methodology used in the primary studies, and the likely use to which the findings are to be put.
• Some issues to be considered in the analysis of included studies are presented in Figure 2.5

Figure 2.5

Issues to be Considered in the Analysis of Included Studies the appropriate comparisons (if any) to be made by the analysis the study results that are needed for each comparison the assessments of validity to be used in the analysis any other data or information needed from authors of studies included the transformation or manipulation of data for analysis the heterogeneity/homogeneity of included studies the possibility of meta-analysis (see sections 4.0-5.6 of The Magenta Book background paper 'What Do We Already Know?') the likely effect sizes of the proposed policy intervention the main findings of the review the main caveats associated with the findings

Summary Answers to the Question(s) Asked

• A systematic review should provide summary answers, as well as detailed analysis and conclusions, to the policy question(s) being addressed.
  The review should be as clear as possible as to what it is saying about the policy issue in question.

Implications for Policy and/or Practice

• A systematic review may offer suggestions about the implications of the review for policy and/or practice, with the proviso that this is the opinion of the authors of the review and is not self-evident from the evidence itself.

Implications for Further Research

• A systematic review should identify any weaknesses or limitations in the existing evidence on the topic in question, and suggest how future research might be focussed to improve the evidence base in this area.

Conclusions

• A systematic review should end with as clear a message as is possible as to what the review has and has not concluded.

Date of the Review

• A systematic review is only as good as its most recent update of the available evidence. Hence, any review should be dated and its most recent update noted.

Analysts' Checklist for Undertaking a Systematic Review

• The Government Social Research Unit has compiled a checklist for analysts to use to ensure that the key methodological issues and questions governing systematic reviews have been addressed.

Policy Makers' Checklist for Undertaking a Systematic Review

• The Government Social Research Unit has compiled a checklist for policy makers which addresses some of the questions they should ask when reading a systematic review.

References

• GCSRO, Cabinet Office Strategy Unit, 2003. Quality in Qualitative Evaluation: A Framework for Assessing Research Evidence.
• Nutley, S., Davies, H.T., and Walter, I, 2002. What is a Conceptual Synthesis? ESRC Research Unit on Research Utilisation, (www.evidencenetwork.org).
• Oakley, A., Gough, D., Harden, A., 2002. Quality Standards for Systematic Synthesis of Qualitative Research', Research project, EPPI-Centre, Institute of Education, University of London.
• Pawson, R. 2002. Evidence-based policy: the promise of 'realist synthesis', ESRC Evidence Network, Queen Mary College, London
• Popay, J (Ed), 2006. Moving beyond effectiveness: Methodological issues in the synthesis of diverse sources of evidence. National Institute for Health and Clinical Excellence, London.

> Magenta Book contents