|

Search

National School of Government

Related Links

• Magenta Book
• Assessing quality of qualitative research
• How evidence contributes to policy making
• Research and evaluation evidence
• Evaluation guidance
• Researching Social Policy
• Training in evaluation
• ERA Demonstration Project
• Government Pilots Review
• Further Resources
• GSRU Publications

Feedback

• Request email bulletins
• Website feedback

How are the data collected? Data collection and survey design

This chapter looks at research activities that are concerned with the collection of new data - these are known as primary research activities.

A distinction can be made between the collection of qualitative and quantitative information: the former using words to describe the social world, the latter using numbers and statistics. This chapter begins with a discussion on the choice of an appropriate methodology, be it qualitative, quantitative or a mixture of the two.

The remainder of this chapter focuses on quantitative methods of data collection and survey design, the choices and compromises that have to be made, and the factors that affect these decisions.

Page 1 of 4
[ Next page ]

More detailed guidance is contained in the Background Document (pdf - 578kb).

Methods of data collection

The selection of a particular method of data collection will be influenced by the nature of the research question(s). Specifically the research question or questions will determine the most appropriate methodology to be employed.

When deciding on how the data should be collected (the methodology) it is useful to consider the types of questions the research is attempting to address. The following illustrates the range of questions one might wish to ask about a particular issue, such as teenage pregnancy or homelessness.

1. How many people are in this situation, are affected by this problem, have been helped by this initiative?
2. How prevalent is this problem?
3. Which groups are most affected by these issues/ are most at risk?
4. How much of a difference does the initiative/programme make to the prevalence of these problems?
5. Why are people affected by this problem?
6. How do people end up in this situation?
7. How does the initiative / programme work?
8. Why does it work, not work?
9. What do people think about the intervention?/How could it be improved?

If we were principally concerned with knowing the answers to questions 1 to 4 then a quantitative methodology would be more appropriate. If, however, we are principally concerned with knowing the answers to questions 5 to 9 then a qualitative methodology may be more suitable.

For a more detailed discussion of qualitative approaches to evaluation see How do you know why (and how) something works? Qualitative methods of evaluation

Strengths and limitations of qualitative and quantitative methods

The strengths and limitations of qualitative and quantitative are outlined in Figure 6.1 below:

Figure 6.1

	Qualitative	Quantitative
Strengths	. Flexible	. Produces statistical data
	. Enables exploration of the meaning of concepts, events	. Where random probability samples are used, survey estimates can be defined within specified bounds of precision
	. Produces valid data as issues explored in sufficient depth to provide clear understanding	. Can measure the extent, prevalence, size and strength of observed characteristics, differences, relationships and associations
	. Enables study of motivations and patterns of association between factors	. Can determine the importance of factors in influencing outcomes
	. Provides a detailed understanding of how individuals interact with their environment, cope with change etc.	. Uses standardised procedures and questioning, enabling reproducibility of results
Limitations		. Can be costly, particularly if population rare or 'hard to reach'
	. Need to be able to anticipate factors associated with issues to be studied, to design 'good' sampling strategy	. Sampling frame may not be available
	. Interviewing methods rely on respondents being reasonably articulate	. Structured interview hinders detailed exploration of reasons underpinning decisions or views
	. Analysis of data to generate findings is not always transparent or replicable	. Standardised questionnaire design and administration means there is little flexibility to be able to deal with respondents' misunderstanding the question (or its intention), leading to problems of validity
	. Generalis-ability of findings can be an issue	. Requires key concepts to be clearly defined and translated into meaningful survey questions. 'Fuzzy' concepts are difficult to measure

Combining qualitative and quantitative methods

Qualitative and quantitative methods can be combined and this is a useful strategy for both measuring the topic of interest and providing a detailed understanding of its nature or origins.

Figure 6.2 illustrates how the different types of evidence obtained from quantitative and qualitative methods would contribute to a study about bullying among school children and the effectiveness of an intervention.

Figure 6.2

Contribution of qualitative and quantitative evidence to answering research questions: bullying among school children*
Qualitative methods investigate/ understand	Quantitative methods measure
. The nature of different forms of bullying	. The extent to which different forms of bullying exist
. The experience of being bullied and being a bully	. The characteristics of those bullied and of bullies
. The events leading to bullying/ the circumstances in which it occurs	. Factors associated, statistically, with being bullied/ being a bully
. Why bullying continues	. Characteristics/ circumstances that correlate with length of time being bullied/ bullying
. Appraisal of any interventions experienced	. Extent to which schools have anti-bullying policies
. Influential factors in bringing periods of being bullied/ being a bully to an end	. Extent to which policies have an impact on levels of bullying in school
. Suggestions / strategies for supporting those bullied/ bullies	. Prediction of resources required to deal with bullying effectively
	. Prediction of future levels of bullying
* Based on Ritchie, 2003.

When deciding whether to combine qualitative and quantitative (survey) methods of data collection, it is important to consider what types of evidence or information are required and at what stage in the research process this evidence or information will be needed.

To get the most out of combining qualitative and quantitative methods requires:

• A clear set of research questions
• A reasonable timeframe for the research
• Close working relationships between the qualitative and survey researchers (if they are different people)
• Sufficient funding to allow this close working.

Quantitative (survey) methods of data collection

There is a range of different types of data collection methods that can be employed in collecting quantitative survey data. These are outlined in Figure 6.3 below.

Figure 6.3

Types of data collection methods
Interviewer-administered methods	Self-completion methods
. Face-to-face	. Postal
. Telephone	. Web/email

In broad terms there are three sets of factors that will influence the decision over which data collection method to employ:

• Survey administration and resource issues
• Questionnaire issues
• Data quality issues.

The difference between face-to-face, telephone and postal surveys, in terms of these factors, are outlined in Figure 6.4 below.

Figure 6.4

Summary of the strengths and weaknesses of different modes of data collection*
Design parameter	Face-to-face	Telephone	Postal
Cost of data collection	Usually most expensive method	Usually around 50-70% of face-to-face cost for same interview	Relatively cheap (but q'naires need to be kept short and simple)
Amount and type of resources required	Specialised fieldworker skills and field-force management resources needed	Specialised interviewer skills and management resources needed	For samples < 1,000 normal office resources suffice
Timetable considera-tions	May require several months unless respondents are easily accessible or 'captive'	Usually the fastest mode of data collection, but depends on respondent availability	With response reminders, may require several months
Operational control	Best for control of field sampling and data collection	Good for interviewer supervision, but respondent tolerance may be limited	Few means of controlling how q'naires are completed
Amount/ complexity of data to be collected	Best/ mandatory for long and complicated q'naires	Limitations on length and data collection complexity compared with face-to-face	Weaker for groups with poor literacy or motivation, but can be good for experts
Likely quality of the data	Best for complex topics and issues. Computer assistance improves quality. May incur interviewer effects	Good for simple factual and attitudinal questions. Computer assistance improves quality. Interviewer effects less likely	Worst for missing data, routing errors, misunder-standings
Statistical efficiency	To reduce fieldwork costs less efficient clustered samples needed for national surveys	Does not require clustered samples, but may have sampling problems	Does not require clustered samples
Expected response rate	Usually gets highest rate	Likely to be 10-40% lower than face to face	Usually lowest rate. Can be well below 50% for less literate/ motivated
* Based on Lynn & Thomas, 2003.

Combining methods

There are times when it may be appropriate to combine different methods of data collection, for example to:

• Save money
• Improve geographical coverage
• Overcome sample frame bias
• Improve data quality, such as response rates or item non-response
• Speed up data collection
• Overcome resource problems, such as a lack of face-to-face interviewers.

Despite these advantages there are potential pitfalls to combining data collection methods.

• Development time may need to be extended, as two or more data collection instruments will need to be designed.
• Survey management costs will be increased, as different groups of people will receive different treatments (although these may be offset against savings in the overall cost of the survey).
• Keeping track of the outcome (interview, refusal, non-contact etc) for each case will be required at each stage of the data collection process, particularly if the design involves a follow up of non-responders using an alternative data collection method.
• Particular care will be required to avoid data being lost or duplicated, as a result of having to stitch together data collected from different sources.
• Mode effects can impact on the reliability and validity of the data collected. These effects result from differences in the way in which questions are presented to respondents, for example whether the answer options are presented on a card or are read aloud, and are a particular problem for attitudinal questions. Mode effects can also be observed when asking about sensitive behaviour, when social desirability factors can influence how respondents decide to answer the question.

Page 1 of 4
[ Next page ]

> Magenta Book contents