Evaluators use either qualitative, quantitative, or mixed method social science research methods to collect data. Qualitative data collection dates back to human prehistory during the Egyptian civilization and the metalwork age in the Benin empire. In this resource, I will discuss the different qualitative approach to collecting data, while establishing the theoretical framework on which these methods hinges. I will also relate these methods to the evaluation focus, and the type of evaluation that best fit the methods.it will be worth noting that this resource does not discuss qualitative data analysis.
Evaluators use either qualitative, quantitative, or mixed method social science research methods to collect data.
Theoretical Approaches to Qualitative Data Collection
At the turn of the prehistoric times, Biblical and Quranic texts used to be interpreted. Likewise wisdom literature and philosophical text. This practice is known as Hermeneutics as described by the German philosopher – Wilhelm Dilthey. This remains the theoretical origin of the qualitative collection of data.
A second theoretical approach is the ethnography which is the collection of data through social meanings and ordinary activities of people in naturally occurring settings that are commonly referred to as “the field” as highlighted by John Brewer in his 2000 book – “Ethnography”.
Glaser and Strauss in their 2009 ground theory laid the foundation for the third approach – which states that instead of starting with a hypothesis, the first step should be data collection, through a variety of methods which we will discuss in the following paragraph.
Qualitative Data Collection Sources
Qualitative data can be collected from different sources namely by acquiring existing data, using existing instruments, or developing new instruments. Existing data are types of data that might yield qualitative understandings. Documents that might be available at program sites, or in the community, artefacts containing text or narratives. eg minutes of meetings, photographs, work samples produced previously by program participants. Brochures and newspaper articles describing the program are examples of existing data where qualitative data can be extracted. The practice of hermeneutics establishes a theoretical framework for this mode of collecting qualitative data.
Existing instruments are in the form of existing questionnaires which has open-ended questions. This means it is embedded within a quantitative method, as such extracting qualitative data from existing instruments is less relevant to qualitative data collection. Let’s take a step back to the existing documents in your program – photos, emails, project reports. shadow reports, minutes of meetings, newspaper clips. of course, you never intended to collect them to do a qualitative assessment. Isn’t it? So now you have to collect your qualitative data by developing your own instrument! I will discuss the use of observation and interviews. Interviews could mean individual interviews or focus group discussions. Michael Patton developed a qualitative evaluation checklist for evaluators that wants to use the qualitative data collection method and you can download it in the link below: http://irantvto.ir/uploads/qec.pdf
The table below highlights the relationship between the method chosen for qualitative data collection, and the evaluation object, and type of evaluation. (Adapted from Prof. Dr Dieter Filsinger)
| Object/Focus of Evaluation | Qualitative Data Collection Method | Type of Evaluation/Evaluation Design |
| Access to subjective viewpoints of stakeholders (e.g description, the definition of situations, aims, judgements etc) | Document Analysis
Interviews (open, semi-structured) Expert interviews |
Ex-ante evaluation
Baseline exploration Interim exploration Impact exploration |
| Access to special knowledge about the evaluation object, especially contextual knowledge | Expert interviews (semi-structured)
Narrative Interviews |
Ex-ante evaluation
Baseline exploration In all types and phases within the evaluation |
| Access to collective viewpoints, and orientations | Focus groups (structured)
Open Group discussions |
No specification |
| Access to processes, to constructions of social situations and social environment | Document analysis (project reports)
Narrative interviews Observations Video records Group discussions |
In all types and phases of qualitative evaluation analysis.
Ex-post evaluation Process evaluation Pre and Post design |
| Access to complex interactions in an organization. Reconstruction of causal social processes. | Document analysis
Narrative interviews Open group discussions Records of interactions |
Pre-post design
process evaluation |
Observation as a Data Collection Method
Observations vary in terms of protocol – style and accordingly, the quality and nature of the information that can be collected. As an evaluator, protocols are the systems of rules and procedures for recording information that guides the way that you will go about your work. These protocols could be completely structured and inflexible (by using a checklist) to being completely unstructured (by using a blank notebook). A checklist is used when evaluators are interested in documenting whether a prescribed activity was carried out or if they wish to tally the frequency with which a particular behavior was exhibited.
Observations may also take place in an unstructured way. The evaluator only captures what is striking to him/her at any given moment. This provides opportunities to gain experiential data which is not clouded by preconceived themes or categories. In this kind of observation, decisions are left solely to the evaluators personal judgement, and this may call for additional effort if certain themes and categories are needed later.
Observing requires training and preparation. Good observers are familiar with the nature of human interaction in order to understand what is transpiring. Good observers are able to separate detail from trivia and describe what is seen vividly. The ethnography theory applies to this method of qualitative data collection. Observation data are a source of incredibly rich information that has the potential to help you fully grasp a context and the direct experiences of individuals. On the downside, this method is costly, and time – consuming.
Interviews as a Qualitative Data Collection Method
The overall objective of any kind of interviewing is to elicit and collect the interviewees’ knowledge of the evaluation object. the special characteristics of qualitative interviews is to create a natural conversation situation. Interviews can take place in a format ranging from a fully structured interview, a pre-established protocol consisting of key questions. It guides the course of the conversation with little room for probing and adaptation. Examples of this are the individual guided interview, focus group interviews, problem-oriented interview, and guided expert interviews. Structured interviews may be appropriate when there is a need to have responses to the same questions from each interviewee. it is also used when there is concern about the variability of response and unequal representation of issues addressed.
Structured interviews may be appropriate when there is a need to have responses to the same questions from each interviewee.
In semi-structured interviews, questions do not need to be asked in a specified order, rather interviewers are free to probe and shuffle questions as needed. The interviewer can ask for clarifications and choose to go deep into the question as she or he thinks is needed. In semi-structured interviews, it is important that the evaluator is aware of the potential questions to be asked and not have to continually refer to your notes. As such, you will need to think about possible follow-up questions. Semi-structured are used when you have a good sense of the general topics that you would like to discuss. Semi-structured interviews can also be used for expert focused interviews.
Semi-structured are used when you have a good sense of the general topics that you would like to discuss.
In unstructured interviews, there aren’t any predetermined questions, and the agenda is set by the interviewer. The interviewer brings up topics so that there is an element of discovery which lets things emerge during the course of the interview. In this kind of interview, evaluators always say “I see” or “what do you mean by that” and so on. An example of an unstructured interview is the narrative interviews used in community studies. It is also used in life course and biological research, in family research, in professional research, and political research. The aim is to elicit stories. The narrative interview aims to get closer to experience and action in everyday situations. Another example is expert interviews which aim to generate expert knowledge. The expert is interviewed about a group of people or a field of work about which he has knowledge though, the expert is not part of it.
Focus Group Discussions as a Qualitative Data Collection Method
Focus group is characterized by homogeneity, but with sufficient variation among participants to allow for contrasting opinions. By homogeneity, I mean they have something in common that you, as an evaluator is interested in, like an occupation e.g practicing dentists; past use of a program (you want to evaluate an educational program, so you focus on a group that dropped out of the program last year; Age (you want to interview 20 – 25-year-old entrepreneurs to find out how they access loans); Gender (you want to talk to female who are farmers who have just received improved seedlings to find out how this impacted their income);Family characteristics (You want to talk to women who have had babies in the past year to help in designing a program for new moms.
Focus groups usually involve a group of between four and 10 respondents. It requires a focus group moderator, possibly the evaluator, who has set of question to ask the group and the idea is that members of the focus groups will have something to contribute or say in response to the question asked. This discussion is carried out in a quiet environment where conversations can be recorded, otherwise, another notetaker is hired. At times, it is unwise to mix genders in focus groups, particularly if the discussion is experienced differently by each gender. Men may have a tendency to speak more frequently and with more authority when in groups with women – sometimes called the “peacock effect”.
At times, it is unwise to mix genders in focus groups, particularly if the discussion is experienced differently by each gender.
Focus groups provide the benefit of enhanced understandings as a consequence of the interaction among participants. One person’s comment provides the gist for another’s and extends the idea. Disadvantages include uneven participation by those within the group, observing a phenomenon called “groupthink” whereby members of the group falsely come to share a common opinion regarding the issue at hand, and lack of control over the conversation. While the focus group can produce powerful data, you will need to be a skilled facilitator to encourage participation when necessary and to balance participation while limiting the degree to which conversations may head into uncharted waters.
While the focus group can produce powerful data, you will need to be a skilled facilitator to encourage participation when necessary and to balance participation while limiting the degree to which conversations may head into uncharted waters.
You might be wondering where surveys and questionnaires fall into. They can also be used to collect qualitative data. Open-ended items are built into these instruments that would be useful for qualitative data collection purposes. Open-ended items are questions that respondents can answer in a word, a phrase, or one or two sentences. Questions that may require lengthier answers are best suited for interviews and focus groups that we have discussed as qualitative data collection methods in this resource. We hope this will be helpful to you, as you decide on which data collection method will be appropriate for your next evaluation.
References:
Stake, R.E. (2010): Qualitative Research. Studying how Things Work. New York: Guilford Press.
Patton, M.Q. (2007): Qualitative Research & Evaluation Methods (3rd edition). Thousand Oaks: Sage Publications.
Creswell, J.W. (2013): Qualitative Inquiry & Research Design. Thousand Oaks, CA: Sage Publications.
Marvin C Alkin and Anne T vo.(2018) Evaluation Essentials from A to Z. The Guilford Press. New York
