Approaches to Evaluation

Since the early 20th century, the field of evaluation and its approaches have become dynamic, while being responsive to the political, social and economic development that has shaped our world. Thousands of evaluations have been carried out since the Paris Declaration on Aid Effectiveness by the Organization of Economic Development in 2005. Recently, our clients and scheduled course participants have become wary about the use of the word “evaluation”, and several methodologies. One thing that is certain is that methodologies are a function of the approaches used to carry out evaluations and that several approaches can be embedded in an evaluation. In this article, I will share a narrative on the different approaches to evaluation and give examples with the hope that this deepens our understanding of different ways in which evaluation can be carried out.

Evaluation approaches can be categorized into Methods – in which the evaluator is an objective, neutral party; Use – in which the evaluator is seen to be a social relations manager who will be responsible to facilitate the use of the evaluation; Values – in which the evaluator is a communicator who engages in meaningful dialogue; and Social Justice – in which the evaluator is a relationship builder based on trust and cultural respect, and an investigator of structural inequalities.

Approaches to Evaluation: The method approach is the one in which the evaluator is an objective, neutral party.

Expert – Oriented Approach

The quality of a University can be assessed by looking at its education programs, its classrooms, its operating facilities, laboratories, technology labs, the quality of its graduates etc, by experts in education administration. For example, the National University Commission of Nigeria uses its accreditation methodologies to evaluate universities in Nigeria. Likewise, for airlines to continue to operate in countries for a period of time, there are the airline operations and regulatory agency that provide certification of “fit – to – fly” for aeroplanes in their fleet.

This kind of approach to evaluation is called the expertise – oriented approach, and is the oldest type of formal, public evaluation; and as the name implies, it relies primarily on the judgement of professional expertise to evaluate the quality of an institution, program, product or activity. Variants to this approach include doctoral examinations administered by a committee, proposal review panels, site visits and conclusions drawn by professional accreditation associations, reviews of staff performance for decisions concerning promotion, driving test undertaken before licences are issued, and so forth.

The Consumer-Oriented Approach

Have you ever filed a report to the Consumer Protection Council of Nigeria? Have you filed a complaint form at an airport in Nigeria owing to a delay in flight? Have you completed a review on Amazon, eBay, Tripadvisor or Airbnb? If you have done any of this, then you must have been a respondent to a product evaluation as described in Michael Scriven’s 1991 Evaluation Thesaurus. This approach differs from the expert – oriented approach in terms of methodology. The COA relies on more transparent and quantitative methods, with the judgement typically made by an evaluator, a person with expertise in judging things, but not with the particular content expertise of an export-oriented evaluation.

Scriven’s work in product evaluation focused on describing the criteria to be used while taking into consideration that providing a checklist might be difficult. His product checklist published in 1974 reflects the potential breadth of criteria that he recommends using in evaluating educational products. The criteria are comprehensive, addressing areas from the need to process to outcomes to cost. He also developed a guide for evaluating program evaluations which can be found at http://michaelscriven.info/images/MS_KEC_8-15-15.doc

Objectives – Oriented Evaluation Approach

The Home Grown School- Feeding (HGSF) programme objective includes tackling hunger and improving nutrition; Increasing children’s access, participation and achievement in school; and supporting local livelihoods. In the Objectives – Oriented Evaluation approach, the evaluator will focus on determining whether some or all of the three program objectives are achieved, and if so, how well they are achieved. The information gained from an Objectives – Oriented Evaluation is used to make decisions on whether to continue to fund the program, change significant portions of it, or consider other approaches to evaluation.

The Objectives – Oriented Evaluation does not take into consideration activities, and/or input that leads to the objectives. Lesson objectives in courses and pieces of training are evaluated using this approach. It will allow evaluators to make judgements on the extent to which students have learnt a topic or course. Ralph Tyler’s work on this approach influenced the Elementary and Secondary Education Act (ESEA) of 1965, the first federal act to require the evaluation of educational programs in the USA.

Logic Models and Theory-Based Evaluation Approaches

Building a program theory involves first identifying those elements that the stakeholders believe are necessary to achieve their desired objectives, and then developing a model that shows how the elements relate to each other in that process. Those elements are known as inputs, activities, output, and outcomes. In this kind of approach, the evaluator focuses on the extent to which each of the elements have been met. Theory-based evaluation results in tables, charts or diagrams that are called logic models, logic frameworks or program theory models.

In recent times, this approach has dominated the thinking and development of evaluation amongst multilateral and bilateral agencies in the development sector. However, the theory-based approach oversimplifies the complexity of program delivery and context as succinctly asserted by Ray Pawson in his 2003 ‘Nothing as Practical as a good Theory’. Nonetheless, Carol Weiss, the mother of program theory, building on earlier works of Edward Suchman recognized that, if an evaluation was examining whether a program achieved its goals and that program failed, one should find out if the program failed because of implementation failure or theory failure.

Classification of the various approach type into use, stakeholder types, and methods

Goal-Free Evaluation Approaches

So let’s look at an evaluator who is told HGSF’s objective is to improve the nutrition of school children, and also increase their learning outcomes. According to the objective and theory-based evaluation approach, the evaluator may spend all his/her time designing and applying measures to look at indicators such as the number of children that are in the programme, that have access to the given food pack, whose learning outcomes have improved. She tends to forget other children who do not have access to the food pack but whose learning outcomes have also improved. This brings to fore, the Goal-Free evaluation approach which allows evaluators not to focus on the goals of a program.

Michael Scriven, a proponent of this approach, believes that the most important function of a goal-free evaluation is to reduce the bias that occurs from knowing program goals, and thus, to increase objectivity in judging the program as a whole. Nevertheless, it might still seem that the objectives-oriented and goal-free evaluation supplement each other. An internal evaluator who might be a program manager will be interested in the extent to which they have met the objectives of a program, while other unintended objectives that were met will also be considered.

Approaches to evaluation: The Use Approach is one in which the evaluator is seen to be a social relations manager who will be responsible to facilitate the use of the evaluation

The CIPP Approach

This approach allows evaluators and program managers to begin evaluation when ideas for programs are first being discussed. For example, the recent social programmes of the Nigerian government which had the Conditional Cash Transfer, Trader Moni, and Market Money could have undergone an evaluation before the programme kickstarted. This approach examines issues concerning the context, input and process, and product which can be helpful in identifying problems before they have grown and in suggesting solutions that will work better at achieving outcomes (referred to as product in this approach). But in the real world, this kind of approach becomes limited due to limited resources and time.

Stufflebeam’s work on this approach laid more emphasis on evaluators being more practical and improving programs through improving decisions. The approach considers the involvement of stakeholders routinely, at all phase of program development and execution. It involves the use of a wide range of qualitative and quantitative methods coupled with triangulation procedure to assess and interpret the multiplicity of information.

Approaches to evaluation: The Values Approach in which the evaluator is a communicator who engages in meaningful dialogue.

Utilized Focus Evaluation Approach

The first step in this approach concerns identifying the intended users or users of the evaluation. Michael Patton defines this approach as evaluation done for, and with specific intended primary users for specific, intended uses. To help primary users think about their needs for the evaluation, Patton pushes users to be more intentional and prescient about evaluation use during the design phase.

Cloneshouse Nigeria uses the CIPP and UFE approach with clients that are about to start a programme and are thinking of monitoring and evaluation as a core aspect of the program plan. Our recent work with Fredrich -Ebert Stiftung on their Open Minds Young Voices Program had this two approach integrated into our capacity building plan for the 23 implementing individuals within the group. It was intended to learn what the group needs to achieve, how they will achieve it, and help them in delivering how they can evaluate their program, while using lessons from it to improve the program.

Responsive Evaluation Approach

As described by Robert Stake, the evaluator is an observer and a negotiator in this kind of approach. The evaluator leads several facilitators to observe a program, and he then produces charts, brief narratives and portrayals. He gets program personnel to make suggestions and comments on his portrayals. He uses the most preferred medium to deliver his reports to the evaluation audience.

This approach does not involve all stakeholders, and direct attention is paid to the problems and interests of those that are involved. The use of qualitative data is generally preferred in this kind of approach. This approach does not test theories, rather it uses local knowledge and local theories in intervening in program delivery.

Fourth Generation Evaluation Approach

If you are looking for an evaluation approach that focuses on the beneficiaries more than the project implementers, you will want to consider this kind of approach. Egbon Guba and Yvonna Lincoln’s 1985 naturalistic Inquiry and 1985 Fourth-Generation Evaluation made evaluators to see their role as one of gathering views and values from a wider variety of stakeholders, and, then, the evaluator takes the role of a negotiator and facilitator, working with stakeholders, to inform each other about appropriate actions to be taken for program effectiveness.

In this approach, the evaluation questions are derived from engagement with the beneficiaries and other stakeholders. Transparency of evaluation process leads to greater acceptance. This kind of approach will require that all stakeholders are interested and knowledgeable in the topic, which can also mean consensus might not be quickly reached.

Approaches to evaluation: The social justice approach is one in which the evaluator is a relationship builder based on trust and cultural respect, and an investigator of structural inequalities.

Social Justice Approach

Are you concerned about the plight of people living with disabilities, indigenous people, youth groups, gender inequalities, and other marginalized groups? You can be an evaluator focused on using this approach. The social justice approach focuses primarily on the viewpoints of marginalized groups and interrogating systemic power structures through mixed methods to further social justice and human rights. In this approach, there is an interactive link between the evaluator and the marginalized groups. The knowledge used in evaluation is socially and historically situated and addresses issues of power and trust.

Most non -government organizations working around social accountability, democracy and human rights, the girl – child right could produce evaluators from their organizations. Methodologies of inquiries and engagement used by these organizations are used in this kind of evaluation. Some theorists in this field include Donna Mertens on her work on Human rights evaluation; Rodney Hopson and Stafford Hood on their African American work and Ernest House on Deliberative Democracy Evaluation. An evaluator in this area in Nigeria might be interested in answering evaluation questions related to finding the extent to which bills passed by parliamentarians in the National Assembly in the last five years have increased the income of smallholders farmers.

Recently, the social justice approach is becoming prominent in this evolving field of evaluation. There are new forms of it. For example, the culturally responsive evaluation in which staff and evaluators are those whose backgrounds and expertise match the relevant dimensions of diversity. Data collection tools and strategies are adapted to be suitable to the community. Data analysis involves sensitivity to, and understanding the cultural context, use of appropriate disaggregation, discussion of possible interpretation.

As the world evolves, so also does the evaluation field – from expert oriented approaches to the query of success models, to approaches that consider program initiators, and now to a more participatory approach. It is imminent that these approaches have shaped the way evaluators think, however, the most intriguing part is that there are similarities in any of the approaches that one decides to use and that the inherent principles for evaluators take cognizance of these approaches, and how it helps us in changing our world. Do you know of other approaches, or have some thoughts about new approaches? Feel free to drop your comments below.

References:

Donna Mertens and Army T. Wilson, Program Evaluation Theory and Practice, 2012

Jody Fitzpatrick et al, Program Evaluation: Alternative Approaches and Practical guidelines, 2011