Make it Better- Participatory Evaluation

Inherent in the term "evaluation" is the word "value." Thus, evaluation is the process which assesses the worth or value of a project or program, with an eye to determining the influence it has had on a population. This assessment of what has been done in the past serves as the basis for determining what should be done in the future.

Responsiveness

MASHAV is responsive to evaluations, making changes and additions when it appears a better direction is indicated. This, for example, was done in Kenya recently where MASHAV - in cooperation with the University of Nairobi - had set up the Kibwezi Irrigation Project, an irrigation field station and demonstration farm (see Shalom 1996-2). After the research and demonstration work had been completed, some of the local farmers indicated an interest in additional extension assistance. MASHAV acceded to their request. Again, working in cooperation with the University of Nairobi, along with Kenya's Extension Service, it added an extensive outreach component to the program.

In the past, evaluation was often done by donor countries who sent in their experts to check whether the results obtained warranted their continued investment. This was called result-oriented evaluation. In the words of Dr. Rodney Fink, a training, research and instructional consultant, the situation today is quite different. "There has been," he says, "a major change in the evaluation picture. We are now thinking in terms of participatory evaluation, a process which involves not only those who are responsible for planning and implementation of an activity, but also those who are its stakeholders, i.e., the students or trainees who are the beneficiaries of a course."

Since 1993 Rodney Fink has served as the United States half of a US-Israel evaluation team, visiting 11 countries. The Israeli half of the team, Dr. Shimon Amir, former MASHAV head, veteran diplomat and experienced evaluator, died in early 1997. Although his personal brand of practical wisdom and analytical way of thinking was sorely missing, MASHAV went ahead with the evaluation workshop he had planned with Rodney Fink, as much to honour his memory as to continue the work to which he had devoted so much of his life's effort. Dr. Fink was in Israel last March to help MASHAV re-examine the way it evaluates its various training programs and, by so doing, formulate new guidelines for improving its evaluation procedures.

"Ingrained in this approach of participatory evaluation," notes Fink, "is the need to think of evaluation as a process which commences from the day a course or project is planned and its objectives spelled out in specific detail. From the beginning, planners must know what they want to achieve. They must also be prepared during the course to stop and assess where they are going and if they are going in the right direction. Consequently, they must be prepared to make changes in the middle or even at the start of a course if necessary."

Thus, in a project in Uzbekistan, for example, Israeli experts initially experienced difficulties in presenting a viable program of dairy development. On-going evaluation enabled them to try different models in order to find systems that would work. Similarly, in an irrigation project in Kyrgystan, evaluation during the course of the project pointed out a significant lack of progress. The result: The expert in charge was alerted to shift from working with one unit to providing information to a number of units. In other words, constant evaluation can indicate when a program, as initially designed, is not working as well as it should, thereby enabling the trainers and instructors to make rapid and meaningful corrections and changes. "Evaluation," stresses Fink, "is a constant, ongoing process that does not begin with the end of a project, but is built into its planning and execution. From the onset, it involves the need to set targets and indicators as to how these targets are being met."

Further training

Proof that a course has achieved its objectives often comes when participants request further training in their home countries. Such was the case when a teacher from Cagayan de Oro in the Philippines attended a course in Early Childhood Education at the Golda Meir International Training Centre in Haifa. Her enthusiasm led to a request for a follow-up on-the-spot course in Cagayan de Oro. The ultimate result: commitments by local schools, private companies and city officials to institute preschool frameworks in the public, private and welfare assistance programs. Today, these programs reach hundreds of local preschool children.

As a rule, MASHAV-sponsored courses operate according to these principles, with courses planned according to objectives clearly defined from the beginning. There is also an awareness that these objectives cannot and need not be rigid or fixed. Many training centres, for example, have an initial intake session with participants during which the course director may discover that the bulk of the participants are overtrained for the course as initially designed. The result: Course content may have to be upgraded. Conversely, if a large number of the participants lack sufficient ability to cope with the material as planned, it may have to be downgraded, i.e., simplified so that they can benefit from it.

Rethinking process

Sometimes a similar rethinking process may take place ten days or two weeks into a course when trainees, individually or in groups, are asked if their expectations, at this stage, are being fulfilled. The aim behind these interviews: to check that the material being presented is relevant to their needs and to the work they will be doing on return to their home countries.

To further assess whether or not objectives have been met, trainees are usually asked to prepare a final project, designed to check how well they have absorbed the material. Some of these projects are done by groups, while others are "solo" projects. In a course on media strategies for community organization, for example, the group was asked to devise a media campaign, as well as to list the innovations they would like to introduce upon their return home.

In addition, at the end of every course, trainees fill out a lengthy evaluation sheet, as do the course directors who assess the course from their perspective.

There is little doubt that MASHAV does involve its participants in the evaluation process, but - to a large extent - this is done during and immediately following a course. MASHAV has been less successful in its follow-up and feedback procedures with graduates after completion of a course. With over 4,000 trainees graduating from courses every year, there is now a growing need to rethink the nature of MASHAV's contacts with these graduates.

Questionnaires

Many training centres do send out questionnaires to graduates a year or so following their return home in which they are asked, among other things, what information they found most valuable, what techniques were most useful and how they were able to adapt and implement the material learned into their current work. Responses to these questionnaires are, however, somewhat minimal. MASHAV is now asking itself whether or not it is time to develop a standard follow-up questionnaire, one that can be sent after one year, after five years and even after ten years. This type of continuous, structured and formalized evaluation could help MASHAV gauge the long-term results of its training.

Even the mere formulation of such a questionnaire is, however, fraught with difficulties. Long questionnaires, as devised for academic research, often ask the same question in a variety of ways in order to test for validity. Valid as they may be, their length may discourage respondents from even tackling them. Thus, although long questionnaires may be more accurate than shorter ones, their usefulness is negated if only a few people reply. On the other hand, although shorter questionnaires may elicit higher rates of response, they may not be accurate.

Doing better

In its most basic and fundamental sense, evaluation can be defined as learning how to do something better. But this "something better" requires proper long-term follow-up procedures. It must include continuous feedback from participants from former courses. The key to offering better and more meaningful courses in the future lies in the way current programs are evaluated.

Evaluation by participants may, for example, indicate that the time has come to eliminate a particular course or program which is no longer serving as useful a purpose. As Rodney Fink, however, notes, most courses are not designed with a built-in phase-out stage. There is also need to formulate guidelines that indicate when a project, such as a demonstration farm or a field experiment, becomes self-sufficient and is able to sustain itself, ready to operate on its own.

Development work is educational in every sense of the word, teaching and training others how to make a better life for themselves and their countries. But it cannot be compared to traditional educational operations in which instructors tend to measure their impact largely by testing students via written, oral and/or practical tests. In these settings, the final examination is the end of the course. MASHAV's training, however, has a different focus and a different objective. Although it is true that, in its early years, MASHAV's basic concern was on transmitting knowledge and know-how to individuals who, themselves, were its primary beneficiaries, today it has moved from "training the trainees" to "training trainers." On the whole training is now geared to teaching those who are able to hand on to others what they have learned, achieving what is called a spin-off or multiplier effect.

This move to more teacher-oriented training raises questions with which MASHAV must now grapple. One, has there been a proper assessment of the consequences, including the advantages as well as the limitations, that lie behind this approach? Does it not, for example, favour the better-educated, higher level candidate at the expense of those from other sectors of society? Two, is it not time to review the entire procedure for selecting candidates? Does not MASHAV need to develop more effective methods for selecting participants and aim, perhaps, to choose even more who can apply their training in as broad a fashion as possible? It is becoming more and more obvious that training that is not applied is of minimal value.

Follow-up data

How can one check how well training has been applied? MASHAV is aware that it has yet to devise a systematic way to collect written follow-up information. Necessary as this is, there are other avenues for collecting data that can be utilized. One possibility: regional workshops or seminars for former trainees. In evaluation one wants to check if what has been taught has made a contribution to the development and economy of the participants' country: What better way than actually asking the recipients of the training? Regional meetings could bring together former graduates who can report where an impact had been made and what the future needs of their countries are. Such workshops or seminars, even of one-day duration, could serve as a forum wherein former participants can share their practical experiences and, in the process, gain the sense that they are not alone in their attempts to improve conditions.

Even informal, often impromptu meetings between former trainees and MASHAV officials visiting various countries can elicit valuable insights. The graduates can tell what they sensed was lacking in their own training, thus helping MASHAV choose the directions in which it should move and on what new courses and new arenas of activity to focus.

Valuable resource

More and more MASHAV is realizing that its former trainees are a valuable resource for assessing the value of its training. First of all, their evaluation can serve as one of the building blocks on which to build a new course or program in Israel. Second, they can play a significant role in the planning and actual execution of an on-the-spot course, as well as helping with the instruction. Experience has shown that the most effective on-the-spot courses are those which utilize the talents, experience and services of former trainees.

It cannot be stressed enough that evaluation is not a one-sided operation. Both the planners and the stakeholders have significant roles to play. The question that now begs to be answered: How can MASHAV increase the involvement of its former trainees and how can it establish systematic avenues for collecting and collating evaluation data?

Participation by former trainees makes them partners for progress. This is not a glib phrase, but the basis for meaningful and effective evaluation, for showing what can be done better and where and how. As Rodney Fink stresses, "Evaluation is a desirable operation and not a process to be feared. All parties to an evaluation (the evaluators, stakeholders, experts doing the delivery and governments involved) have the same objective, which is to make the program better."