16573.pdf
Media
- extracted text
-
1
Sambodhi
Research & Management Institute
A subsidiary of Sambodhi Research & Communications Pvt. Ltd.
A
fe;
u
A.?
■
i
; t?
[READING MATERIAL]
Training Programme on Monitoring & Evaluation
of Development Projects
26th - 28th May, 2011
New - Delhi
I
Reading Material 1
Monitoring and Evaluation of Development Projects
Goals, Objectives and the Logical Framework
Goal & Objective
The most critical step in designing a project is definition of setting-up goals and
objectives. And of course, these are two different concepts often used interchangeably.
Specificity and clarity of the goal and objective is quintessential for any projects as the
entire project design flows from and aligned towards the objective.
©
©
V:?:"
The goal of project is the general statement of the desired conditions. A goal thus have
connotations of long-term and vision e.g. improvement in quality of life.
The objective of the project is the specific statement of the desired state or the target
condition. SMART Criteria is often used for designing objective, this being
Specific
Measurable
Attainable
Realistic
Time-bound
The Logical Framework Analysis
<■
r
r©■
•t' ■
0
fe
€
iu
c
r
I f
The Logical Framework Analysis or LFA is a simple project design tool that helps
organize and structure thinking in the project. As the name suggests, it’s a logical
approach for designing efficient and effective project. On one hand it faciliates optimal
resource allocation while on eth other hand it sets performance measures and standards
that provide for a framework for monitoring and evaluation. This all helps in efficient
project management.
The process steps in LFA are
1. Situation analysis
2. Problem Analysis
3. Objective Analysis
4. Alternative Analysis
5. Risk Analysis
6. Development of Project Planning Matrix
The LFA leads to development of Project Planning Matrix (PPM) or logframe that is a
snapshot view of the entire project.
The LFA-PPM
The logframe or the PPM is a four-by-four matrix that details the logical connect between
the various project components as well as gives the framework for assessing
performance.
The basic philosophy of the LFA is the logical approach in achievemnt of the impacts. It
states that, inputs given through the activties would lead to outputs, which would lead to
outputs, which would lead to outcomes which will lead to impacts. This hierarchy forms
SamGodEi ^search
Communications
1
u
Reading Material 1
Monitoring and Evaluation of Development Projects
the rows of the PPM matrix and the logical connect between various levels is termed as
the vertical logic.
For each of the hierarchial level, performance measures are set, these being called as
indicators. And for each of the measure, sources of information are also listed. Finally,
the assumptions that must happen for the activties to translate into outputs, or outputs to
purpose, or purpose to goal are also listed. This forms the horizontal logic of the PPM.
The columns and rows of the PPM are given below.
HORIZONTAL LOGIC
nmar
feU
R
T
I
C
A
L
^^issuniptTons^^^
S
oft? i
! Q‘oa^
gw
L
O
G
I
C
C•
©
©
Sam6o<fiii Qtesearcfi c£ Communications
I-'*
i
2
Reading Material 2
Monitoring and Evaluation of Development Projects
THE PROJECT PALNNING MATRIX - ZOPP
fsWbgAssur
Overall Goal: ?
Superior strategic
goal for the
project
n
JD^veJppmegtJI
■'Goal:;;g^'^|y
IS..........
V5'
V.- .-■
IndicatorsioUhe
9 ^tassumptioris^g;
‘it'S
|®v0tosig^riizg
S iwjWF*4
I
€?
tiori^'
. .y : '
go?
designedib:
Project
How to recognize
Purpose:
r that ;5the
..y project
Change in action .purpose has,
of the users of . been
hppn .anhipvprl
achieved
the project’s
X;
services
Matters outside
How to recognize
trio influence of - that the
tatptgroups : assumption has
to achieve their
developmental
goal
■
■
■
Results:
flpoducts a®
VlajdT^B|
•
^atterstbtftsideg
.'the project -
generated^p^
..
...H
Tib
i ggtggM
).w to re^od n i
a
Activities:
To achieve the
results
Quantities and
costs
: S
*e
•
SamGodHi ‘Rpsearvli d Communications
:
: I
3
Reading Material 3
Monitoring and Evaluation of Development Projects
Monitoring and Evaluation Basics
Understanding M&E
©
Monitoring essentially is the process of tracking of implementation ot activities of the
project and attainment of planned outputs. It is the process of systematically collecting
data in order to provide information for all stakeholders (managers, funders, participants)
on the progress of implementation and the achievement of desired outcomes.
Critical functions of monitoring are to gather feedback from participants; analyse
contextual changes; and provide an early warning system of potential challenges. The
analysis of monitoring data is critical to making informed mid-term programmatic
changes. Monitoring is applicable to all programme levels (from output to outcome) most commonly the focus is on output data, although tracking the goals and objectives is
also important.
The monitoring results help:
■ Improve strategies and targeting, allowing decision-makers to concentrate
resources where they can have the most leverage and impact
■ Provides a framework for understanding of barriers to the envisioned objectives
■ Ensures a focus on impact level changes throughout the project, rather than
just at the end of project evaluation.
Evaluation is the assessment of outcomes and impacts. It is to answer the question first
that whether the envisaged objectives and goals have been achieved or not. And
secondly, whether the achievement is because of the project interventions or not. So to
say, evaluation is to see that there is no deviation from the goals and objectives, the the
change achieved can definitely be attributed to the project interventions. Evaluation
established the cause-effect relationship between the .activities and outputs with the
objectives and goals.
V-’
e
While monitoring facilitates mid-course correction in attainment of project outcomes;
evaluation helps analyze variances from envisioned objectives and goals. Providing
feed-forward, to the project functionaries, M&E thus facilitates .learning by doing.
Development and enhancement of in-house capacities to anchor the M&E functions thus
is prerequisite for learning organizations.
Monitoring Vs Evaluation
To put in simple words, monitoring is to see “what we are doing” whereas evaluation is
to assess “what we have done”. Some of the distinctions between M&E are given in the
matrix below.
ifil
s: ■:
- elements'She per
rformance of a
atetigi
L
t
r
c
;
SambodHi fysearcfi
Communications
4
e ’S5
<?'■
ir
Reading Material 3
Monitoring and Evaluation of Development Projects
A
0
I
M&E levels
ma'tdxbelow eVenient' W'th reSPeCt t0 the l0giC’the M&E levels are given in the
;l^?Ompacf
$
0
^Evaluation
Hr Ou
.i£Qutputs:'j£i »jWisi •
,7
Or to say, with respect to the Logical Framework, the top two rows are the domain of
evaluation whereas the bottom two rows are the domain of evaluation.
MB
fa
Thus, the major M&E levels are:
■ Inputs
■ Activities
■ Outputs
■ Outcomes
■ Impacts
Monitoring typologies
Monitoring generally is a task that is undertaken by the ‘doers’ or the oroiect
S>mepSerS themtselves- Therefore. monitoring is inherently is Internal monitoring
Sometimes projects may involve parties external to the project for facilitatino the
monitoring functions. This is classified as external monitoring.
facilitating the
Another way to look at monitoring is on the basis of who all are involved in the
themSv^^H eSS’
thS Pr°ieCt imPlementers restrict the monitoring process all to
inform^ Ln
“’rin9: The communities here just remain mere
ormation providers and have no role in analyzing the information and providing inputs
other^slakeho ders"?/'
When ftUnctio,lial Participation of not only the communfties^ut
p^o,ec, ,s i,'so soiicited ,n ,he
i
r
r
«-
Evaluation typologies
JnnT°K t0?H9 is inherent|y an internal activity, evaluation is an external activity usually
done by those external (mdiViduals/agencies/institutions) to the project. Generally
speaking, evaluation is external evaluation. However, the project implementers may
dertake evaluation all by themselves. In such a case evaluation is internal evaluation
Sam6odHi ^seared cZ Communications
5
e
r '55’
a
1
?a
Reading Material 3
Monitoring and Evaluation of Development Projects
nr'^ppTf0'1 per Se 'S a less frecluent activity generally undertaken at completion of a
project for assessment of attainment of objectives. This is the post-oroiect
nnct r ta
evaluation. For a longer duration project, it may be !eqS thTthe
"of
achievement is assessed half-way through the project. If so, a mid-term evaluation is
undertaken Another scenario would be when on a more regular basis it is thouaht
necessary to assess the achievement of objectives. Then, it is also poSbfe to assess
I
™S 'S
I
de!*"
-<
ixt; xxx
1 ■
a 1
S'dXXXXXr18”-"can ,he""’XXXK
1i
xxxxx
Indicators
Central to M&E are the indicators therefore the first step for desionino
monitoring
system or evauation is development of indicators. Indicators are units^fa information
the^arioi
! th^ documents changes in the specific conditions. With respect to
With respect
to
the various M&E levels specific indicators need be developed. Thus there
would ha
different set of indicators for goals, different for objectives and outputs and activities
Also, for one particular level there can be more than one indicator.
Si’S SLVS1X XSXSTa,' ctnte
ScatS wouto
^h’ SXd’SSe
SMART and SPICED Indicators
r.
ffi
s
SamSodHi ^seared cZ Communications
c
S
G
6
)
-
L
Reading Material 3
Monitoring and Evaluation of Development Projects
Designing indicators
Be it SMART or SPICED, an indicator has to document change and therefore any
indicator finalized should essentially be able to capture change in the condition that
being assessed using the indicator. A good indicator would therefore be:
■
■
■
■
■
i
•
Simple: As all the good things in the world are
Measurable: would provide a metre for depicting change
Precise: Defined in the same way by all
Consistent: would measure the same thing and would not change over time
Sensitive: Would be able to capture the smallest amount of change in the
target condition
For designing indicators, the first step is to brainstorm for identifying candidate indicators
for a specific condition. Once we have listed a number of indicators for a given specific
condition, the next step is of assessment of each of the indicator with respect to the
characteristics of a good indicator. So we see that whether the candidate indicator is
simple, measurable, precise, consistent and sensitive or not. Candidate indicators that
satisfy the criteria can then be taken as the indicators for assessment of that particular
condition. We can also modify the candidate indicators till they are' in tune with the
characteristics of a good indicator.
v-
ۥ
■t
L--
1
f
(
r
(
SamSodhi
search
Communications
7
Reading Material 4
Monitoring and Evaluation of Development Projects
RIGHT INDICATORS
Using and designing indicators that capture development perspectives is challenging. It
is particularly hard to measure things such as participation, non-discrimination, equality
and accountability. Often it is not possible to find a direct measure of the expected result
of programme or project activities on these perspectives
if?:
Assessing and monitoring whether programmes have respected and promoted
participation, non-discrimination, accountability and equality throughout the programming
process is also complicated. Measuring development perspectives requires the use of
disaggregated data to ensure that marginalized and vulnerable groups have been
reached. It also requires greater use of qualitative indicators; for example in monitoring
non-discrimination and equality, it is important to hear the views of marginalized groups,
such as the poor and women, on the programme or project. Equally, measuring the
principle of accountability requires peoples’ opinions of the accountability of the
development practitioners and development agency themselves. Developing these
qualitative indicators and collecting the required data may require extra time and
resources in the evaluation and monitoring stage of the programme. So too will the
highly participatory and transparent process of choosing indicators and collecting the
required data that the principles of participation and accountability demands.
At the ‘beneficiary’ level, indicators will measure such things as how programmes have
impacted people’s participation; whether there is an Increased awareness whether there
is improved access to development programmes and information etc. At the
‘implementer’ level, indicators will measure such things as how project functionaries are
more responsive and accountable; their knowledge and capacities.
This radically changes the way that indicators are used. Whereas traditional
development programmes focused on improvements in material outcomes, there are
other issues such as changes in attitudes, behaviours and relationships; good
governance issues etc. Thus, a mix of quantitative and qualitative indicators need to be
employed to get the real picture. For example, in a livelihood project, quantitative
indicator for income. levels needs to be complemented with qualitative indicator for
vulnerability context, in an HIV-AIDS project; quantitative indicator of surveillance needs
to be supported by discrimination or stigma indicators that are qualitative.
c- i •
Sambodhi fysearcfi cd. Communications
>
8
Reading Material 5
Monitoring and Evaluation of Development Projects
Developing Monitoring Plan
The monitoring plan
Monitoring is not a one-time activity but an on-going process. It requires collection of
information from varied places, at varied times in different forms, from different people
and by different people. Therefore, it makes sense in organizing all the monitoring
efforts. A written document detailing and integrating all these at one place would
tremendously facilitate monitoring efforts in a project. This as well would facilitate
optimization of human and financial resources allocated for the monitoring process.
©
•
Monitoring plan is an outline for the steps you will undertake to ensure that the project is
on track. It lists a project's audience, their information needs, the strategies that will be
used for data collection, the indicators, the methods that will be used to collect data, and
when, by whom, and where data will be collected.
A monitoring plan brings in one place all the aspects of monitoring in a project. It details
the monitoring indicators, their periodicity, method of information collection, from where
the information is collected and by whom the information in collected.
Developing the monitoring plan
The process of developing a monitoring plan is essentially finalizing indicators for
various M&E levels, assigning periodicity to the indicators, finalizing method of
assessment, defining the source location and assigning responsibility for getting the
information. These are thus the components of a monitoring plan. Upon finalization, a
monitoring plan would something like that shown in the matrix below.
(T/??What
(indicator}
a?’
I
Kelppnsibjiity)?:!
1 ’ ’'*■
72^ WOW
.<•
Hs-'d
- ■-••• '.r I ’.A ’• s ‘
What to assess?
The decisions of what to monitor are essentially the decisions of finalization of the
indicators. We have already discussed the process of developed of good indicators. We
will here use the indicators agreed upon for the various M&E levels.
{
When to assess?
The next question to answers for each of the information unit is, how frequently the
assessment is going to be. This is the frequency or periodicity of the indicator.
Periodicity of a particular indicator would be decided by the nature of the information unit
and how frequently the specific attribute would change in the project.
SamSodfii <Rgsearcli & Communications
(
9
Reading Material 5
Monitoring and Evaluation of Development Projects
How to assess?
Once we have decided what to assess and how frequently the assessment has to take
place, we move to finalizing the method of assessment. There can be a number of ways
in which a measurement can be taken. Out of these several methods available, we have
to select what is best suited for the project. Few of the things that need be kept in mind
while arriving at the best possible method are:
0
Reliability & Accuracy: Reliability in measurement context relates with repeatedness. Will
the method lead to same results when the process is repeated again and again?
Whereas, accuracy refers to, how close the measurement results would be to the actual
value.
Capabilities: The next critical question in selecting an appropriate method is to keep in
mind the capabilities, both of the personnel as well as technological capabilities available
with the project.
Cost-effective: And finally we have to see, what kind of cost-implications the selected
method is going to have on the project. The idea is to optimize resources and therefore,
a method that builds on the human resource and capabilities of the project as well is
cost-effective has to be chosen for assessment of a specific information element.
Where to assess?
Here we specify from what particular location the information is to be collected. Based
on the indicators there would be different sources like villages, training centres, project
offices, other department offices etc. For each of the indicator we specify this.
Who will assess?
We now assign the responsibility to specific personnel for fetching this piece of
information on a regular basis. Responsibility of the person in question would be for
gathering *the required information from the specific location at the decided periodicity. It
also makes sense to assign responsibilities for supervising or overseeing data collection.
Designing monitoring reports
a
Once we have finalized the monitoring plan, the next step’is to design various Monitoring
reports. A well designed monitoring plan can very easily be translated into various
reports that the monitoring plan would require.
&
e
The first step in designing reports is to club all indicators that are to be assesses at one
place.
i
Next, segregate all indicators from this list, the one having same periodicity.
Transpose the rows to columns
: V:;
you have the monitoring reports ready
Implementing the monitoring plan
♦
We now have the monitoring plan ready for implementation. For implementation of the
monitoring plan, the first step is to orient the monitoring teams to the project framework
in general and monitoring framework in particular. A training workshop or workshops of
all those concerned with monitoring (those collecting information at all levels and those
SamGodhi ^search dt. Communications
10
Reading Material 5
Monitoring and Evaluation of Development Projects
I
overseeing information collection) may first be organized. During the workshops, shared
understanding of the M&E system and monitoring plan has to be developed among the
monitoring teams. During these workshops, the understanding of the various periodic
reports and method of information collection also has to be developed. A pilot round of
the entire information collection process can greatly contribute towards having an
efficient and effective monitoring system. This will ensure common understanding and
therefore bring in uniformity in the monitoring process.
Once the monitoring system is ‘rolled-out’ trouble shooting and streamlining would be
the next step. Field data collection may also require some hand-holding support. Further,
there may be modifications alterations in information collection methods or even some of
the reports need be modified. Once, the final system is in place and continuously
providing information as desired, we can start analyzing the monitoring information.
L
V
<<u-
c
If
♦
I
Sam6odEi ^fsearcH
i
|r
J
(
Communications
11
Reading Materia! 6
Monitoring and Evaluation of Development Projects
3
DATA BASICS AND MEASUREMENT SCALES
i
A variable is defined as the attribute of a case, which varies for different cases. Its
variability is usually captured in measurement scale, varying between two scale values
to potentially an infinite number of scale values for binary scale or continuous metric
scale.
©
-
fe-
Research as a process is nothing but an attempt to collect information about variable of
interest and assessing change in that variable as a function of internal and external
environment. The process of grouping observation about variable of interest in a
systematic and coherent way provides us data, which could be qualitative or quantitative
in nature, depending on the nature and type of observation. For sake of simplicity, as of
now we can segregate qualitative data by words, picture or images and quantitative data
by numbers on which we can have basic mathematical operations.
Returning to definition of variable, instead of defining variable as an attributes of a case,
some researcher prefer to say that the variable takes on a number of values. For
example, the variable gender can have two values, male and female. Variables can be
further classified into three categories:
■ Dependent variable: Dependent variable is also referred by some researcher as
response variable /outcome variable. It is defined as variable, which might be
modified, by some treatment or exposure, or a variable, which we are trying to
predict through research.
■ Independent variable: Independent variable also referred as explanatory variable
is variable which explains any influences/ change in response variable.
■ Extraneous variables: Extraneous variables are variables that are not part of
study as per conceptualized design, but may affect the outcome of a study.
Types of Data
Data can be broadly described into a) Qualitative data and b) Quantitative data based on
objects they measure.
Qualitative data, measures behavior which is not computable by arithmetic relations and
is represented by pictures, words or images. Qualitative data are also called categorical
data, as they can be classified in categories based on or class, individual, object, or
process they fall in.
t:-
•
V
Quantitative data are numerical records that result from a process of measurement and
on which basic mathematical operation could be done for example though we may
represent gender variable values, male and female as 1 and 2, but aS no mathematical
operation could be done on these values (adding 1 and 2 doesn’t make any sense), as
data remains qualitative in nature.
Quantitative data can be further classified into metric and non-metric data based on the
metric properties defining distances between scale values. Scales are of different types
and vary in terms of the ways in which they define the relationships between scale
values. The simplest of these scales are binary scale where there are just two
categories, one of the cases that possess those characteristics and one for the cases
that do not. Nominal scale and ordinal scale can have several categories depending on
I
?-
SamSodhi (R^searcfi & Communications
12
Reading Material 6
Monitoring and Evaluation of Development Projects
the variables of interest for example in case of gender we have only two categories such
as male and female, but in case of occupational qualification, we can have several
categories, depending on the way we decide to define categories.
1) Non-Metric Data
Data collected from binary scale, nominal scale and ordinal scale are jointly termed as
non-metric data i.e. that is, and they do not possess a metric with which distance
between scale values can be measured.
F;
©
©
2) Metric Data
Though for some scale, there is a metric with which we can define distances between
scale values. Hence quantitative data can be further classified into Metric and Non
metric based on presence of metric scales used to collect data.
Metric data can be further classified into two groups: a) discrete data and b) continuous
data. Discrete data are countable data, for example, the number of students in a class.
Continuous data, when the variables are measurable, are expressed on a continuous
scale, for example, measuring the height of a person.
L
Measurement scales
There are four types of scales that are used in measurement: nominal, ordinal, interval,
and ratio scales. In fact they follows a hierarchy of measurement scales, nominal being
at the lowest rung of hierarchy and even application of statistical procedure are classified
in relation to scale used. They are categorized into two groups: categorical and
continuous scale data, wherein nominal and ordinal scales are categorised together as
categorical data whereas interval and ratio scales are grouped together as continuous
data.
Nominal data having unordered scales are called nominal scales for example gender
categories male and female. Categorical data having ordered scales are called ordinal
scale. In case of continuous data, scale-having representing interval data scales are
called interval scales and data having both equal intervals and an absolute zero point
are called ratio scales.
€->
©
■■
e
©
k. ■
©
©
©
a Nominal variables: The values of the nominal variable data have no numeric
meaning as no mathematical operation except counting can be done on the data.
They are infact used to classify whether the individual items belong to some
distinctively different categories. For example, all we can say is that individuals are
different in terms of variable for example in terms of gender or caste, but more than
that nothing can be said about the variable and typical examples of nominal variables
are gender, race, color, city, etc.
b Ordinal variables: Ordinal variables unlike nominal variables allow us to rank the
items we measure in terms of order and we can specify that higher order item
definitely represent more of the quality represented by the variable, but still we can’t
tell how much more than other item. A typical example of an ordinal variable is the
rating assigned to impact of programme i.e. excellent, average and poor. Now we
can say that x % rated programme to be excellent and y% rated average and
another z% rated poor, but researcher can’t say for sure about the difference
Sambodhi Qtysearcft c£ Communications
13
Reading Material 6
Monitoring and Evaluation of Development Projects
between excellent and average is same as that of average and poor. In case of
ordinal variables, only certain mathematical variables such as greater than or less
than are feasible and only measures such as median and range can be calculated.
c Interval variables: Interval variables provide more flexibility in terms of
measurement as it allows us not only to rank the measured items but can also help
in quantifying the sizes of differences between them. For example, temperature, as
measured in degrees Fahrenheit or Celsius, constitutes an interval scale. We can
say that a temperature of 80 degrees is higher than a temperature of 40 degrees, but
still we can’t say 80 degree is twice as much as hooter as 40 degrees. Another
example could be of time system such as B.C or A.D, wherein we measure time
taking B.C. or A.D as reference (initial point of reference is assumed as zero), but it
doesn’t mean that time doesn’t exist before B.C or A.D. We have simply constructed
a reference scale tcrmeasure time, which doesn’t have a true or rational zero.
d Ratio variables: Ratio variable measured by scale not only have equidistant point
but also have a rational zero. Thus in addition to all the properties of interval
variables, they feature an identifiable absolute zero point. Typical examples of ratio
scales is Kelvin temperature scale, wherein not only can we say that a temperature
of 60 degrees is higher than one of 20 degrees, we can also specify that temperature
of 60 degree is thrice as high as 20 degree. Most of the variables we use to measure
in field situations conform to ratio scale properties, though most statistical data
analysis procedures do not distinguish between the interval and ratio properties of
the measurement scales.
■■
'<
©
t
V-
j ty
r
I i
Sambodfii Q&scarcfi <SC Communications
14
J
Reading Material 7
Monitoring and Evaluation of Development Projects
SAMPLING TECHNIQUES
PROBABILITY SAMPLING METHODS
Probability sampling techniques are those that involve the random selection of study
units by chance. For this to be possible, a sampling frame must exist. There are a
number of probability sampling techniques and any of them could be used in a study
depending upon the study problem and the purpose of the study. These comprise;
Simple Random Sampling (SRS)
This is the simplest form of probability sampling. It involves:
■
■
■
•g?
making a numbered list of all units in the study;
deciding the size of your sample;
choosing the number you require.
For example, suppose we are faced with a problem of selecting 60 households in a
community of 360 households to learn more about their attitudes towards the use of
antibiotics in cough. We could proceed by first giving each household a number. These
numbers are then written on small pieces of paper and placed in a box. Numbers are
then picked from the box up to the required total, making sure that that the box is well
shaken each time to ensure the principle of equal and random selection.
Systematic Sampling
The principle involved in using this method is similar to the SRS, but with an added
condition that the selection of units is based on regular intervals starting from a
determined point. To use the example above, we can obtain our interval by dividing 360
by 60 which gives 6. Once the sample interval has been determined,'the first unit is then
drawn between 1 and 6.
Afterwards, every sixth case must be drawn until the total sample of 60 has been drawn.
The technique is particularly useful where large numbers are involved such as the
sampling of prescription records. For instance, if we need 30 prescriptions from, a
register in a health facility which contains 50 pages of 25 lines per page, the sampling
interval must be calculated by dividing the product of 50 and 25 by 30 to obtain 41.7.
Based on this figure, every 42nd prescription would be selected for the study.
Stratified Sampling
This is an alternative method to systematic sampling and preferable where the sampling
frame contains distinct populations. For example, assuming our antibiotics study covers
an entire district, it would be necessary to ensure that both rural and urban facilities are
adequately covered. This would not be possible using the SRS or systematic sampling
techniques.
-
Alternatively, using a procedure that groups study units who are similar in certain
characteristics into strata ensures that all relevant groups are appropriately covered. In
drug use studies stratified sampling could be used to sort facilities into various levels of
care, i.e., community clinics from health centers and both from district hospitals. When
the method is used, the final stage of selecting the units is also based on the principle of
equal selection by chance.
Cluster Sampling
Assuming our district is a large one with a landscape that makes travelling around
Sambocffii ^seared dC Communications
15
Reading Material 7
Monitoring and Evaluation of Development Projects
facilities difficult, then cluster sampling offers a better alternative. This involves selecting
clusters or groups of study participants or units rather than on individual basis. Clusters
may be villages or families or a specified group of patients or providers.
For example, if we want to select two facilities for study in a geographically difficult area,
then we could select a cluster of two facilities by randomly selecting a single facility and
then selecting another one close to it. The method saves time in data collection.
Multi-Stage Sampling
This is a combination of al! the methods described above and is very* useful if the
purpose of the structured observation method is to generalize study results from diverse
populations. As the name implies, it involves sampling in two or more stages as follows:
1. Forming clusters and selecting a random sample of these.
2. Making a separate list of all study units within each of the selected clusters.
3. Selecting study units separately from each cluster in the sample.
4. Interviewing selected study units.
For example, if we intend to select 12 health facilities in a district with 36 facilities, we can
first group them into the various clusters or levels of health care such as community
clinics, health centers, and hospitals. We can then randomly select our 12 facilities from
the 3 groups. This is two-stage sampling. If we want to select 30 prescriptions from
each facility, we can continue from this stage with the process described in selecting
every 42nd prescription under systematic sampling. This is three-stage sampling.
NON-PROBABILITY SAMPLING METHODS
There are two common types of non-probability sampling, namely convenience or
purposive sampling and quota sampling.
Convenience Sampling
Convenience sampling is that in which the study units that happen to be available at the
time of data collection are selected for purposes of convenience. Most clinic based
studies use this method. It is the common method for selecting participants to a focus
group discussion.
©
V:-’
Quota Sampling
Quota sampling is used when it is considered that convenience sampling would not
provide a desired balance of study units. For example when assessing patient
understanding of how to take drugs, a quota sample could be used to ensure that both
males and females are covered in the study.
Non-probability sampling methods are less representative than probability sampling
methods and thus inappropriate if the objective of a study is to generalize findings. They
are often used in qualitative methods because of their flexibility in adapting to local
situations and settings.
Sambo(flu ^search
Communications
16
v
r
f
Reading Material 8
Monitoring and Evaluation of Development Projects
ASSESSING IMPACT OF COLLECTIVE ACTION
Serial |
||
||
|
District
Block
□□
RESULT*
□□
□□ □□ OD □□
VISIT
DATE
day
month
□□□□
year
day
month
□□□□
year
01
Completed
02
03
Postponed
04_________
Others(Specify)_
Now, I would like to ask you some questions about various stages of group development and its impact
on your livelihood. These questions would help us in understanding the impact of collective group
action on poverty reduction and sustainable livelihood.
Name of Client
_________________________
Sex: Male -1 or Female -2
Address
________
Name of group
_________________________ ________________
Number of group membersQLJ No. of male membersOO No. of female members,
Group formation date,
101
©
lOLJLJ
||
|
MMYY
Section 1 Group Formation .____________________
When did you join the group? Please specify the entry date?
Number of members in the group, when you joined and now?
102
'ti?
What was the first saving that you deposited when you joined the group?
103
What are the factors, which motivated you to join the group?
104
105
"106
107
ToT
I,
What expectations (in terms of services) you had from the group
when you joined?_______ ___________________________
Did you had some specific monetary expectations when you joined
the group?_______
If yes, what was the moneatry expectation (monthly) ?
What are the objective you have in mind , when you joined the
group?
(
Sambodfd ‘Rgsearrfi
z
r
Communications
17
Reading Material 8
Monitoring and Evaluation of Development Projects
r'--
109
Who approached you/ or who did you approach to join the group?
110
Who all did you consult before joining the group?
111
Who finally decided that you should join the group?
112
Please specify whether internal lending has increased or
decreased?___________________________________
Have more groups been formed in your village since you have
joined the group?______________________________
If yes, How many?
113
114
f-':.
V
Section IV Group meetings/records_____________________
101
What is the frequency of group meeting?
402
When did you have your last meeting? Or how may days back?
403
How many members participated in the last meeting?
404
What are the issues usally discussed in such meetings?
405
Who decides when to have meetings?
406
When do you come to know about meetings?
407
When do you come to know if there is a change in the meeting
_ date?______________________________________ ___________
408
Has there been any change in members in the group, since you
joined the group? ________________________
409
If yes how many members dropped out since you joined the group?
C-:-
410
How may new members have droped?
411
What do you think is the reason for their dropout?
412!
What are the records that are managed in your groups?
413‘
Who maintains the records?
414
Does the group pay the member/accountant for his/ her services?
415
If yes, How much?
416
Can you get the information related to recors whenever wanted?
v,
Sambodfn fysearcfi d Communications
f
r
18
Reading Material 9
Monitoring and Evaluation of Development Projects
ASSESSING IMPACT OF COLLECTIVE ACTION
Serial^ ]Q[
i
District
Block
RESULT*
VISIT
DATE
□□
□□
□□ □□
□□□□ □□□□
month
day
year
month
day
•
1
year
Completed
01
02
03
04
Postponed
Others(Specify).
Now, I would like to ask you some questions about various stages of group development and its impact on
your livelihood. These questions would help us in understanding the impact of collective group action on
poverty reduction and sustainable livelihood.
Name of Client
Sex; Male -1 or Female -2
Address.
Name of group
tv
Number of group members: |
!Uv-
|
No. of male members
I|||
Group formation date:
|__ ||__ | I__ ___ |
*
t
||
SamGodJii ‘Research
M M
Communications
No. of female members
Y Y
19
Reading Material 9
Monitoring and Evaluation of Development Projects
Section 1 Group Formation
Qno
101
When did you join the group? Please specify the entry date?
104
MM
When you join
What was the first saving that you deposited when you joined the group?
Amount in Rs. Fjl
What are the factors, which motivated you to join the group?
To earn profit
To improve substience
To have a social standing
To become part of community
Others______________
Credit/ loans
Savings
Linkage to livelihood activity
Linkage to Government
Programs/ Schemes
Any Other__________
Yes
1
No
2
What expectations (in terms of services) you had from the group
when you joined?
105
106
107
Did you had some specific monetary expectations when you joined
the group?_________________ _______ ___________
If yes, what was the moneatry expectation (monthly) ?
108
What are the objective you have in mind , when you joined the
group?
109
Who approached you/ or who did you approach to join the group?
110
Who all did you consult before joining the group?
&
L-
(
(
C' ■
YY
Number of members in the group, when you joined and now?
102
103
□□ □□
SambodKi (Rgsearcft dC Communications
Now
l|
I
1
2
3
4
8^
1
2
3
4
5
8
>108
"Rs. 50-100
F"
Rs. 100-200
2
Rs. 200-300
3
Rs 300-400
4
Rs 400
5
Other____________ ,
__ 8_
To be able to contribute to
1
household income
To be able to spend more on myself
2
To be able to spend more on child
3
education
To be able to spend more on
4
family health
Other__________________ _
8
''IGO workers
1
Other group members
2
Group's president/secretary
3
Government Official (ANM/
4
^atwari/ Panchayat Secretary etc.)
Any Other (Specify)_____ _____
8.
Self
1
Husband
2
Head of Household
3
Mother-in law
4
Father-in-law
5
Friends/realative
6
Others
8
20
Reading Material 9
Monitoring and Evaluation of Development Projects
111
112
e
Who finally decided that you should join the group?
114
Please specify whether internal lending has increased or
decreased?
Have more groups been formed in your village since you have
joined the group?
____________________________
If yes, How many?
115
What has been the reason for the formation of such new groups?
113
Self
Husband
Head of Household
Mother-in law
Father-in-law
Friends/realative
Others_________
Yes
1
No
2
1
2
Yes
No
1
2
3
4
5
6
8
> 201
When you joined
Success of our group
Initiative taken by NGOs
Efforts by community
Support of Panchayats
Initiative on own
Others
Now
1
2
3
4
5
8
Section IV Group meetings/records
401
What is the frequency of group meeting?
402
When did you have your last meeting? Or how may days back?
403
How many members participated in the last meeting?
404
What are the issues usally discussed in such meetings?
405
Who decides when to have meetings?
■
j @
Once in every week
Once in a fortnight
Once in a month
Once in two month
Once in six month
Whenever needed
Others
1
2
3
4
5
6
8
months back
Credit lending decision
Saving issues
Repayment issues
Defaluters issue
Enterprise realted issue
social issue
Others
1
2
3
4
5
6
Group President
Group secretary
Other group member
Not specific
Pre decided group norm
Others
1
2
3
4
5
8
8
V
Sambodhi fyseardi & Communications
I-
21
Reading Material 9
Monitoring and Evaluation of Development Projects
406
407
©
When do you come to know about meetings?
When do you come to know if there is a change in the meeting
date?
408
Has there been any change in members in the group, since you
joined the group?_______________ ___________________
409
410
If yes how many members dropped out since you joined the
group?___________________ _________________
How may new members have draped?
411
What do you think is the reason for their dropout?
A day in advance
A wek in advance
On same day
Pre decided
Others_______
A day in advance
A wek in advance
On same day
Pre decided
Others
Yes gkj
1
No ugha
2
1
2
3
4
8
1
2
3
4
8
>412
Couldnt repay the loan
Have problems with group
members
Family Problems
Community problems
Members were not sincere
Others
3
4
5
8
1
2
412
What are the records that are managed in your groups?
Attendance Book
Minutes Register
Cash Book
*
Ledger
Individulal Member Passbook
Stock Register
Any Other____________
1
2
3
4
5
6
8_
413
Who maintains the records?
External facilitator
Local boy or girl
One of the members
Other
1
2
3
8
414
Does the group pay the member/accountant for his/ her services?
Yes
No a
415
If yes, How much?
Rs.
416
Can you get the information related to recors whenever wanted?
Yes
No
1
2
1
2
■
i
SamSodhi
(
search
Communications
22
Reading Material 10
Monitoring and Evaluationof Development Projects
Guidelines for Data Collection
a
Participatory methods in poverty and vulnerability assessment are most commonly
associated with the spread of diagramming and visual techniques which began to be
developed in the 1970s. These originated in a number of scientific disciplines interested in
analysis of complex systems: biological science, ecology, agricultural economics and
geography. It became increasingly important to work with farmers to develop more
sophisticated models to explain their responses to development programmes. The
increasing influence of applied anthropology in development agencies from the 1980s also
led to greater awareness of the need for a more sophisticated understanding or poverty,
social processes and grassroots perspectives on development. By the end of the 1980s
diagramming techniques bringing together the insights from these different disciplines were
combined into a flexible methodology commonly referred to as Rapid Rural Appraisal (RRA).
By the mid 1990s it had become increasingly evident that the mechanical application of
these techniques was often failing to really reach and capture the views of poor people,
particularly women, children and socially excluded. There was renewed interest in
methodologies for participation drawing on earlier traditions of participatory action research
which had been long established as an integral part of many grassroots organisations in the
South.
In India for example SEWA and other women's organisations based their
programmes on the findings of focus group discussions in the 1970s arid T980s.
Initially the term RRA (Participatory Rural Appraisal) was used to describe the bringing
together of RRA and activist research. It was emphasized that the most important aspect
was not the diagramming tools but their flexible application based on a number of underlying
principles:
• Embracing complexity and seeking to understand it rather than oversimplifying reality in
aocordance with predetermined categories and theories
• Recognition of multiple realities to be taken into account in analysis or action.
• Prioritizing the realities of the poor and most disadvantaged as equal partners in
knowledge creation and problem analysis.
• Grassroots empowerment i.e. aiming not only to gather information, but to make the
assessment process itself a contribution to empowerment through linking grassroots
learning and networking into policy-making.
A
More recently the term Participatory Learning and Action (PLA) has become preferred
because it more effectively incorporates the underlying human rights tradition through
emphasizing the importance of:
‘changing from appraisal to learning and hence moving away from the use of participatory
methods as an extractive process by outsiders to a sustainable learning process involving
different stakeholders as equal partners. The importance of relating learning to action
incorporating programme and policy improvement as an integral part of the learning
process. ’
1-
v.
Participatory methods are therefore a diverse and flexible set of techniques for
representation and stakeholder involvement characterized by a set of underlying ethical
principles. There is no one set of techniques to be mechanically applied in all contexts for all
participants. There is on the one hand a set of visual tools to be flexibly applied to assist the
synthesis and analysis of information, which can be used in group settings and also as part
of individual interviews. On the other hand are a set of guidelines for facilitating participation
and negotiation in focus group discussions and workshops bringing together different
stakeholders. These may or may not make substantial use of visual techniques. Generally
>
Sambodlu ^seared & Communications
r
\ f
23
Reading Material 10
Monitoring and Evaluationof Development Projects
both visual techniques and participatory facilitation are combined in different ways. The
emphasis is on innovation and creativity in adapting previous practice to new contexts and
needs.
Participatory methods have the potential to bring together information from a diversity of
sources more rapidly and cost effectively than quantitative or qualitative methods alone. A
particularly important contribution of participatory methods is their role in capacity building of
the different stakeholders. Where sufficient attention is paid to this participatory methods
have the potential to build up the necessary information resources and networks for a
learning process which will be sustainable beyond the term of the one particular impact
assessment. This can include group-level learning, participatory monitoring and evaluation
and multi-stakeholder networks for policy assessment. Thus the costs of integrating
participatory methods can be seen as a contribution to development in themselves, leading
to much longer term benefits. These benefits have included reducing the costs of project
administration, reducing default in micro-finance programmes, making training programmes
more attractive to clients prepared to pay for services. This is therefore also a contribution to
longer term financial sustainability of interventions.
v..-
Participatory data collection methods have a number of potential key contributions in
increasing:
• Stakeholder representation
• Reliability of understanding of development processes
• Credibility of practical inference
• Relevance of impact goals and indicators
However participatory methods also face a number of inherent challenges that need to be
taken into account. Some of these are common to many methodologies, some are due to the
visual tools and some to the participatory process. The degree to which participatory
methods realize their potential contribution depends critically on how carefully they are used
and in what context. As noted above, participatory methods are not a fixed set of
mechanistic tools but a diverse range of possible techniques which need to be flexibly
adapted to particular situations and needs. In some cases problems can be resolved through
innovation in the methods themselves. Sometimes limitations can only be addressed through
triangulation with other quantitative and qualitative methods. Some of the most significant
challenges faced in using participatory methods are outlined below:
•
•
O
•
•
Standardization: Because of the erpphasis on locally identified indicators there are added
challenges for comparative assessment. It is however possible to agree on common
impact goals to be applied across contexts and then weigh locally specific indicators by
which they are to be measured.
Sensitive information: It cannot be assumed that the participatory process will
necessarily yield in-depth information on sensitive issues. Including techniques like role
play has been found useful in highlighting dimensions of power relations. In other
contexts this sort of sensitive information may require in-depth investigation by
qualitative methods.
Emphasis on consensus may serve to privilege dominant views and further marginalize
the most disadvantaged through giving the appearance of participation. Explicit attempts
are needed to include the very poor, women and socially excluded and also to ensure
that they are not only present, but also their voices are heard.
Skills, contextual knowledge and experience required to meaningfully record and analyze
the visual outputs and participatory process is probably the greatest challenge of all for
participatory methods. When badly conducted and recorded the outputs often appear
arbitrary and meaningless, albeit rather quaint, to people who were not present during
the meetings. This may seriously jeopardize their reliability and credibility. Interviewer
Sambodfa Q^searrh
Communications
24
Reading Material 10
Monitoring and Evaluationof Development Projects
subjectivity inevitably influences the recording and analysis of both quantitative and
qualitative information. In using participatory methods to some extent interviewer bias
can be countered by the greater numerical strength of those interviewed and crossed
checked by them. It is nevertheless crucial when using participatory methods to keep
detailed notes on the process of investigation, how the findings compare with information
from other sources, who participates and who does not and how particular diagrams are
generated.
r:i
c-
Guidelines for conducting Household Surveys
Preparation for household survey starts by orientation of professionals through internal meetings and
workshops where all professionals associated with the project are briefed on the objectives,
methodology, research techniques to be followed, study instruments and the guidelines for the
training of field staff. This creates a common understanding among all professionals. These guidelines
present in a step-by-step form, the activities to be done by a core team of professionals lead by a
Project Coordinator before, during and after a survey
Recruitment of field staff
The project Coordinator in association with the core team members may look after the
recruitment of field staff. The recruitment should be made against the desired qualification
and experience in conducting surveys. Most surveys take in people at two levels i.e.
Supervisors and Investigators depending upon the scale, geography and sample size. Each
Supervisor heads a team of 3-8 investigators during the fieldwork for data collection.
However, the core team mernbers can also take up this role in smaller surveys. It is
advisable to recruit in excess (about 20 percent) to account for the attrition at various stages
of the field work.
It is important to point here that supervisors and investigators should be recruited based on
their educational qualifications as well as the ability to spend long duration in the field and
their prior experience with similar kind of surveys. It is advisable to conduct another round of
interviews for final selection of supervisors and field investigators after completion of training
and prior to initiating the data collection.
a
t-
Translation of tools in local language
While translating survey items one must consider three dimensions of the problem: semantic
equivalence, conceptual equivalence, and normative equivalence of items (Behling and Law,
2000). Among these semantic equivalence is the most widely used. It is sought through the
translation/ back translation method, having independent translators translate from one
language to another, and then back again, to see if the original and re-translated item
remains the same. It is quintessential to ensure that translated schedule is available before
training starts.
Training Investigators and Supervisors
The professionals involved in the study should brief the recruited field staff about
interviewing technique, procedures for filling survey forms, time schedule and each item in
the questionnaire followed by mock exercise between the participants. Such training
sessions for the survey team including investigators, supervisors and other executives
involved in the survey may also discuss sampling design and expected data quality.
Following are a few points that may be followed at the time of briefing/ trainings to improve
effectiveness of such surveys:
t.
I
(
r
(
SambodJH ^search d Communications
25
Reading Material 10
Monitoring and Evaluationof Development Projects
•
•
•
Grading: Investigators may be graded (even screened out, in certain cases) based on
skill shown during training and this should be kept in mind for deciding composition of
survey teams.
Field-testing: Field-testing exercises aim to acquaint investigators and supervisors to the
field situation. It is advisable that field-testing be done in a few non-selected large
villages where all tools can be tested with the target respondents in quick time.
Debriefing: This helps minimize interviewer bias by making questions simpler and clear.
It tries to assess whether interviewer has understood the question correctly. Trainers
along with supervisors should be present during debriefing sessions to sort out problems
encountered in canvassing schedules, questionnaires and other instruments. Field
testing and debriefing also serves as a medium for screening out investigators.
Field Work
The fieldwork for the study should be initiated immediately after training of field staff is
completed. To prepare for fieldwork, the field executive shall:
• Ensure field plan for main survey is ready keeping in mind the scheduled time
• Ensure that all supervisors have requisite number of schedules and material
w
Also, prior to initiating the fieldwork, an appropriate sample of respondent categories must
be selected using the sampling technique specified in the methodology. The Project
Coordinator and core team members may take the responsibility for this exercise.
During the fieldwork, assigning work to individual supervisors and interviewers should take
into account the linguistic competence of individuals and ensure that there is an equitable
distribution of the workload. Supervisors may assign more work an investigator can actually
do in one day to account for the respondents who may not be available at the time of the
investigator’s visit.
Monitoring Quality of data collected
Controlling the quality of the data collected is an important function of the supervisors and
the core team. This can be done by observing interviews and carrying out field editing. By
studying the filled up survey instruments regularly It can be ensured that the quality of the
data collected remains satisfactory throughout the survey. It may be necessary to observe
the interviewers more frequently at the beginning of the survey and again toward the end. In
the beginning, the interviewers may make errors due to lack of experience or lack of
familiarity with the tool; these can be corrected with additional training as the survey
progresses. Toward the end of the survey interviewers may become fatigued in anticipation
of the end of fieldwork.
The core team members and supervisors should expect to spend considerable time
monitoring and helping investigators at the start of fieldwork. If it is felt that the data quality
is not up to the desired standards, further data collection may be stopped until all issues and
problems have been fully resolved. In some cases, an investigator may fail to improve and
will have to be replaced.
Collection of high-quality data is crucial to the success of the survey. Supervisors may adopt
one or more of the methods such as editing, spot checks, back checks and re-interviews to
ensure data quality. This is especially important during the initial phases of fieldwork, when it
is possible to eliminate interviewer error patterns before they become habits.
Minimizing non-response
One of the most frequently encountered problems in most sample surveys is non-response
i.e. failure to obtain information from respondents selected in the sampling plan. Non-
i V,
SamGodfii ‘Research eSC Communications
26
Reading Material 10
Monitoring and Evaluationof Development Projects
response is generally classified into some basic types as mentioned in below. Various ways
of dealing with these types of non-response are also outlined.
•
•
•
The interviewer is unable to locate the selected respondent such as household or person
Respondent is temporarily absent at the time of call
Respondent refuses to be interviewed
It is suggested that multiple (at least 2-3) visits be made to locate the respondent.
Sometimes it may be necessary to visit at mealtime, early morning, evening, weekend
depending upon the availability and convenience of the respondent. The procedure for
callbacks may also be followed.
V?'
Monitoring the number of refusals reported by each investigator may also help understand
the issues in non-response. If an investigator reports an unusually high number of refusals, it
may indicate that she/ he gives up too easily or explains the survey inadequately..Refusals
may stem from misconceptions about the survey or other prejudices. The surveyor must
consider the respondent’s point of view, adapt to it, and reassure her/ him.
Guidelines for conducting Focus Group Discussions (FGDs)
FGDs are semi-structured group meetings during which participants contribute to the
generation of data on specific questions of concern to communities, stakeholders, projects or
policies. The meetings can serve as a forum to address a particular issue, can build
community consensus about implementation plans and also provide an opportunity to cross
check information with a large number of people in a short period of time. An FGD is a group
discussion of approximately 6-12 persons guided by a facilitator, during which group
members talk freely and spontaneously about a certain topic. An FGD is a qualitative
research method to obtain in-depth information on concepts, perceptions and ideas of a
group. An FGD aims to be more than a question-answer interaction. The idea is that group
members discuss the topic among themselves, with guidance from the facilitator.
’ f-V-
FGDs can be a powerful research tool which provide valuable information.in a short period of
time and at relatively low cost. Implementation of FGDs is an iterative process. Each focus
group discussion builds on the previous one, with a slightly elaborated or better-focused.set
of themes for discussion. Provided the- groups have been well chosen, in terms of
composition and number. One of the preemptive conditions in most cases is homogeneity of
group. Homogeneity among group members means that they have common concerns. But it
is the similarity of participants' orientation toward the issue at hand which allows for
information to be shared freely and for deeper insight into the issue to be raised.
FGD should not be used for quantitative purposes, such as the testing of hypotheses or the
generalisation of findings for larger areas, which would require more elaborate surveys.
There are different roles in an FGD such as organizer (leads planning and developing
questions), recruiter (invites participants), moderator, recorder/ reporter and an analyst (to
summarize the data and prepare reports). FGDs can be conducted better with at least two
researchers wherein one of the members of the research team acts as a ‘facilitator’ or
‘moderator’ while the serves as a ‘recorder’. The facilitator should preferably be as close as
possible to the participants, in their characteristics. Any FGD requires the facilitator to have
good understanding of local conditions. Communities are seldom homogeneous. There are
always differences between community members, for example in education, political power,
gender, economic status and ethnicities. These differences get reflected in their perceptions
V'
w
Sambocflu fysearcfi cZ Communications
9
27
Reading Material 10
Monitoring and Evaluationof Development Projects
of the problems they suffer from and possible solutions. A researcher must be aware of
these differences, and should select group members accordingly.
Following are the steps and considerations in conducting FGDs:
©
Recruitment of participants
• Individuals who have the characteristics, experience or knowledge to provide rich
information on the topic may be invited for the FGD.
• It is advisable to limit the size of the group to 6-8 members. It may be noted that an FGD
must have enough people to generate diverse ideas but not so many that they do not get
a chance to share.
• Power differentials among participants must be avoided and all participants should feel
comfortable talking with one another.
Physical arrangements
• FGDs are best held in familiar and neutral settings to encourage participants to freely
express their views.
• Sitting in a circle ensures mutual visibility facilitates freer exchange of ideas.
< The facilitator must also make sure that there are minimal disturbances, sufficient
quietness, adequate lighting, etc.
Skills required for facilitating an FGD
• Moderators must be good at listening and make the participants feel comfortable. For
groups of people who are used to being in powerful positions, the moderator must keep
the group on track and control dominant participants.
• The moderator should be prepared to ask a set of predetermined questions designed to
derive the information needed.
• The moderator must create an open environment to assure participants that it will be
safe to talk in the focus group.
Functions of the facilitator
Introducing the session
The first few moments in a focus group discussion are critical. In a brief time the
moderator must create a thoughtful receptive atmosphere, provide ground rules and
set the.tone of the discussion. Much of the success of the moderator can be attributed
to creating an open environment.
<s;s-
c
The recommended pattern for introducing the group discussion includes a welcome, an
overview of the topic, the ground rules and the first question. Self-introduction by the
moderator and the recorder followed by explaining the kind of information needed and
the purpose for which it will be used may help put the participants at ease.
Encouraging discussion
The moderator needs to be enthusiastic, lively and show interest in the groups’ ideas.
Asking the right questions in the right way can encourage participants to express their
views. Most of the questions in FGDs are open-ended of non-directive. These
questions deliberately give the participants as much latitude as possible to express
. their responses. The moderator must avoid asking question that can be answered in
one word such as a ‘yes’ or ‘no’ as this may limit the desired detail. Another type of
questions that may be avoided during FGDs are ‘why’ questions as they may make the
participants defensive. Techniques such as ‘think back’ questions may be used to
obtain information based on experience.
i
Sambodfa ‘Research & Communications
28
I
|
i
Reading Material 10
Monitoring and Evaluationof Development Projects
I
Sometimes new and unanticipated questions may occur to the moderator. There is a
risk these questions during the focus groups, as they might interrupt the sequence of
the planned questions and throw participants off topic. The moderator must save such
unplanned questions and ask them at the end of the discussion.
Encouraging involvement
Facilitators should show involvement and should keep track of way in which discussion
is progressing. In case dealing with a dominant participant, avoiding eye contact or
turning slightly away to discourage the person from speaking, or thanking the person
and changing the subject. Further while dealing with a reluctant participant, using the
person s name, requesting his/her opinion, making more frequent eye contact to
encourage his/her participation.
i
A focus group is working well when the participants begin to talk to one another and
build on one another’s comments rather than continually responding directly to the
moderator. Ideally, participants become engaged, and the focus group becomes a
forum for their own discussion. The moderator should begin to play a less central role
as participants share experiences, debate ideas and offer opinions. Some groups
arrive at this point quickly while others take longer to reach it.
The moderator must also avoid being an expert when asked for ideas or views by a
respondent. It must always be remembered that the role of a moderator is not to
educate or inform the participants.
Controlling the rhythm of the meeting, but in an unobtrusive way
The moderator s job is not to make sure everyone speaks the same amount in a group.
However, everyone should have the opportunity to share. Participants speaking more
must be allowed to continue if they are. adding new and useful information. The
moderator must listen carefully, and move the discussion from topic to topic. He/ she
must also control dominant talkers by thanking them for their input and asking for
others to share. The quiet participants must also be called on as they may be reflective
thinkers and have ideas- to offer. Small pauses can also be used to draw out more
responses.
The moderator should also avoid giving verbal or non-verbal clues of approval or
disapproval. For example, it is often tempting for the moderator to give a broad smile
and nod his/ her head when hearing certain comments. Participants can- quickly spot
this behaviour and assume that more of such ‘approved’ comments are wanted.
w
Sk..-
C
(
Concluding the discussion
At the end of an FGD the reporter needs to summarize the key points of discussion
and ask for confirmation. It is advisable not to summarize the entire focus group but
'"stead 3-5 of the most important points. The moderator may review the purpose of the
FGD and ask participants if any point may have been missed before thankinq them and
concluding the session.
There are no standard time limits but a typical focus group discussion session may last
between 1-2 hours depending upon the group and issues discussed. Generally the first
session with a particular type of group is longer than the following ones because all of
the information is new. Thereafter, if it becomes clear that all the groups have a Similar
opinion on particular topics, the facilitator may be able to move the discussion along
more quickly to other topics which still elicit new points of view.
(
Sambodlii
CiS
search
Communications
29
I /
I
Reading Material 10
Monitoring and Evaluationof Development Projects
i
!
Capturing the data
Thought must be given to capturing the comments of the focus group participants. Multiple
methods are recommended, as no single method is perfect. The resources available and
circumstances should determine the researchers’ choice of methods. An examination of the
alternative methods of capturing data is as follows:
• Memory is the most fallible. Memories of the moderator and recorder fade quickly and
can be prone to distortion. This is always one of the methods used but it should not be
the only method.
• Filed notes can capture meaning quotes plus the content of the discussion. The field
notes of the recorder should capture details of the group, whereas the field notes of the
moderator may be sketchy.
• Audio tape recording can also be done wherever possible as it is low cost and reliable.
• Video cameras can prove to be useful in some circumstances but researchers need to
be cautious as there may be a tendency among people to be apprehensive and less
candid on video.
•
Laptop computers used by a fast typist can be used to capture a nearly complete
transcript in real time.
: ©
■
: <3
Analysis of data
The analysis of FGD data needs to be systematic and as per a protocol that follows a
predetermined and verifiable set. of steps. There is no single ‘best’ systematic process. One
of the skills that an analyst must have is to match the level of analysis to the problem at
hand. A complex study such as one in which researchers are trying to understand how
different types of people think or feel about a cultural practice may require transcript based
analysis. However, if a research is trying to understand simpler issues such as which out of
a few sets of educational material is more appealing, analysis based on notes may be all
that is required. No matter what level of analysis is selected, breaking analysis into
manageable chunks keeps the material from becoming overwhelming. The analysts must
look for the major themes that cut across groups as well as a key insight that might have
been shared by only one person, in order to do this effectively, the analysts must have a
clear understanding of the purpose of the study.
When analyzing FGD data, the analysts need to consider many different aspects of the
focus group and its participants’ responses. Some of these are mentioned below:
Words
The analysis needs to consider both the actual words used by the participants as
well as the meaning of those words. Different participants may have used different
words and the analyst may need to determine the degree of similarity among these
responses.
&
Context
A participant’s response may have been triggered by a stimulus such as a question
asked by the moderator or a comment from another participant. It is imperative to
examine the context by finding the triggering stimulus and then interpreting the
comment with its environment in mind. The response should be interpreted in light
of both the preceding discussion and the tone and intensity of the oral comment.
4.
(
’
Internal Consistency
• Participants in FGDs may sometimes change, even reverse, their positions after
interactions with others. This phenomenon occurs rarely in individual interviews
due to a lack of interaction with other participants. When there is a shift in opinion,
the researcher may need to trace the flow of the conversation to find clues that
might explain the change.
c
SambotnU <J{gseafcli c£ Communications
€
r•
€
30
'02^
Reading Material 10
Monitoring and Evaluationof Development Projects
Frequency
Frequency is a measure of how often a comment was made although it alone does
not tell us how many different people made this comment. The analysts should not
assume that frequency is an indicator of importance. It may not be necessarily true
that items that are discussed most often are most important.
Extensiveness
Extensiveness is the measure of how many different people made a particular
comment. This measure gives the analyst a sense of the degree of agreement on
an issue. However, it is not possible to determine extensiveness using only the
transcript unless names are attached to comments. In FGD analysis,
extensiveness is as useful as frequency.
©
©
Intensity
In FGDs, the participants may occasionally talk about a topic with a special
intensity or depth of feeling. They may sometimes use words that connote intensity
or tell the moderator directly about their strength of feeling. Intensity may be difficult
to gauge with transcripts alone because the voice tone, speed and emphasis on
certain words are key to communicating emotion. In the analysis, it is important to
pay attention to what is said with passion or intensity.
• B
©
Specificity
Responses that are specific and are based on experiences may be given more
weight in the analysis that responses that are vague and impersonal. Greater
attention needs to be placed on responses that are in the first person that those
that are in the hypothetical third person. The specificity of any comment can be
gauged by the degree to which the respondent can provide details during a follow
up probe.
*
Finding big ideas
One of the traps of analysis of FGD data is focusing so much on the detail that*the
analyst misses the big ideas. To facilitate this, the researchers may need to step
back from the discussions by allowing extra time for big ideas to percolate. For
example, after finishing the analysis, the moderator might set the report aside for a
brief period and then jot down a few of the most important findings. The reporter
and others in the research team can help verify the ‘big ideas’.
f---.
Guidelines for conducting In-depth Individual Interviews
In-depth interviews include both one to one interviews as well as group interviews. This
could be community group meetings, which are somewhat larger and less homogeneous
than focus groups but follow a similar format. In-depth interviewing is characterized by its
length, depth and structure (Ticehurst and Veal 1999). They are longer in duration than
quantitative based interviews typically lasting between 45 and 90 minutes. In-depth
interviews also allow the interviewer to probe more deeply into the question at hand, getting
the respondent to explain their answers and obtain greater meaning and insight. In-depth
interviewing, also known as unstructured interviewing, is a type of interview which
researchers use to elicit information in order to achieve a holistic understanding of the
interviewee’s point of view or situation; it can also be used to explore interesting areas for
further investigation. This type of interview involves asking informants open-ended
questions, and probing wherever necessary to obtain data deemed useful by the researcher.
As in-depth interviewing often involves qualitative data, it is also called qualitative
interviewing.
V--'-
i
■
SamGodfU ‘R/seartfi c£ Communications
31
i c
P-s-uo
H5 T3 P"
!
Reading Material 10
Monitoring and Evaluationof Development Projects
In depth interviews differ from direct observation primarily in the nature of the interaction. In
such interviews it is assumed that there is a questioner and one or more interviewees. The
purpose of the interview is to probe the ideas of the interviewees about the phenomenon of
interest. Following are a few strengths and limitation of this tool.
©
Strengths
• Flexibility to explore responses through probing
• Ability to observe body language
• Ability to control questions
• Ability to tackle more complex topics
Limitations
• Time consuming
• Potential for interviewer bias in interpretation
• Findings may be subject to circumstances at the time of interview
• Accessibility of respondents
Organizing In-depth Interviews
Free association, style allows for unconstrained responses from the respondent. The
interviewer’s job is to start the ball rolling on a general topic and thereafter keep the
conversation progressing with gestures of agreement.
Applying an unstructured style of questions places more restriction on the respondent. It
places a greater constraint on the subject matter but leaves the responses within the subject
matter up to the respondent. The interviewer here has to introduce the topic and keep
discussions around the subject matter. This style improves the reliability and replication of
the data.
Use of semi-structured questions constrains the subject topics and places an additional
constraint on responses by using a range of pre planned subject areas which are used to
reveal hoped for data and responses. Questions are still open ended but interviewers must
constrain the conversation to the topic and subject areas. Probes can still be used but
become mo’re directive in nature.
(■-
C-.'
The next thing to do is to assemble questions into logical topjc groupings. This will assist the
researcher and respondent concentrate on the question at hand and keep the discussions
focused, as well as having some benefit in the analysis phase. Having a balance of hard and
easy questions around a topic area enables an adept interviewer to be able to dive in and
out of these questions as the situation dictates.
Skill required in conducting In-depth Individual Interviews
Listening
Interviewers need to concentrate on what respondents are saying and what they are not. Not
only does the interviewer need to listen but they also need to watch. A respondent’s body
language can give some hidden clues to their feelings.
Probing
9
Interviewers will be required to ask multiple sub questions often having to restate or
rephrase these until the true meaning is uncovered.
Sambodlii Q^searcfi dt, Communications
32
Reading Material 10
Monitoring and Evaluationof Development Projects
Silence
In order to elicit more information from a respondent the interviewer simply needs to remain
quiet at the end of a response. Most people feel uncomfortable when there is a long silence,
the way they resolve this nervousness is to continue talking.
Questioning techniques for In-depth Individual Interviews
Individuals vary in their ability to articulate their thoughts and ideas. With good questioning
techniques, researchers may be more able to better facilitate the subjects’ accounts and to
obtain quality data from them. Current literature suggests some questioning techniques,
summarised in the following points:
t
Asking clear questions
It is important to use words that make sense to the interviewees, words that are sensitive to
the respondent’s context and worldview. *[0 enhance their comprehensibility to the
interviewees, questions should be easy to understand, short, and devoid of jargon (Kvale
1996:130).
Asking single questions
Patton (1987:124) points out that interviewers often put several questions together and ask
them all as one.. He suggests that researchers should ask one thing at a time. This will
eliminate any unnecessary burden of interpretation on the interviewees.
Sequencing the questions
This refers to using a special kind of questioning technique called ‘Funnelling’, which means
asking from general to specific, from broad to narrow. It is also useful to ask questions about
experience or behaviour before asking questions about opinions or feelings as this helps
establish a context for the informants to express the latter. For example, asking "What
happened?" before "How do you feel now?"
*
&
Probing and follow-up questions
The purpose of probing is to deepen the response to a question, to increase the richness of
the data being obtained, and to give cues to the interviewee about the level of response that
is desired. This can be done through direct questioning of what has just been said, for
example, "Could you say something more about that?"; "Can you give a more detailed
description of what happened?; "Do you have further examples Of’this?" Alternatively, a
mere nod, or "mm," or just a pause can indicate to the subject to go on with the description.
Repeating significant words of an answer can lead to further elaboration (Kvale 1996:133).
Interpreting questions
Throughout the interview, the researchers should clarify and extend the meanings of the
interviewee’s statements to avoid misinterpretations on their part. Researchers may use
questions like ‘Is it correct that you feel that
?"; "Does the expression
cover what you
have just expressed?" to allow the interviewees to confirm or disconfirm what has been
interpreted by the researchers.
Encouraging a free rein while maintaining control
The researchers should be prepared to let the interviewees ‘travel’ wherever they like, but a
rough checklist of ideas or areas the former want to explore is useful. Palmer (1928:171)
suggests that proficient interviewers should be always in control of a conversation which
they guide and bend to the service of their research interest.
SambodfU
\ (
earcft dC Communications
33
5
Reading Material 10
Monitoring and Evaluationof Development Projects
Establishing rapport
This can be achieved by, for example, respecting the informants’ opinions, supporting their
feelings, or recognising their responses. This can also be shown by the researchers’ tone of
voice, expressions or even gestures. A good contact is established by attentive listening,
with the interviewer showing interest, understanding, and respect for what the subjects say^
A good interview allows subjects to finish what they are saying and lets them proceed at
their own rate of thinking and speaking.
©
At the meeting, the interviewer should brief the respondent as to the nature or purpose of the
interview being as candid as possible without biasing responses and attempt to make the
respondent feel at ease. S/He should explain the manner in which he will be recording
responses, and if he plans to tape record, he should get the respondent’s assent. Whenever
possible and with the consent of the respondent it is best to tape-record the interview. This
ensures the depth and breadth of data is obtained, and allows researchers to understand not
just what people say, but the way in which they say it. At all times, an interviewer must
remember that he is a data collection instrument and try not to let his own biases, opinions,
or curiosity affect his behaviour.
Once the interview is completed, it is best to sit and reflect on the past hour. Notes should be
taken on how the interview progressed, where the interview took place, any other feelings
about the interview and the setting. Observations should not be confined to the interview
alone, adept interviewers take note of the surroundings and inter office behaviours between
interviews.
0
Sambodiii <Rf search c£ Communications
34
Reading Material 11
Monitoring and Evaluation of Development Projects
Mean
= Z X =3 + 7 + 44-6 =20 = 5
N
4
4
For the data set A & B example
A
73
74
75
75
B
68
73
Mean
©
A
75
75
75
78
84
B
75
Choosing a Measure of Central Tendency
How do you decide which measure of central tendency to use? Often it is best to
calculate more than one measure since each tells us something different, although all
refer to the center of the data. To most accurately present the data, if the data departs
from normality, then one should generally present more than one measure of central
tendency so that the data is better described (this also means that it is not a normal
distribution since the values are different)
Generally, the mean is the most preferred measure of central tendency because it uses
every score in the distribution. (Remember the goal is to use one score to best represent
the entire data set). However, it is not always the best.
When the Mode is best: Since the mean has to use interval or ratio data, often the
mode is best in nominal or ordinal data (it can be used for all types though). It is very
easy to use, since no calculations are necessary.
When the Median is Preferred over the Mean: There are several times when the
median is the preferred statistic:
(1) When there are extreme scores: An extreme score pulls the mean toward
that score (an extreme high score pulls the mean higher and an extreme low
score pulls the mean lower).
(2) If some scores are missing
(3) If the distribution is open-ended
(4) If you have ordinal data: Since this data is discrete, the mean is not generally
not used.
SambodHi <Rpseardi & Communications
37
J
Reading Material 12
Monitoring and Evaluation of Development Projects
MEASURE OF VARIABILITY: DEVIATION AND VARIANCE
4-':
To more fully understand a data distribution, a measure of central tendency alone does
not tell the story. Variability is fundamental to research since we are trying to determine
what causes the variability!
r ■
Unlike measures of central tendency which capture similarities, a measure of variability
is a measure of the dispersion of the scores around the mean; how spread out the data
is, how different the data is. The fact that the actual scores deviate from the average
score or the mean indicates that they is variability. The goal here is to determine how
much variability is in the data set or the distribution.
W
Measures of variability have 3 main purposes:
(1) To describe the distributions (how disperse or variable they are)
(2) It gives an idea of how accurately the mean describes the distribution
(3) It gives an indication of how well a sample represents the entire population
(distribution)
Going back to our practice data sets:
A
B
73
68
Summary
Mean
Median
Mode
A
75
75
75
74
73
75
75
75
75
78
84
B
75
75
75
The two sets look identical if only measures of central tendencies are looked at. Let’s
look at the variability measures.
Range
V-':
The range is the-difference between the highest and lowest score in the distribution. It is
expressed as a single number, the result of the following formula:
Range = highest score - lowest score
e
©
I.-.-;.
I;
For example, with scores of 3,7,12, 8, 5 and10; Range =12-3 = 9
The range is very straight-forward and can be a good initial check on the data. In reality,
however, people usually list the minimum and maximum rather than the single number
range.
The range is not a very good measure of variability in that it only takes two numbers of
the entire data set into Recount: the largest and the smallest. The range has two primary
limitations: (1) it is affected easily by extreme scores since it only uses the highest and
lowest (therefore unstable) and (2) it says nothing about what happens between the
highest and lowest. What is better is a measure that is responsive to every data point in
the distribution (like the mean is for central tendency).
SamBodHi (Rgsearcfi dC Communications
I
f"-
38
Reading Material 12
Monitoring and Evaluation of Development Projects
For our example data sets:
A
B
73
68
74
73
75
75
75
75
78
84
The range for A is 5
The range for B is 16
The range, then, in this case does tell us more about the variability than does either of
the measures of central tendency (which are identical), but is it still the best measure?
Variance
The variance, s2, is the mean of the squared deviations:
s2 = Z(x-£)2or SS
N
N
i..VS-'
where SS = sum of squares, which refers to the sum of squared deviations from the
mean Z(x - x )2
The variance will detect differences that the range will not detect because it uses all of
the data points in the calculation, not just the minimum and maximum.
The key to variance is the SS since if the SS-is large, the variance is large, etc.
The variance is very useful in more sophisticated inferential statistics, but its use in
descriptive statistics has one major flaw: the calculated value is expressed in squared
units of measurement! Because of this, it is very seldom used for interpreting descriptive
variance.
Standard* Deviation
f-
©
f. ■
v.
The remedy to this major flaw with using the variance is simple! To get back to original
units, take the square root
or
aTsS
n
n
The
standard
deviation
is the most commonly used
This is the standard deviation.
It
is
frequently-cited
with
the
mean
as the main piece of
measure of variability.
information about a distribution.
It takes into account every piece of data in the distribution (as does the mean) as it
measures, on the average, how much each piece of data deviates from the mean.
This is done in several steps:
a. Measure the deviation of each score from the mean X - X
(*note the sign can be - or +)
: tSambodHi Rpsearcfi & Communications
39
Reading Material 12
Monitoring and Evaluation of Development Projects
b. Calculate the mean or average of all of these deviations by adding up each
deviation and dividing by N
For scores 8,1,3 and 0;
N=4, X =3
X X-X
8 +5
1 -2
3 0
This is always true!!!
0 -3
Z(x-X) = p
0 !!!
: ’r
Since we want a summary score (one value) that represents the average
deviation, we are in a dilemma since we cannot divide zero by anything!
c. Square the deviations to get rid of the( -) signs and the sum will not be zero
X
X-X
(X - X )2
8
+5
25
Z (x - x )2 = 38 = 9.5
N“
4
1
-2
4
3
0 0
9.5 is the variance
0
-3
9
=0
=38
Even though variance does provide some idea of deviation and it can be used to
compare one distribution to another, it still has a problem. Squaring the numbers
magnifies the numbers considerably and the units are NOT in their original form
(now in a squared form).
d. Therefore, we take the square root of the variance, and that takes us back to our
original units; it makes the correction for having squared the distances from the
mean (the deviations) and gives us the standard deviation.
standard deviation = y[~ variance
To find the standard deviation, we need to use the sum of the squared deviations
in the formula: there are two ways to do that
L:
definitional formula, o = 7~ Z (X- X )2
..;
fe
n
X
8
1
3
0
te
X-^X
+5
-2
0
__ -3
=0
or
7~ss
n
(X - X )2
25
4
0
9
=38
k
Sambodhi (Rgsearcfi
i
r
Communications
40
Reading Material 12
Monitoring and Evaluation of Development Projects
{
r
Plugging the numbers in the formula gives you T” 38 / 4 = 3.08
The standard deviation (and the variance) is very sensitive to extreme scores (as is the
mean). Therefore be very careful in interpreting the standard deviation or the variance
for a distribution that is markedly skewed or contains even a few very extreme scores.
So what does this tell us? It gives an idea of how much dispersion around the mean
there is in a particular distribution. It gives us that overall notion of dispersion in a
distribution that we need to complete our description of the distribution!
The standard deviation is a critical statistical value and one needs to come to terms with
its meaning. One should think of it as the “average dispersion" in a distribution; the
average distance of any one point from its mean.
The Standard Deviation and the Normal Distribution
(
Not only does thinking of the standard deviation as the average dispersion in a
distribution assist in understanding it, it also helps to know how it works under various
conditions.
It is very useful in looking at is use as a distance measure in normal distributions.
X ± 1 o contains about 68% of the data points
X ± 2 o contains about 95% of the data points
X ± 3 o contains about 99% of the data points
Although this applies to normal distribtutions, even in skewed distributions one can
safely state that X ± 1 a picks up the majority of the cases, X ± o picks up even more,
and X ± 2 o picks up all but a very few data points.
rf -
e
SambodHi ^search & Communications
41
fee
Reading Material 13
Monitoring and Evaluation of Development Projects
CORRELATION AND REGRESSION
f
Correlation
Correlation is one of the most widely used measures of association between two or more
variables. In its simplest form it signifies the relationship between two variables i.e.
whether an increase in one variable results in increase of other variable. In a way
measure of correlation are employed to explore presence or absence of correlation that
is whether or not there is correlation between variables in equation. Correlation
coefficient also describes direction of correlation i.e. whether it is positive or negative
and strength of correlation whether an existing correlation is strong or weak.
r-
Though there are various measures of correlation between nominal or ordinal data,
Pearson product-moment correlation coefficient is a measure of linear association
between two interval-ratio variables. The measure, represented by letter r, varies from -1
to +1. A zero correlation indicates that there is no correlation between the variable.
•
A correlation coefficient indicates both type of correlation as well as strength of
relationship. Coefficient value determines the strength whereas sign indicates whether
variables change in same direction or in opposite direction .A positive correlation
indicates that as one variable increases other variable also increases in similar way. A
negative correlation signified by a negative sign indicates that there is an inverse
relationship between two variables i.e. an increase in one variable is associated with
decrease in other variable. A zero correlation suggests that there is no systematic
relationship between the two variables and any change in one variable is not associated
with change in other variable!*
Regression
•
F-
Regression is one of the most frequently used techniques in social research. It is used in
estimating, the value of one variable based on the value of other variable. It does so by
finding a line of best-fit using ordinary least square method. The relation between
variables could be linear or non-linear and thus regression equation could also be linear
or non-linear.
Regression analysis extends correlation by asserting a direction of causality (from
explanatory/independent variable to explained/dependent/endogenous one) and
measuring the influence of the explanatory variable(s) on the explained variable.
By convention we use Yfor the dependent variable, X for the explanatory.
The aim is to find the ‘line of best-fit’ to the data.
between Y and X as a linear one.
Initially we define the relationship
Y=a+bX
In regression equation, a is defined as intercept and b is known as regression coefficient.
The value of b indicates the change in dependent variable for every unit change in
independent variable.
SamBodJii ^search & Communications
f
42
Reading Material 13
Monitoring and Evaluation of Development Projects
Regression coefficient is another widely used measure of association between two
interval-ratio variables. Regression coefficient is an asymmetric measure of association
and that’s why regression coefficient of dependent variable on independent variable is
different from the regression coefficient of independent variable on dependent variable.
Further, whether one should use an asymmetric measure of association or a symmetric
measure, it depends on the application of regression method. In case when one is trying
to predict one variable by other variable, then an asymmetric measure is preferred.
<'£•
f-
t.'
Sambodhi <Rgseardi c£ Communications
L
j.-. f
43
Reading Material 14
Monitoring and Evaluation of Development Projects
PARTICIPATORY MONITORING AND EVALUATION: OVERVIEW OF TOOLS1
f ■
f'-'
&
ACCESS AND CONTROL
This is a tool used during stakeholder analysis where people are asked who has access
over resources and who makes the decision on the use and disposal of these
resources. An example is in the field of gender and development where we try to
determine the degree of access and control over resources of women as compared to
that of men.
ACTION PLANNING
4'.
o
Purpose: To identify the specific tasks, resources, timetables and responsibilities
required to achieve a particular objective. e
Description: Develop a table with tasks, resources, due date and responsible persons
across the top and then complete the details for all the tasks that will be required to
achieve a particular project objective.
ACTIONS
WEN RtSiOfcES
• ---
!♦ —
AFTER ACTION REVIEW
i
. €
!
Description: An after action review (AAR) is a discussion of a project or an activity, to
enable the individuals involved to learn for themselves what happened, why it happened,
what went well, what needs improvement and what lessons can be learned from the
experience. It is not a performance assessment but rather a way to learn from
experience.
Benefits
• can be applied in different stages of a project or activity
• can be applied with different numbers of people
• duration is variable
• can be formal, informal and personal
e
1 http://portals.wi.wur.nl/ppme/
Sambotflii Q&searcfi c£ Communications
3
I
5
44
Reading Material 14
Monitoring and Evaluation of Development Projects
©
i-.
How to do it
1. A formal AAR meeting should be conducted as soon as possible after the event.
Invite team members and maybe some external people.
2. Create a climate of trust, openness and commitment to learning. The purpose is
the purpose is to help future projects run more smoothly by identifying the
learning points from this project
3. Appoint a facilitator that is to help the team to learn by drawing out answers,
insights and previously unspoken issues; to ensure that everyone has an
opportunity to contribute; and to help create the right climate and ensure that
blame is not brought in. The facilitator should be someone who was not closely
involved in the project, so that they can remain objective
4. Revisit the objectives and deliverables of the project. Ask ‘what did we set out to
do?’ and ‘what did we actually achieve?’.
5. Ask ‘what went well?’. Find out why, and share learning advice for the future
6. Ask 'what could have gone better?’. Find out what the problems were, and share
learning advice for the future
7. Ensure that everyone feels fully heard before leaving the meeting
8. Record the AAR in order to effectively share that learning with others. Include
things like: lessons and guidelines for the future; some background information
about the project to help put these guidelines into a meaningful context; the
names of the people involved for future reference; and any key documents such
as project plans or reports.
9. Share the learning with others.
BRAINSTORMING
Purpose: To quickly gain a lot of ideas from a group without getting caught up in detailed
discussion.
r.-
^7
Description:
1. Begin by asking the group to think of as many ideas as they can about the topic
in question. You can give them several minutes for this.
2. Go around the group asking each person to briefly state his/her idea. The ideas
can
be
captured
using rich
pictures, nominal
group
technique, mindmapping or card techniques. Everybody’s ideas should be
treated equally at this stage. Do not let people start debating each other’s ideas.
3. Once all of the ideas have been noted somewhere visible to everyone (e.g., on a
flip chart or chalkboard), then there can be some analysis.
4. The emerging issues, topics and questions can later be grouped, sorted and
prioritised.
Advantages & Disadvantages
It's a quick and enjoyable process. It stimulates involvement and cross-fertilisation of
ideas. However, most ideas are contributed from a few quick-thinking people.
Tips:
• Note that this method does not, on its own, suffice as a data gathering or
analysis method
• The method can work with small or larger groups and can take as little as five
minutes, depending on the subject, detail needed and number of people. A
brainstorming session should not take very long, as it really is only meant to get
out ideas that can be discussed in detail later.
SamGodHi ^seared <& Communications
45
Reading Material 14
Monitoring and Evaluation of Development Projects
.
.
.
©
People find it very difficult not to comment or evaluate when ideas are generated
in a brainstorm. Set a rule at the beginning that all judgements made during the
brainstorm will be ruled out until a later discussion. As with most group
discussion methods, some participants may dominate. To avoid this problem,
you can distribute cards to all individuals on which they brainstorm their thoughts
or ask them to brainstorm in sub-groups.
To avoid the problem of only a few people getting involved, you can include
some individual thinking time before the brainstorming session starts.
This method is commonly used in combination with other methods, for
example, to start a focus group session.
CAUSE AND EFFECT MAPPING
Purpose: To understand the contributing causes or reasons for a particular problem or
issue, or to identify effects or impacts of a particular change.
Description: Start by putting the topic - with a symbol, photograph or in words - in the
centre of a group (on the ground or a large flip chart). To work well, the topic must be
specific. Ask what happened as a result of that activity (or trend/event), both positive and
negative, These consequences are noted as symbols or with words and placed on the
diagram to show how cause and effect are linked, with arrows or lines. If quantitative
information is needed, then questions can be asked about the amounts related to each
impact that has been identified. You can also ask if the impact has been the same for
everyone and symbolise that on the map, with different groups having their own
symbols. Repeat the exercise with an agreed frequency. You can use past diagrams for
comparison to generate a discussion on why changes might be occurring and how the
rate of change is progressing. If several flow diagrams are made with different groups
and aggregation is required, they can be compiled into a single diagram, which then
forms the basis of discussion.
^0
d*aihi X
2^;
fpoorer^
®
C/*
women
Cl*
men
£ childrcn
0G
prioritised impact
sadrwM 'k
problem
1
of lack of watef'
JJ
O’
poverty
i
more
deforeiution
aod drought
. 'V
O*
^(2>
■
%
Q
livestock
agri^l^e
-'
CONCEPTUAL MODELING
i
Purpose: a conceptual model is a diagram of a set of relationships between certain w
factors that are believed to impact or lead to a target condition.
Sambotflii (Rfsearcb & Communications
i
46
^■7
Reading Material 14
Monitoring, and Evaluation of Development Projects
A good Conceptual Model:
• Presents a picture of the situation at the project site
• Shows assumed linkages between factors affecting the target condition
• Shows major direct and indirect threats affecting the target condition
• Presents only relevant factors
• Is based on sound data and information
• Results from a team effort
Factor
a
Activity
>
• Factor
>
Factor
>
Activity
>
Factor
Factor
T
A
Factor
Factor
Target
Condition
<
Description. Constructing your Conceptual Model:
1. Construct an Initial Conceptual Model that shows the situation before the start of
a project or intervention. What is the target condition, what are factors and
relationships?
2. Add op the planned activities to carry out goals and objectives for the project or
intervention. The result is a (Project) Conceptual Model that shows how you think
your project or intervention will influence the baseline situation at the project site.
3. Write the Conceptual Model down Th words.
DECISION MATRIX
Also called multi-criteria analysis.
Purpose: make a good decision by scoring different strategies or scenarios.
&
Example:
I
SamGocflii ‘Research & Communications
.1
l
Description:
• Broadly discuss content of clusters, strategies or scenarios
• Develop scoring / selection criteria.
• Indicate order of priority of criteria (column 1).
• Make a decision matrix of alternative approaches / strategies by criteria
• Mark approaches (each strategy in another column): the more positive, the
higher the score
• Identify (combination of) key approaches /strategies
47
Reading Material 14
Monitoring and Evaluation of Development Projects
Strategy 1
Selection criteria
4
_______
Relevance______
J_____
Cost__________
2_______
Sustainability
Subtotal
____
2 _____
3 ______
Feasibility in time
Participation
by 1
stakeholders
technical 1
of
Avaibility
capacity
12
TOTAL
/
Strategy 2
J_____
3 _______
4 _______
8_______
2_______
4
Strategy 3
2_______
2_______
J____ _
Strategy 4
3 _______
4 _______
3_______
10______
5_______
4_______
2
J_____
4
3
2
18
14
16
3
An advantage is that in a quick overview the best options are appearing in the table. A
disadvantage may be that it does really not take into account that some criteria may be
more important than others.
EMPOWERMENT CIRCLE
It is a recently developed participatory tool to understand the empowerment situation of
people. This tool can easily be applied with illiterate people either individual or in a
group. The main objective of the tool is to generate information from the targeted people
about the progress of their empowerment situation after project intervention.
Application Process:
• Draw a big circle and then draw a line from the centre of.the circle measuring 50
percent inside and 50 percent out side.
• Explain to the people how to mark in the circle of their situation from the centre (o
%) going outside the circle towards the 100% mark..
• Ask them what their situation was before the project and where they are now.
They can mark 10 % intervals.
Example:
;
0
i
V
J®
€:
EVALUATION WHEEL
Purpose: Evaluate different aspects in a visual way
Description:
I
Sambotflii Gtysearcfi c£ Communications
§
48
Reading Material 14
Monitoring and Evaluation of Development Projects
Cl
•
•
e-
Decide upon criteria for evaluating (not too many)
Ask each individual to draw a wheel with the same number of spokes as criteria
chosen. The spokes should be marked with the various criteria. The spokes
represent a scale with low or zero at the centre and high or 10 at the edge.
Individuals should assess the process according to the various criteria and score
each criterion by marking the spoke at the right point along the scale.
Example:
r
sr:>■'t:
UJ
e
g-.-.
r
f-
r-
• 1:
i-
r
I
*
FISHBONE ANALYSIS
Purpose: Identify possible causes of problems with more than one possible solution.
Tool for situation analysis.
Description:
e
• Draw a horizontal line (central spine) near the centre of a page. Label one end
with the problem or goal.
. Collect infomation from the participants on aspects of the situation. For the mam
aspects, draw lin^s off the central spine. Aspects related to a particular mam
spine are then drawn off that spine.
• Set priorities. Select the most important main spine then rank the items drawn off
that spine. Continue this process with the other main spines.
• If the top priority spine has no branches, use this aspect of the situation for the
next step in problem solving.
SamSodHi Reared c£ Communications
kJ
49
Reading Material 14
Monitoring and Evaluation of Development Projects
i.
Curriculum I
Architecture]
Faculty
Support
I
|
Institutional
Support
ter
---- i® Higher
SatiSfaC,iOn_J^^n
i Vi Be«er
.
‘
Student
Support
(T*cn.Trarantf
l.y^
La
L
Course
Design
i | Physical
' .Environment]
k Rate.
- }®1
Problem.
Not Meeting
Deadline
e•
Advantages and disadvantages
It can show much of a situation’s structure. It can however become messy for large
situations. Positive and negative aspects of the situation are not being distinguished.
Source: Khanya- Managing Rural Change CC. 2001. Manual for community-based
planning, draft 1.
FLOW DIAGRAMS
Purpose: To illustrate and analyse the consequences (positive and negative) of
particular issues or actions using diagrams.
Description: Take the action to be considered, for example, the employment of a
property management planning coordinator, and map out the steps that need to be taken
and the factors that need to be taken into account.
FOCUS GROUPS
Purpose: To collect general information about an issue from a small group of selected
people through group discussion.
r>
Description: A broad question, for example, ‘What impact do you think the landcare
group has had in achieving sustainable land use?’ is given to a group of about eight to
discuss for one or two hours. There is minimal intervention by the focus group facilitator
other than to make sure everybody has a say. The discussion is either recorded or
detailed notes are taken and then later analysed. Focus groups should be conducted in
pairs: one person to facilitate the discussion and the other for note-taking.
V--
II u
«
I
SamGodhi Reared & Communications
50
Reading Material 14
Monitoring and Evaluation of Development Projects
€
r
f:':
•
If facilitated well, this method can bring out detailed information. It generally
stimulates rich responses and also provides a valuable opportunity to observe
discussions and to gain insights into behaviours, attitudes, language and
•
However, facilitation of a focus group requires considerable skill - both in
moderating the group and in adequately recording the responses. Group
dynamics, due to individuals being too shy, dominating, disruptive, etc. can
hamper the discussion.
This method can be used to obtain a consensus view. However, a small group ot
people cannot represent all views held by, for example, an organisation or
community. On the other hand, if the group is not homogeneous enough, there
can be great disagreement. So think carefully about the composition of the
<5.
•
V
•
This method can generate focused insights more quickly and generally more
cheaply than through a series of key informants or formal social surveys.
FORCE FIELD ANALYSIS
(
Purpose: Force Field Analysis is a tool for systematically analysing the driving and
restraining forces in a situation.
r
DeS^riPOefine the problem or situation as a common goal. Note: the goal is a situation
r;’
■
i-
•
•
•
.
that requires analysis and not a solution that requires implementation.
List all the driving forces that help archieve the goal (e.g. by brainstorming)
Rate each driving force according to their strength and importance from 1 to 10
List all the restraining forces that hinder archieving the goal and rate them as well
Discuss how each of the most important restraining goals can be reduced and
how driving forces can be enhanced.
i
£■
t !
"Z—..la
I9 •
SamSodHi Q&searcH
5
i
Communications
51
<&-•
Reading Material 14
Monitoring and Evaluation of Development Projects
Current^
State
^Desired
.
l
State
Restraining Forces
Driving Forces
Driving Force 1
Restraining Force 1
Driving Force 2
Restraining Force 2
Driving Force*
Restraining Forcer
Tip: the list of most important driving forces (opportunities) and restraining forces
(contraints / threats) can be used to make a SWOT analysis.
HISTORICAL ANALYSIS
Purpose: To understand the history and background to a situation or project. It is a
valuable way of exploring how change has occurred, why things are the way they are
and why different groups or individuals hold the views they do.
Description: Set up a large sheet of butchers paper with rows and columns. Put dates
down the side and beside them put topics such as key local events, key external events,
influence of local personalities/groups, major changes (social, environmental, economic)
and key trends. With a group, fill in the table that has been created. It is usually best to
complete the trends for each time period as a way of rounding off the exercise. This
takes from one to several hours and can be effective with quite large groups.
r
f
r-.
| Hit
<- •
'.WWE 1
i
■■
f.'
r-:
SamGodfii (Rpsearcfi
Vi«5'
■
Communications
52
V
Reading Material 14
Monitoring and Evaluation of Development Projects
(
INTERRELATIONSHIP DIAGRAMS
rr
Purpose: To identify which out of a series of contributing causes are the most important
and how they relate to each other.
Description: On a whiteboard or butchers paper, write down in a circle between five and
20 factors contributing to, or causing, a problem or issue. Examine each factor in relation
to each of the other factors and ask, ‘Is it caused by or a cause of the other factor’. If it is
caused by the other factor, draw a line with an inward arrow between the pair, if it is the
cause of the other item, draw a line with an outward arrow between the pair. Draw the
arrow only in the direction of the strongest effect, do not have two-way arrows. If there is
no interrelationship do not draw a line between them at all. When you are finished, the
factors with the most outward arrows will show up as the drivers. These will generally be
the factors that will drive change; focus attention on them.
This technique can be used to stimulate discussion in a group, analyse information and
determine priorities.
w
i.
V >
Ortero*
/i/T
2D
ISSUE ANALYSIS
Purpose: To identify the major issues that have been raised by people from a range of
other tools, such as focus groups and semi-structured interviews, and then to group
these into major themes. This process is important in making sense of a lot of qualitative
information.
£ ■
Description: Go through the notes or recordings of the particular activity and identify the
major issues that a group or individual has raised. Common issues from a range of
activities, individuals and groups are then grouped and the underlying theme that links
the issues together‘is identified. It is possible to measure the number of times a
particular issue emerges and hence provide some quantified results from qualitative
information.
: ©
I
I
SamSodfii <l&searcfi & Communications
f' <
I
■
53
Reading Material 14
Monitoring and Evaluation of Development Projects
Data
LLPPA
x
Local level participation planning approach (LLPPA) is a kind of participatory planning
approach we developed and used to prepare local level plans and implement them. The
ideas generated by communities will be enriched by additional simple biophysical and
sociao-economic surveys. Expert knowledge to facilitate community ideas and extract
needs and sometimes assisting them to concretize their need in more systematic way
plays vital role.
<
■
-r?-.
: t'..'
IJ
!
I
Thames
©
r-'
i
issues
The approach has the following procedures:
1. Site selection
2. First consultation with representatives (formal and informal) of communities
3. Second consultation and detailed discussions with representatives
4. Discussion with communities about the general intention of the project, if agreed
elect planning team
5. Planning team should composed of representatives from different wealth groups,
geographical locations and gender. In some places two communities are allowed
when women are socially not comfortable to sit with man and discuss issues.
6. The planning team with the assistance of the extension agent appraise major
problems and set in a priority order. The agent will assist them with different
mechanisms to deepen their thought in addres.sing major constraints within the
community.
A
tentative development option is suggested for the different problems according
7.
to the priority order.
8.. The problems identified and the options suggested will be indicated on simple
baseline and development plan map.
.
9. Community assembly will be called by the planning team and the identified
' problems and suggested development options presented to communities. The
communities will discuss on problems, priorities and suggested development
options. IT can be amended, accepted or rejected.
10. Once the plan is accepted by communities it will get its final shape by the agent
together with the planning team and some baseline data will be incorporated. A
copy of the plan will be given to the planning team, community leader, the agent
and one copy will be sent to the district officer.
11. Implementations started as per activity plan.
12. Reporting done by the extension agent
13. Monitoring is also done by district officer and upper level experts.
SamGocflii ^search
Communications
54
VReading Material 14
Monitoring and Evaluation of Development Projects
Limitations:
• Outspoken people within the community can dominate ideas and influence
priority setting.
• The quality of the plan depends on the facilitation capacity of the agent. Getting
genuine thrust from communities is very important.
• The M&E part is a bit weak and not very much participatory
Strengths:
• Simple with very little and less complicated procedure.
• It addresses the needs of the different community groups.
• It has good baseline data for impact monitoring.
• The ownership feeling of communities for the plan is high.
• If properly handled doesn’t raise expectations.
• It combined both expert knowledge and ideas of communities
• It was developed and revised together with field staff based on field observations.
©
History:
• Watershed Management Plan (1984-88)
• Minimum Planning (1989-1993)
. LLPPA
(I,
II,
111,
IV)
(1994-upto
present)
Assistance and application:
The development of the approach was stared by the assistance of FAO to the Ministry
Of Agriculture at the early stage but the final refinement was supported by World Food
Programme Development Section in Ethiopia. The approach is extensively used in the
country especially in WFP assisted districts by project name called MERET. This project
covers 72 Districts in the country.
LOCALITY MAPPING
Purpose: To draw on the knowledge of local people to develop a map of the local area.
This is a good way, for example, of identifying who is undertaking land conservation
activities, where land degradation problems are and where improvements have been
noticed.
Description: Using large sheets of butchers paper, draw the outline of the local area, for
example, roads, towns, rivers and property boundaries. This can be done by projecting
an overhead map onto butchers paper and tracing the required information. Having
prepared the map, which could be as large as a whole wall, people can then add their
information either directly or by using sticky notes.
r--‘
r•
(•■
*
SamSocffii ^J^searcd g£ Communications
I
I
”
55
Reading Material 14
Monitoring and Evaluation of Development Projects
<'■
a
LOGFRAME
1-7
A logical framework (logframe) is a matrix that summarizes what a project intends to do
and how, what the key assumptions are, and how outputs and outcomes will be
monitored and evaluated. The logframe is the product of the Logical Framework
Approach. The first column of the logframe is the objective hierarchy. A logframe
outline used in the IFAD Guide to M&E is this:
Goal
Performance
and Indicators
QuestionsMonitoring mechanism
jinformation sources
andAssumptions
Component
Purpose
Output*
Activity
€;
The vertical logic identifies what the project intends to do, clarifies the causal
relationships, and specifies the important assumptions and uncertainties beyond the
project manager’s control (colurfins 1 and 4). The horizontal logic defines how project
objectives specified in the project description will be measured, and the means by which
the measurement will be verified (columns 2 and 3). This provides the framework for
project monitoring and evaluation.
In most projects, the once-made logframe serves as a rigid means of assessing whether
the project has achieved what it should. We think it would be better for learning and
creating real impact of a project, when the logframe is considered as a more flexible
document.
MATRIX ANALYSIS
Purpose: Rank the value of a particular activity or item according to a range of criteria.
Description: First decide what it is that you want to compare, then determine what
criteria will be used. Then each item can be scored against the criteria. If some criteria
are deemed more important they can be given extra weight. The totals can be added to
show which is the most beneficial item.
SamSodfii (fysearcfi
Communications
56
Reading Material 14
”
Monitoring and Evaluation of Development Projects
I T
R-
i
4
2^
•M
H
2
0
o
Mt
-w*
«
•»»
u +
&?■
•t*
M
a
s**r mi
r l«»
z
1
ru
U+T" I
IV
MINDMAPPING
Purpose: To cluster similar ideas, to see links between them and to pick out the most
important issues when discussing or brainstorming. As in rich picturing, this is a good
way of making sure all aspects of a situation have been considered.
^.-3.
Description: On butchers paper or a whiteboard, start with the central issue or question
and then build a dendrogram (like a tree) of ideas from the central question. You can put
down the most important or higher order things first and then build on these.
4:-
(ho*.
♦*- —f |n?SNt
'<•
^7
r*^>i
—
>
_______
■
L.
LSamSodHi ^seared e£ Communications
I.,• r
57
Reading Material 14
Monitoring and Evaluation of Development Projects
NOMINAL GROUP TECHNIQUE
Purpose: To enable a group to quickly develop a ranked list of problems, issues or
actions.
©
Description: Develop a list of the problems, issues or actions that need to be ranked.
Express each as clearly as possible to avoid confusion. Each person then ranks the
statements according to what they see as the priorities. Each person should give the
highest number (if there are six statements the highest number will be six) to their
highest ranked statement and the next highest number to their next highest priority and
so on through the list, The rankings from each person are added up. The total scores for
each statement will enable them to be put in order of importance. This can be done in
small or large groups and can take from 15 minutes to an hour depending on the size of
the group and how much debate there is over the initial statements.
teissvr
L.-.,
I I T
To!^
io
r
a
io
i
1
&
+
14
3
This tools can support other methods or tools. It can help, for example, to generate a list
of priority performance questions or indicators, to prioritise stakeholders during
a stakeholder analysis and to follow up on impact flow diagrams to prioritise impacts.
PROBLEM TREE
The problem tree is a core tool in the Logical Framework Approach.
. W
<-
Purpose: Identify the main problems and establish the cause and effect relationships
between these problems (so that these are sufficiently addressed in a project design).
Perform this exercise with a group of different stakeholders.
• Brainstorm all problems in the situation and put each on a card
• Identify a common agreed the core problem (linked to most other problems), and
write a precise definition on a card.
• Divide the other cards into causes and effects of the core problem; put then
respectively below and above the core problem. Some cards may also turn out to
be overall constraints (move to the side of the core problem).
• Try to find all cause and effect relations and move the cards accordingly. There
can be more causes to one effect or more effects to one cause.
• Review the result, check the logic and revise if necessary
• Draw vertical links to show cause-effect relationships, and horizontal links to
show joint causes and combined effects.
• Copy the diagram on a sheet of paper
SamSodHi ^seared
r
Communications
58
Reading Material 14
Monitoring and Evaluation of Development Projects
OBJECTIVES TREE
An objectives tree is a hierarchical flowchart of objectives. Within the Logical Framework
Approach, this is the positive opposite of the problem tree.
~
©
Description:
1. Taking the problem tree as your base, invert all the problems in order to make
them into objectives. This process then leads into an “objectives tree” with the
central objective simply being the inverse of the central problem.
2. Ask participants then to look at these objectives and discuss which of these can
be tackled by the project.
3. The problem and objectives trees are a first step towards producing a logical
framework matrix
Tips on use:
The problem and objectives tree provide a comprehensive though simplified view of
cause and effect relationships. In this way, the process of creating a logical
framework can become more accessible to primary (and other) stakeholders, making it
easier to involve them in revising the project design or developing their own
activities.Linkages are represented with lines or arrows. If arrows are to be used, make
sure that everyone is clear about what arrows mean as they are not a universally
understood symbol.
TIMELINE
Timeline is a widely used participatory tool to understand. a kind of history of a
community. It gives a quick impression of the community how it is moving forward or
what history has been carried by the community. It generates information of the major
events (e.g, earthquake, epidemic, landslide, flood, new school building, electricity, new
road built, new technology etc) of the community which has a certain impact in the
society.
This tool is useful to build rapport in the community as well. Without good rapport
building there is not possible to apply participatory tool in the communities. It is a very
easy tool to apply and makes the key informant happy to tell about their story/history ’’’’
which helps to build rapport.
This tool is also very useful to understand and analyse the conflict situation on how the
conflict evolved and came to the present situation. It generates information from the
conflicting parties that according to them what happened in different time interval.
i.-
Application Process:
• Identify the Key Informant in the community (in many cases old knowledgeable
people would be the Key person for the historical events)
• Clarify the objective of using this tool to the community
• Ask them to tell about the past major events happened the community that has
certain impact and they still remember it.
• Facilitate to explain about only one event at a time with date so that you can note
down the information
• Make a simple format containing columns of Date, Major events, Impact and
record the information
Sambodhi <Rgsearcfi & Communications
i
59
Reading Material 14
Monitoring and Evaluation of Development Projects
e
Example
Major Events
Date
Effects/ Impact
Remarks
TIME TREND
This is a simple participatory tool to understand the situation of any development
intervention such as agriculture production, forestry, livestock, health condition etc etc. It
helps to understand and analyze the fluctuation situation of development progress and
their reasons during the different time interval.
Application Process:
• Identify the key informants according to the sector that you are going to use it.
For example if you are applying it to analyze information of paddy production you
have to identify the people who are mostly involve in the sector
• Draw a right angle and below the line write the year since when you are required
to get the information (may be past 10-20 years)
• Ask them to point what was the situation of the paddy production in the year
1995 then 1996 or 1998 etc. what time interval (every 1 year or 2 years or 5
years) you required the information
• Draw the line accordingly and if there is major fluctuation why it was happened
discuss on it and record the information
Example
ۥ
1996
■
■
e
1998
2000
2002
2004
TRANSECT WALK
Transect walk is a way of data collection in informal surveys and participatory studies. It
is essentially a walk over the transect of an area to observe and document the
similarities and differences of socio-economic and bio-physical features. It is usually
used in areas where there is spatial diversity.
VENN DIAGRAMS
Also called institutional linkage diagrams or Chapati diagrams
SamSotflii Q&searcfi & Communications
60
r
Reading Material 14
Monitoring and Evaluation of Development Projects
v:;
•r;:
? -
Purpose: To illustrate the extent to which individuals, organisations, projects or services
interact with each other or overlap and the importance of each, and their efforts, to the
issue being evaluated.
Description: Each entity is represented by a circle. The larger the circle the more
important it is, the closer circles are to each other the more interaction there is. Large
circles represent powerful organisations, overlapping circles represent interacting
organisations and a small circle within a larger circle represents a component of that
organisation. The diagrams may be created using cut out circles or by drawing. The
group may combine their diagrams and discuss any differences. Further discussions
may tackle issues such as conflict resolution or organisational capacity building.
VISIONING
$
Purpose: To develop a shared vision of what a group would like the outcome of a project
or evaluation exercise to be. This helps people think creatively and let go of immediate
problems. It is also a way of finding common ground between conflicting interests.
Ui -f«ve
/ r wivW
Description: Ask people to describe how they would like things to be in the future. It is
possible to do this in an imaginary way. For example, ask people to imagine they are
giving a presentation at a conference at some point in the future describing why their
project has been successful. This can be done with from one individual up to a large
group and the time taken varies accordingly.
b
i
(
SamGodfii ^searcfi
(
f
f
Communications
61
"7
Reading Material 15
Monitoring and Evaluation of Development Projects
MOST SIGNIFICANT CHANGE (MSC)
The most significant change (MSC) technique is a form of participatory monitoring and
evaluation. It is participatory because many project stakeholders are involved both in
deciding the sorts of change to be recorded and in analysing the data. It is a form of
monitoring because it occurs throughout the program cycle and provides information to
help people manage the program. It contributes to evaluation because it provides data
on impact and outcomes that can be used to help assess the performance of the
program as a whole.
Essentially, the process involves the collection of significant change (SC) stories
emanating from the field level, and the systematic selection of the most significant of
these stories by panels of designated stakeholders or staff. The designated staff and
stakeholders are initially involved by ‘searching’ for project impact. Once changes have
been captured, various people sit down together, read the stories aloud and have
regular and often in-depth discussions about the value of these reported changes. When
the technique is implemented successfully, whole teams of people begin to focus their
attention on program impact.
What’s in a name?
MSC has had several names since it was conceived with each emphasising a different
aspect.
Monitoring-without-indicators
MSC does not make use of pre-defined indicators, especially ones that have to be
counted and measured.
The ‘story’ approach
The answers to the central question about change are often in the form of stories of who
did what, when and why - and the reasons why the event was important (Dart 1999a
1999b).
Monitoring
MSC was first developed as a means of monitoring changes in a development aid
project (Davies, 1996). We think it can also be used for evaluation purposes.
Impact monitoring
Unlike traditional monitoring techniques that focus largely on monitoring activities and
outputs, MSC focuses on monitoring intermediate outcomes and impact.
Evolutionary approach to organisational learning
This was the original name given to the technique by Rick. The name reflects the
epistemology that informed the original design. In 2000, we settled on the name Most
Significant Change technique. This embodies one of the most fundamental aspects of
the approach: the collection and systematic analysis of significant changes.
•
The MSC story
f
P'
e
Sambocflii (fysearcfi
Communications
62
I
Reading Material 15
Monitoring and Evaluation of Development Projects
! S
V’.
The most significant change (MSC) technique was invented by Rick Davies .in an
attempt to meet some of the challenges associated with monitoring and evaluating a
complex participatory rural development program in Bangladesh, which had diversity in
both implementation and outcomes. The program was run by the Christian Commission
for Development in Bangladesh (CCDB), a Bangladeshi non-government organisation,
which in 1996 had over 500 staff and worked with more than 46,000 people in 785
villages. Approximately 80 per cent of the direct beneficiaries were women. The large
scale and open-ended nature of the activities posed a major problem for the design of
any system intended to monitor process and outcome (Davies, 1996).
Rick developed the MSC technique as part of the fieldwork for his PhD on organisational
learning in non-government aid organisations (Davies, 1996). Both the thesis and MSC
were informed by an evolutionary epistemology. While you don’t need to know this
background theory in order to use MSC, you can find out more about it in Chapter 7. It is
also worth noting that Jess and others have analysed the -use of MSC from different
theoretical perspectives to that used by Rick. This flexibility is consistent with the
underlying design of MSC.
Overview of implementation steps
f -
<■
■■
f-
v.
r
M8C is an emerging technique, and many adaptations have already been made that will
be discussed throughout this Guide. Before getting into modifications, we present a
comprehensive overview of what a ‘full’ implementation of MSC might look like. We have
described this using ten steps.
1. How to start and raise interest
2. Defining the domains of change
3. Defining the reporting period
4. Collecting SC stories
5. Selecting the most significant of the stories
6. Feeding back the results of the selection process
7. Verification of stories
8. Quantification
9. Secondary analysis and meta-monitoring
10. Revising the system.The first step in MSC generally involves introducing a range of stakeholders to MSC and
fostering interest and commitment to participate. The next step is to identify the domains
of change to be monitored. This involves selected stakeholders identifying broad
domains—for example, ‘changes in people’s lives’—that are not precisely defined like
performance indicators, but are deliberately left loose, to be defined by the actual users.
The third step is to decide how frequently to monitor changes taking place in these
domains.
SC stories are collected from those most directly involved, such as participants and field
staff. The stories are collected by asking a simple question such as: ‘During the last
month, in9your opinion, what was the most significant change that took place for
participants in the program?' It is initially up to respondents to allocate their stories to a
domain category. In addition to this, respondents are encouraged to report why they
consider a particular change to be the most significant one.
f-
f-
f-
SambodJii ^search e£ Communications
63
Reading Material 15
Monitoring and Evaluation of Development Projects
The stories are then analysed and filtered up through the levels of authority typically
found within an organisation or program. Each level of the hierarchy reviews a series of
stories sent to them by the level below and selects the single most significant account of
change within each of the domains. Each group then sends the selected stories up to
the next level of the program hierarchy, and the number of stories is whittled down
through a systematic and transparent process. Every time stories are selected, the
criteria used to select them are recorded and fed back to all interested stakeholders, so
that each subsequent round of story collection and selection is informed by feedback
from previous rounds. The organisation is effectively recording and adjusting the
direction of its attention - and the criteria it uses for valuing the events it sees there.
After this process has been used for some time, such as a year, a document is produced
with all stories selected at the uppermost organisational level over that period in each
domain of change. The stories are accompanied by the reasons the stories were
selected. The program funders are asked to assess the stories in this document and
select those that best represent the sort of outcomes they wish to fund. They are also
asked to document the reasons for their choice. This information is fed back to project
managers.
r' ■ •
fe.--
The selected stories can then be verified by visiting the sites where the described events
took place. The purpose of this is two-fold: to check that stories have been reported
accurately and honestly, and to provide an opportunity to gather more detailed
information about events seen as especially significant. If conducted some time after the
event, a visit also offers a chance to see what has happened since the event was first
documented.
The next step is quantification, which can take place at two stages. When an account of
change is first described, it is possible to include quantitative information as well as
qualitative information. It is also possible to quantify the extent to which the most
significant changes identified in one location have taken place in other locations within a
specific period. The next step after quantification is monitoring the monitoring system
itself, which can include looking at who participated and how they affected the contents,
and analysing how often different types of changes are reported. The final step is to
revise the design of the MSC process to take into account what has been learned as a
direct result of using it and from analysing its use.
The kernel
The kernel of the MSC process is a question along the lines of:
Looking back over the last month, what do you think was the most significant change in
[particular domain of change]?1
similar question is posed when the answers to the first question are examined by
another group of participants:
‘From among all these significant changes, what do you think was the most significant
change of all?’
f
This process provides a simple means of making sense of a large amount of complex
information collected from many participants across a range of settings.
(’
■
u,.
Sambocflii (Rjsearcfi c£ Communications
&
e
64
o
Reading Material 15
Monitoring and Evaluation of Development Projects
Telling each level about the choice of significant changes made at the higher levels is an
essential component of the whole process. This helps readjust the focus of searches for
significant change in each subsequent reporting period.
The purpose
There are several reasons why a wide range of organisations have found MSC
monitoring very useful and these include the following:
(
r
f:..
1. It is a good means of identifying unexpected changes.
2. It is a good way to clearly identify the values that prevail in an organisation and
to have a practical discussion about which of those values are the most
important. This happens when people think through and discuss which of the
SCs is the most significant. This can happen at all levels of the organisation.
3. It is a participatory form of monitoring that requires no special professional
skills. Compared to other monitoring approaches, it is easy to communicate
across cultures. There is no need to explain what an indicator is. Everyone can
tell stories about events they think were important.
4. It encourages analysis as well as data collection because people have to
explain why they believe one change is more important than another.
5. It can build staff capacity in analysing data and conceptualising impact.
6. It can deliver a rich picture of what is happening, rather than an overly
simplified picture where organisational, social and economic developments are
reduced to a single number.
7. It can be used to monitor and evaluate bottom-up initiatives that do not have
predefined outcomes against which to evaluate.
When and when not to use MSC
^-9
MSC is better suited to some program contexts than others. In a simple program with
easily defined outcomes (such as vaccination, perhaps), quantitative monitoring may be
sufficient and would certainly consume less time than MSC. In other program contexts,
however, conventional monitoring and evaluation tools may not provide sufficient data to
make sense of program impacts and foster learning. Th.e.types of programs that are not
adequately catered for by orthodox approaches and can’ gain considerable value from
MSC include programs that are:
W
V
c-
i
• complex and produce diverse and emergent outcomes
• large with numerous organisational layers focused on social change
• participatory in ethos • designed with repeated contact between field staff and participants
• struggling with conventional monitoring systems
• highly customised services to a small number of beneficiaries (such as family
counselling).
Monitoring and evaluation in an organisation may serv£ several purposes. MSC
addresses some purposes more than others. In our experience, MSC is suited to
monitoring that focuses on learning rather than just accountability. It is also an
appropriate tool when you are interested in the effect of the intervention on people’s lives
SamGotflii ‘Research tSC Communications
65
Reading Material 15
Monitoring and Evaluation of Development Projects
P-
6.
c
€
r
iI
I
I
1
and keen to include the words of non-professionals. In addition, MSC can help staff to
improve their capabilities in capturing and analysing the impact of their work.
There are also some instances where the benefits may not justify the cost of MSC. While
MSC can be used to address the following, there may be other less time-consuming
ways to achieve the same objectives:
• capture expected change
• develop good news stories for public relations (PR)
• conduct retrospective evaluation of a program that is complete
• understand the average experience of participants
• produce an evaluation report for accountability purposes
• complete a quick and cheap evaluation.
Some program contexts are more conducive to the successful implementation of MSC.
In our experience, some of the key enablers for MSC are:
• an organisational culture where it is acceptable to discuss things that go wrong
as well as success
•
• champions (i.e. people who can promote the use of MSC) with good facilitation
skills
• a willingness to try something different
• time to run.several cycles of the approach
• infrastructure to enable regular feedback of the results to stakeholders
• commitment by senior managers.
Sam6odHi Q&searcfi & Communications
66
Position: 2658 (2 views)