THE MOST SIGNIFICANT CHANGE (MSC) TECHNIQUE
Item
- Title
- THE MOST SIGNIFICANT CHANGE (MSC) TECHNIQUE
- extracted text
-
The ‘Most Significant Change’
(MSC) Technique
A Guide to Its Use
by
Rick Davies and Jess Dart
• Tundedby'
CARE International,'United Kingdom
Oxfam Community Aid Abroad, Australia ] Learning to Learn, Government otSouth Australia
Oxfam New Zealand | Christian Aid’, United kingdom | Exchange, United Kingdom
Ibis, Denmark | MellernfolkeligLSamvirke (MS),.Denmark.
Lutheran World Relief, United'States of America
.
Version 1.00-April 2005
Most Significant Change Guide
Table of Contents
Table of Contents
Acronyms
4
Preface: The Structure of this Guide
5
Acknowledgements______________________________________ ______________________
1
Chapter One: A Ten Minute Overview of MSC
8
What is MSC, in a nutshell?
What's in a name?
The MSC story
Overview of implementation steps
The kernel
The purpose
When and when not to use MSC
Where to get further information
8
8
9
10
11
12
12
14
Chapter Two: Ten Steps to Implementing MSC
15
Step 1: How to start and raise interest
Step 2: Defining domains of change
Step 3: Defining the reporting period
Step 4: Collecting SC stories
Step 5: Selecting the most significant of the stories
Step 6: Feeding back the results of the selection process
Step 7: Verification of stories
Step 8: Quantification
Step 9: Secondary analysis and meta-monitoring
Step 10: Revising the system
15
17
22
23
28
34
36
38
38
44
Chapter Three: troubleshooting
46
Concerns expressed by participants
Concerns expressed by others
46
50
Chapter Four: Building Capability for Effective MSC
54
Building the capacity of MSC 'champions'
Building the capacity of the staff Ip participate fully in MSC
Considering resources and time required
54
55
56
Table of Contents
Most S'gii'hcant Change Gude
Chapter Five: MSC within a Monitoring and Evaluation (M&E) Framework
58
MSC within the program cycle
MSC as monitoring and evaluation
MSC as a specific type of monitoring
MSC as program evaluation
MSC and organisational learning
Other uses of MSC within programs
58
58
58
61
63
64
Chapter Six: Validity and Voice in MSC
67
MSC: a valid technique
Purposive sampling
Bias in MSC
Issues of voice and power in MSC
67
69
70
71
Chapter Seven: How MSC Compares to Other Approaches and Epistemologies
72
Appreciative Enquiry
Participatory Monitoring and Evaluation (PM&E)
Case studies and vignettes
Evolutionary epistemology and MSC
MSC under a constructivist lens
72
72
72
73
74
Chapter Eight: History of MSC
75
History as process
Types of MSC uses
The differences that make a difference
'
75
75
77
Chapter Nine: New Directionsjpr MSC
81
Finetuning
Combining with other approaches
Innovations
Future research areas
An invitation to innovate, review and communicate
81
82
84
86
87
Bibliography
88
Appendices
Appendix 1: Sample Story Collection Formats
Appendix 2: Sample Significant Change Stories
Appendix 3: Sample Story Reporting Format
Appendix 4: Facilitation Guide for Story Collection
3
92
97
103
104
Acronyms
Most Significant Change Guide
Acronyms
ADB
ADRA
AKRSP
BADSP
CCDB
CRS
DFID
IDSS
M&E
MSC
NCO
ONZB
SC stories
VSO
Asian Development Bank
Adventist Development and Relief Agency, Australia
Aga Khan Rural Support Program
Brong Ahafo Rural District Support Project, Ghana
Christian Commission for Development in Bangladesh
Catholic Relief Services
Department for International Development, UK
International Development Support Services, Australia
monitoring and evaluation
most significant change (MSC) technique
non-government organisation
Oxfam New Zealand
Significant change stories
Voluntary Service Overseas
2
4
Most Significant Change Guide
Preface
Preface: The Structure of this Guide
his publication is aimed at organisations, community groups, students and academics
who wish to use MSC to help monitor and evaluate their social change programs and
projects, or to learn more about how it can be used. The technique is applicable in many
different sectors, including agriculture, education and health, and especially in development
programs. It is also applicable to many different cultural contexts. MSC has been used in a
wide variety of countries by a range of organisations. By 2004, MSC had been used both by
NCOs and governments in Africa, Asia, Latin America, Europe and Australasia.
T
The structure of this Guide
The introductory chapter provides a quick overview of MSC. In Chapter 2 we focus on
the practicalities of implementing MSC and divide the process into ten steps. Chapter 3
offers guidance on practical troubleshooting, and Chapter 4 looks at building capacity for
effective use of MSC. In Chapter 5 we examine how MSC fits into the program cycle and
how it can contribute to program improvement.
After Chapter 5, we delve more into the theory. We believe that MSC can be successfully
implemented without a strong understanding of the theory. So if you just want to know
about the practicalities - stop there! But for those readers who enjoy a foray into theory,
Chapters 6 and 7 examine validity in MSC and how it fits with other approaches and
epistemologies. The final two chapters outline the evolution of MSC: where it came from
and where it might be heading next.
The structure of this Guide reflects our attempt to cater for different types of use. We want
the Guide to be of practical help to those choosing to implement MSC for the first time.
But we also want to provide information and analysis that will be of use to those who are
alroadv experienced in using MSC yet want to extend their knowledge of the technique or
reline the way they use it. As well as addressing the needs of practitioners, we also hope to
address the interests of those doing research on MSC and related methods of participatory
and qualitative research.
Notes about the terminology
We struggled at times in writing this Guide to find language that worked equally well in
the development sector and the public sector of developed economies. We have made a
few choices in the name of consistency and we encourage you to consider translating these
terms to ones that makes sense for you.
Firstly, we had to decide how to refer to the people who are targeted by social change
programs. In the development sector, these people are often referred to as beneficiaries. In
the public sector context of developed economies, they are referred to as clients, service
users or participants. In this publication we used the term 'participant' where the meaning
is clear, and 'beneficiary' as a second resort.
5
Preface
Most Significant Change Guide
Secondly, we needed to choose between program, project and intervention. In this
publication we have chosen to use the term 'program' to cover all descriptions of social
interventions with a beginning and an end, regardless ot size.
Thirdly, we needed to describe the people who fund programs and are variously referred to
as donors, investors, funders or purchasers. In this Guide we settled on the term 'funders'.
In the published literature, MSC is sometimes called an approach, at other times a process
- it has even been referred to as a model. The experts have advised us that it is a monitoring
and evaluation technique. However, we also believe that MSC embodies some aspects of an
approach because it has a particular theoretical basis; this is discussed further in Chapter 7.
Disclaimer
This publication is based on the collective experience of Rick Davies and less Dart in
monitoring and evaluating social programs. While we make every eilort to ensure the
accuracy of this work, any judgments as to the suitability of information for the reader's
purposes are the reader's responsibility. We cannot extend any warranties, and assume no
responsibility, for the suitability of this information or the consequences of its use.
Copyright and distribution
Copyright for the MSC Guide is held by the authors, Rick Davies and Jess Dart.
We are interested in making the Guide as widely available as possible. This document is
freely available in pdf format from our websites atwww.mande.co.uk/docs/MSCGuide.him
and www.clearhorizon.com.au You can also download pdf copies from the sponsors'
websites.
Hard copies may be made on condition that the Guide is reproduced in full, in its existing
format, and that these copies are made available free of charge, not tor sale or prom. Wheie
there is a need to reproduce parts of the Guide only, please contact us for permission to
do so.
Third parties interested in selling copies of the MSC Guide should contact us for permission
and to establish appropriate terms and conditions.
Copyright © RJ Davies and J Dart (2004)
Please address all inquiries to:
Dr Rick Davies
Dr Jessica Dart
82 Bishops Rd, Trumpington,
Cambridge CB2 2NH United Kingdom
Clear Horizon, PO Box 341,
Hastings VIC 3915 Australia
rick@mande.co.uk
jess@clearhorizon.com.au
Phone: 44 (0) 1223 841 367
Phone/fax: 61 (0) 3 5979 2957
Mobile: 44 (0) 7855 766 354
Mobile: 0425 735 530 .
Fax (to Email): 44 08701 640 239
6
Most Sgnlt'cant Change Gude
Acknowledgements
Acknowledgements
This Guide would not have been written without the generous sponsorship of the following
organisations.
.mJ<J-wwyJcafeinternational.org/uk
CAREJnternationajUnited Kingdom/
J^yyxfam org^U..
Oxraip Community Aid. Abroad,>Australiy_
..
Learning.to Learn, government .of-Sou'th Austrajia ^< !'www,legrningtolearn:sa.edu.aii ;
••
j-
------------------------------------------------------------------ ~
Oxtarp N^w;Zealand JJ?:
‘*
Christian Aid; United.-Kingdom
:•C ...ft'an-aid:org. uik>• J
.
.
..
*
. .
’ •
■
~
.
;
1
■
•www.healthcomms.org
Exchange, United Kingdom, ’
Ibis, Denmark
’
ijC^ww^
Mellemfolkeligt Samyirke (MS), Denmark
.'
: :■
■ www.ibis.dk .
J-
- :
Lutheran World Relief, United States of America
;■
■'
■■
s •: •
■
.
'www.ms.dk '
;
www.lwr.org
While both authors have considerable experience of using MSC, we have also tried to
make use ol lhe growing body of grey literature on other people's experiences with using
MSC. We would like to acknowledge all the authors of this material, who are listed in the
bibliography al lhe end of the Guide.
We would also like to thank the following for their comments on the draft version of this
guide: Jay Goulden, CARE; Deborah Elkington, Oxfam CAA; Robyn Kerr, ADRA; Silke
Mason, Ibis; Gillian Holmes, Ibis; Peter Sisgaard, MS.
Chapter 1 • <4 Ten Minute Overview
Most Significant Change Guide
Chapter One. A Ten Minute Overview of MSC
What is MSC, in a nutshell?
he most significant change (MSC) technique is a form of participatory monitoring and
evaluation. It is participatory because many project stakeholders are involved both
in deciding the sorts of change to be recorded and in analysing the data. It is a form of
monitoring because it occurs throughout the program cycle and provides information to help
people manage the program. It contributes to evaluation because it provides data on impact
and outcomes that can be used to help assess the performance of the program as a whole.
T
Essentially, the process involves the collection of significant change (SC) stories emanating
from the field level, and the systematic selection of the most significant of these stories
by panels of designated stakeholders or staff. The designated staff and stakeholders are
initially involved by 'searching' for project impact. Once changes have been captured,
various people sit down together, read the stories aloud and have regular and often in-depth
discussions about the value of these reported changes. When the technique is implemented
successfully, whole teams of people begin to focus their attention on program impact.
What’s in a name?
MSC has had several names since it was conceived with each emphasising a different aspect.
Monitormg-without-indicators
MSC does not make use of pre-defined indicators, especially ones that have to be counted
and measured.
The 'story' approach
The answers to the central question about change are often in the form of stories of who did
what, when and why - and the reasons why the event was important (Dart 1999a, 1999b).
Overview of MSC
8
Chapter 1 a A Ten Minute Overview
Most Significant Change Guide
Monitoring
MSC was first developed as a means of monitoring changes in a development aid project
(Davies, 1996). We think it can also be used for evaluation purposes.
Impact monitoring
Unlike traditional monitoring techniques that focus largely on monitoring activities and
outputs, MSC focuses on monitoring intermediate outcomes and impact.
Evolutionary approach to organisational learning
This was the original name given to the technique by Rick. The name reflects the
epistemology that informed the original design (see Chapter 7).
In 2000, we settled on the name Most Significant Change technique. This embodies one
of the most fundamental aspects of the approach: the collection and systematic analysis of
significant changes.
The MSC story
The most significant change (MSC) technique was invented by Rick Davies in an attempt
to meet some of the challenges associated with monitoring and evaluating a complex
participatory rural development program in Bangladesh, which had diversity in both
implementation and outcomes. The program was run by the Christian Commission for
Development in Bangladesh (CCDB), a Bangladeshi non-government organisation, which
in 1996 had over 500 staff and worked with more than 46,000 people in 785 villages.
Approximately 80 per cent of the direct beneficiaries were women. The large scale and
open-ended nature of the activities posed a major problem for the design of any system
intended to monitor process and outcome (Davies, 1996).
Rick developed the MSC technique as part of the fieldwork for his PhD on organisational
learning in non-government aid organisations (Davies, 1996). Both the thesis and MSC were
informed by an evolutionary epistemology. While you don't need to know this background
theory in order to use MSC, you can find out more about it in Chapter 7. It is also worth noting
that Jess and others have analysed the use of MSC from different theoretical perspectives to
liiai used by Riuk. This Ilexibility is consistent with the underlying design of MSC.
More information on the history of the use of MSC, including Jess's role in its promotion in
Australia, can be found in Chapter 8.
Bangladesh - five years later
“During the current year [2000], CCDB has carried on the Most Significant Change
System (MSC) designed for identification and analysis of qualitative changes taking
place in the lives of the reference people. This system has been part of the regular
PPRDP monitoring system since August 1995. However, during the current plan
period CCDB proposes to use the system in all other programs. The system appears
to be very useful in monitoring the changing trends / impact of the programs, as the
stories reflect concrete changes that have taken place in the lives of the reference people
within a given time frame.” (CCDB, 2000:4, Bangladesh)
Overview of MSC
9
Chapter 1 • ATen Minute Overview
Most Significant Change Guide
Overview of implementation steps
MSC is an emerging technique, and many adaptations have already been made that
will be discussed throughout this Guide. Before getting into modifications, we present a
comprehensive overview of what a 'full' implementation of MSC might look like. We have
described this using ten steps.
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
How to start and raise interest
Defining the domains of change
Defining the reporting period
Collecting SC stories
Selecting the most significant of the stories
Feeding back the results of the selection process
Verification of stories
Quantification
Secondary analysis and meta-monitoring
Revising the system.
The first step in MSC generally involves introducing a range of stakeholders to MSC and
fostering interest and commitment to participate. The next step is to identify the domains ol
change to be monitored. This involves selected stakeholders identifying broad domains—
for example, 'changes in people's lives'—that are not precisely defined like performance
indicators, but are deliberately left loose, to be defined by the actual users. The third step is
to decide how frequently to monitor changes taking place in these domains.
SC stories are collected from those most directly involved, such as participants and field
staff. The stories are collected by asking a simple question such as: 'During the last month,
in your opinion, what was the most significant change that took place for participants in
the program?' It is initially up to respondents to allocate their stories to a domain category.
In addition to this, respondents are encouraged to report whv thev consider a nariicular
change to be the most significant one.
The stories are then analysed and filtered up through the levels of authority typically found
within an organisation or program. Each level of the hierarchy reviews a series of stories
sent to them by the level below and selects the single most significant account of change
within each of the domains. Each group then sends the selected stories up to the next level
of the program hierarchy, and the number of stories is whittled down through a systematic
and transparent process. Every time stories are selected, the criteria used to select them are
recorded and fed back to all interested stakeholders, so that each subsequent round of story
collection and selection is informed by feedback from previous rounds. The organisation is
effectively recording and adjusting the direction of its attention - and the criteria it uses for
valuing the events it sees there.
After this process has been used for some time, such as a year, a document is produced with
all stories selected at the uppermost organisational level over that period in each domain of
change. The stories are accompanied by the reasons the stories were selected. The program
funders are asked to assess the stories in this document and select those that best represent
the sort of outcomes they wish to fund. They are also asked to document the reasons for
their choice. This information is fed back to project managers.
The selected stories can then be verified by visiting the sites where the described events
took place. The purpose of this is two-fold: to check that stories have been reported
Overview of MSC
10.
Most Sgnificant Change Guide
Chapter 1 • A Ten Minute Overview/
accurately and honestly, and to provide an opportunity to gather more detailed information
about events seen as especially significant. If conducted some time after the event, a visit
also offers a chance to see what has happened since the event was first documented.
The next step is quantification, which can take place at two stages. When an account
of change is first described, it is possible to include quantitative information as well
as qualitative information. It is also possible to quantify the extent to which the most
significant changes identified in one location have taken place in other locations within a
specific period. The next step after quantification is monitoring the monitoring system itself,
which can include looking at who participated and how they affected the contents, and
analysing how often different types of changes are reported. The final step is to revise the
design of the MSC process to take into account what has been learned as a direct result of
using it and from analysing its use.
The kernel
The kernel of the MSC process is a question along the lines of:
‘Looking back over the last month, what do you think was the
most significant change in [particular domain of change]?’
A similar question is posed when the answers to the first question are examined by another
group of participants:
‘From among all these significant changes, what do you think
was the most significant change of all?'
This process provides a simple means of making sense of a large amount of complex
information collected from many participants across a range of settings.
Telling each level about the choice of significant changes made at the higher levels is an
essential component of the whole process. This helps readjust the focus of searches for
significant change in each subsequent reporting period.
Figure 1. The MSC selection process (examplefrom ADRA Laos)
-Stories of ChangeReported
j Feed
back
ByADRA
Field
Staff
-.............. -............
■/.•« '•
I-
MSC'Selection ComijiiUe.e-j
Feed
back
\ •r
III
back
Overview of MSC
By ADRA
Select pjfa
iMSC-SelectionCommittee j
j Country;Level. M S CS e I e.c t i b ri C b m m i 11 e e
,
• Feed
ByADRA
$: J;.ield,Staff
I I T I■
Tf
Donor Office .
MSC. Selection Committee
11 ’
W
■V
;
■
Chapter 1 • A Ten Minute Overview
Most Significant Change Guide
The purpose
There are several reasons why a wide range of organisations have found MSC monitoring
very useful and these include the following.
1.
It is a good means of identifying unexpected changes.
2.
It is a good way to clearly identify the values that prevail in an
organisation and to have a practical discussion about which of
those values are the most important. This happens when people
think through and discuss which of the SCs is the most significant.
This can happen at all levels of the organisation.
3.
It is a participatory form of monitoring that requires no special
professional skills. Compared to other monitoring approaches, it is
easy to communicate across cultures. There is no need to explain
what an indicator is. Everyone can tell stories about events they
think were important.
4.
It encourages analysis as well as data collection because people
have to explain why they believe one change is more important
than another.
5.
It can build staff capacity in analysing data and conceptualising
impact.
6.
It can deliver a rich picture of what is happening, rather than
an overly simplified picture where organisational, social and
economic developments are reduced to a single number.
7.
It can be used to monitor and evaluate bottom-up initiatives that
do not have predefined outcomes against which to evaluate.
Myanmar - senior staff hear results first-hand
“The senior staff were also fascinated by the stories which came up; they hardly ever get to
hear-these .things!” (Gillian Fletcher, 2004, Advisor to CARE HIV/ AIDS program)
When and when not to use MSC
MSC is better suited to some program contexts than others. In a simple program with easily
defined outcomes (such as vaccination, perhaps), quantitative monitoring may be sufficient
and would certainly consume less time than MSC. In other program contexts, however,
conventional monitoring and evaluation tools may not provide sufficient data to make
sense of program impacts and foster learning. The types ol programs that are not adequately
catered for by orthodox approaches and can gain considerable value from MSC include
programs that are:
•
complex and produce diverse and emergent outcomes
•
large with numerous organisational layers
Overv/ew of MSC
12
Chapter 1 • A Ten Minute Overview
Most S’gn:ficant Change Guide
•
focused on social change
•
participatory in ethos
•
designed with repeated contact between field staff and
participants
•
struggling with conventional monitoring systems
•
highly customised services to a small number of beneficiaries
(such as family counselling).
Monitoring and evaluation in an organisation may serve several purposes. MSC addresses
some purposes more than others. In our experience, MSC is suited to monitoring that
focuses on learning rather than just accountability. It is also an appropriate tool when you
are inteiested in the effect of the intervention on people's lives and keen to include the
words of non-professionals. In addition, MSC can help staff to improve their capabilities in
capturing and analysing the impact of their work.
There are also some instances where the benefits may not justify the cost of MSC. While
MSC can be used to address the following, there may be other less time-consuming ways
to achieve the same objectives:
9
capture expected change
•
develop good news stories for public relations (PR)
•
conduct retrospective evaluation of a program that is complete
•
understand the average experience of participants
•
produce an evaluation report for accountability purposes
•
complete a quick and cheap evaluation.
Some program contexts are more conducive to the successful implementation of MSC. In
our experience, some of the key enablers for MSC are:
•
an organisational culture where it is acceptable to discuss things
that go wrong as well as success
•
champions (i.e. people who can promote the use of MSC) with
good facilitation skills
•
a willingness to try something different
•
time to run several cycles of the approach
•
infrastructure to enable regular feedback of the results to
stakeholders
•
commitment by senior managers.
OverView of MSC
73,:
Chapter 1 • A Ten Minute Overview
Most Significant Change Guide
USA - using MSC for small, individualised programs
"... the services provided through.this program are highly individualised. Families
come to the program withj very, different’needs and skills; We.are charged with
documenting the number of families that.have made progress, but the definition of
progress is different for each family. This makes it very difficult to use any kind of
standardised measure of change. For all of these reasons we’ve begun investigating the
MSC approach.” (Jidie Rainey, 2001, Family Literacy Program)
Where to get further information
The bibliography section of this Guide contains a range of references and suggestions lor
further reading.
For continuing access to information about MSC, including new usages and the experiences
of existing users, you might like to join the Most Significant Changes mailing list at
http://groups.yahoo.com/group/mostsignificantchanges. This was set up by Rick in 2000
and now has more than 200 members. The mailing list has a files section that contains
information on MSC usage in a range of organisations and countries from 1993 to the
present.
Learning about the past quickly - Fortune Magazine
“If you knew what was going to happen in advance every day you could do amazing
things. You could become insanely wealthy, influence the political process et cetera.
Well, it turns out that most people don’t even know what happened yesterday in
their own business. So, a lot of businesses are discovering they can take tremendous
competitive advantage simply by finding out what happened yesterday as soon as
i possible.” (Steve Jobs, 1994:23)
Overview of MSC
14
Chapter 2 • Ten Steps To Implementing MSC
Most Significant Cnange Gtrde
Chapter Two: Ten Steps to Implementing MSC
In this chapter, we discuss how to implement MSC using the following steps:
1.
Getting started: establishing champions and getting familiar with the approach
2.
Establishing 'domains of change'
3.
Defining the reporting period
4.
Collecting stories of change
5.
Reviewing the stories within the organisational hierarchy
6.
Providing stakeholders with regular feedback about the review process
7.
Setting in place a process to verify the stories if necessary
8.
Quantification
9.
Conducting secondary analysis of the stories en masse
10.
Revising the MSC process.
efore we elaborate on the individual steps, it is worth considering which steps are
fundamental, and which are discretionary. We believe that out of the ten MSC steps,
steps 4, 5 and 6 fundamentally define the process.
B
•
collection of SC stories (Step 4)
•
selection of the most significant of these stories by at least one
group of stakeholders (Step 5)
•
feedback to relevant stakeholders concerning which SC stories
were selected and why they were selected (Step 6).
Whclkei the other steps are included will depend on the organisational context and
purpose for implementing MSC.
Step 1: How to start and raise interest
Getting started is perhaps the most daunting step. People may be sceptical about the
validity of the technique and fear that it will take too much time.
It often takes an enthusiastic individual or small group to raise interest in MSC. This can
involve visiting key people and groups and showing them the methodology. It can often
help to present stories from other programs and to show example reports. The message to
be conveyed is that MSC is simple and straightforward to implement. Many practitioners
will not need to understand the theory behind MSC.
II you want to raise interest in MSC, you need to be clear about the purpose of MSC and
the role it will play in your organisation (see Chapter 4). We emphasise that MSC is not a
stand-alone technique for monitoring and evaluation (see Chapter 6).
Implementing MSC
15
Chapter 2 * Ten Steps To Implementing MSC
Most Significant Change Guide
For potential MSC adopters and users who want to know more about the theory, Chapter
7 sets out the underlying ideas and explains how and why the MSC approach differs from
other approaches.
Metaphors for explaining the approach
If you are the person attempting to initiate the adoption of MSC, it may help to use a
metaphor to explain it. These are our favourites.
Newspaper
A newspaper does not summarise yesterday's important events via pages and pages ol
'indicators' (though they can be found in some sections) but by using news stories about
interesting events. Papers are structured into different subject areas (foreign news, domestic
news, financial news, sport, leisure) in the same way that MSC uses domains. The most
important stories go on the front page and the most important of these is usually at the lop
of the front page.
School of fish
A social change program has numerous practitioners (fish) swimming in slightly different
directions, each with individual values but a common goal. MSC helps the individual fish
to communicate with each other: 'Where do we really want to go? Should we swim away
from the sharks and towards a safe place to lay our eggs or first head for food?1 MSC uses
communication to help all the fish swim in roughly the same direction, away from what is
not good and towards what is good. It helps them swim as a school towards a commonly
valued destination.
Another related metaphor is of the organisation as an amoeba, extending in directions
where it wants to go, and withdrawing from areas it does not like, all on the basis of the
signals being received from its surrounding environment.
Holiday memories
What do you remember from an overseas holiday? Do you remember the average things
or the wonderful and terrible things? MSC helps teams ol people focus on the memorable
events and uses these events to help realign effort towards achieving more of the wonderful
things and less of the terrible things. When the focus is on learning, we need to capture
more than just the average experiences.
Restaurant menu
MSC does not present one version of what is happening but a series of glimpses of what
a program is achieving. Stakeholders can select from these glimpses in much the same
way as they would select food from a restaurant menu. Choices are based on individual
preferences. In the MSC restaurant, you are asked to try and articulate the reasons for your
choice. Because the restaurant has a very responsive chef, the choices your table makes
will encourage the chef to present a revised menu to the next groups of customers.
Getting familiar with the approach
Another really important lesson we have learned from experience is to start small. It is a risky
exercise to implement a huge and complicated MSC system without first piloting it on a smaller
scale. Every' organisational context is different, and MSC will have to be moulded to your particular
organisation. It pays to conduct a pilot to find out what works and what does not work.
Implementing MSC
16
Most Significant Change Guide
Chapter 2 • Ten Steps To Implementing MSG
When piloting MSC, try to begin with the people and sections of your organisation that are
most interested and enthusiastic about its potential.
Role of champions
Once the pilot is complete and there is sufficient interest, start working out where MSC
might best fit in your organisation. Even in these early stages, it is worth identifying key
people who are excited by MSC and could champion the technique and act as catalysts in
the process. These people can be involved in designing how to implement MSC across the
organisation. They will need a greater understanding of MSC so they can respond to the
questions that will inevitably arise.
These champions can:
•
excite and motivate people
•
answer questions about the technique
♦
facilitate selection of SC stories
•
encourage people to collect SC stories
•
ensure that feedback occurs
•
ensure that the stories are collected and organised and sent to
review meetings
•
develop protocols to ensure confidentiality where necessary.
Pacific Islands - gaining donor acceptance
‘Tn preparing the M&E framework, IDSS followed best practice, meeting donor
requirements. With compliance met, die MSC approach was.proposed as an additional
element beyond the requirements of AusAID ... IDSS expects that such support
would not have been forthcoming if the MSC had been proposed as a replacement to
the conventional logframe based approach. Based on a year of implementation AusAID
was also in a position to measure IDSS’ capacity and intent to deliver the Program in
accordance with its approach.” (Keren Winterford, 2003, IDSS)
Step 2: Defining domains of change
What are domains of change?
Domains are broad and often fuzzy categories of possible SC stories. In CCDB, participants
m MSC were asked to look for significant changes in four domains:
•
changes in the quality of people's lives
•
changes in the nature of people's participation in development activities
•
changes in the sustainability of people's organisations and activities
•
any other changes.
Implementing MSC
'WHBHi
Most Significant Change Guide
Chapter 2 • Ten Steps To Implementing MSC
A domain of change is not an indicator. Good indicators are supposed to be SMART
(specific, measurable, achievable, relevant and time-bound). Indicators must be defined
so that everyone interprets them in the same way. Domains of change, on the other hand,
are deliberately fuzzy to allow people to have different interpretations of what constitutes
a change in that area.
How to use domains
Using domains of change has immediate practical value. It helps organisations to group a
large number of SC stories into more manageable lots, which can each be analysed in turn.
A benefit of determining the domains in advance is that they can provide some guidance to
the people collecting stories concerning the kind of changes they need to be searching for
without being too prescriptive.
A second reason to use predetermined domains is that many organisations want to use MSC
to help track whether they are making progress towards their stated objectives. For example,
CCDB wanted poor communities in Bangladesh to become less poor (i.e. improved quality
of life), to actively participate in development activities in their communities, and for those
activities and supporting organisations to be sustainable. But because CCDB believed each
local community should individually define these goals in more detail, they did not want
to monitor these developments using very specific indicators of change that might not
apply to all communities. So they adopted three general categories of change (domains)
associated with the quality of people's lives, the nature of people's participation and the
sustainability of people's organisations and activities.
The ‘open window’ domain
Where organisations use domains to track different types of changes, the 'any other type
of change' domain is a useful open category that allows participants to report significant
changes that don't fit into the named domains. This gives SC story collectors more freedom
to focus on things that they think are relevant - in their own context.
Implementing MSC
18
Most Sgnd'cant Change Guide
Chapter 2 * Ten Steps To Implementing MSC
Negative changes
One choice facing organisations implementing MSC is whether to have a domain that
explicitly focuses on significant negative changes. Our experience suggests that 90 to 95
per cent of SC stories within these types of domains tend to be about positive changes.
However, this figure varies according to how clearly those in charge signal that negative
as well as positive changes should be reported. It also depends on the extent to which
negative changes, once reported, are then acknowledged and responded to positively.
Some organisations have set up a domain specifically for negative stories, thus creating
an explicit demand. In the case of Target 10, a major dairy industry extension program in
Victoria, Australia, this extra domain was called lessons learned'. This put a positive spin
on the existence of negative changes and emphasised the need to extract value from such
events. ADRA Laos took a similar approach, using an extra domain called 'changes that
reflect an area to improve (negative)'.
Types of domains
Many MSC users have focused on changes in the lives of individuals. In some cases this
was because individuals were the focus of the program's over-arching objectives. Another
reason is because focusing on people's lives overcomes the problem of quantitative and
indicator-based monitoring systems that focus on activities and outputs.
However, some users of MSC, including CCDB, have used domains that focus on more than
one unit of analysis, i.e. on more than individuals. CCDB asked about significant changes
in the sustainability of people's institutions and MS Denmark asked about organisational
performance. Others, such as the Landcare support program in Australia, have included
domains that focus on changes in whole communities or in policy. Oxfam New Zealand
asked about changes in partnerships. Most of these are people-centred domains, which we
believe are more likely to be understood by participants than those domains that focus on
abstract processes.
Mozambique - difficulty with conceptual domains:
dome domains are more easily grasped than others ...? For example, it was difficult
to explain the domain of ‘Intercultural Cooperation’. In Mozambique it often did
not ring any bells. This was a bit surprising since MS runs a personnel program
positioning Danes with the partner. One declared aim is to stimulate cooperation
across borders ...” (Peter Sigsgaard, 2002:10)
How many domains should be used?
In our experience, between three and five domains is a manageable number. The limiting
factor is how much time participants are willing to spend in discussing each domain.
Participants may find the process too time-consuming if it takes more than two hours to
review changes in all the domains in any one meeting.
Implementing MSC
J?
Most Significant Change Guide
Chapter 2 • Ten Steps To Implementing MSC
Are domains essential?
Domains are not essential. MSC stories can be collected and analysed as a group (see
Step 5 below) without first being categorised into domains. Participants can be asked to go
out and look for significant changes without being given guidance in the form of specific
domains of concern. With smaller organisations where there are likely to be fewer SC
stories to examine, the MSC approach will probably be easier without domains.
In organisations such as VSO, field staff are asked to identify and document MSC stories of
any kind. It is only when the stories reach the country office level that they are categorised
into domains that are of concern to the country office and to VSO headquarters in UK.
Letting middle and upper level staff within an organisation categorise MSC stories into
domains produces some incidental benefits. If the domains are focused on organisational
objectives, then the sorting decisions tell the rest of the organisation how those staff
interpret the meaning of those objectives.
Gujarat - classification brings debate
“In these meetings, the changes noticed were classified under three headings and sent
up to the HQ level. In the whole exercise the cluster staff participated with great
enthusiasm and debates ensued with regards to classification of changes.” (Barry
Underwood, 1996, AKRSP)
At the field level, especially where program participants are involved in identifying and
selecting MSC stories, it may be useful to start without specifying domains. Instead, see
what sort of stories are generated and valued by the beneficiaries, and then divide these
into appropriate domains or have the beneficiaries do so. The choice depends on the extent
to which the organisation using MSC wants to be led by its participants rather than ib own
objectives.
South Australia - deciding not to set predetermined domains
“One of the early debates we had had in our attempts to interpret the process was
whether or not the domains of change should be imposed to match the needs of the
project for particular kinds of evidence for accountability purposes (as they were in
Davies’ work in Bangladesh). Because we wanted to prioritise the learning inherent in
the process, we decided that participants would find the domains more meaningful if
they were arrived at by participants in each of the three Learning Circles based on their
experiences of significant change through involvement in the Project.” (Rosie Le Cornu
and others, 2005)
What should domains focus on?
One question that often arises is whether the domains of change should only be about
changes caused by the organisation that is using MSC, or include changes caused by other
people, organisations or influences in general. For example, increased participation by
individuals could result from changes in government legislation relating to the right to free
Implementing MSC
20
Chapter 2 * Ten Steps To Implementing MSC
X'ost S-gn/.'cant Change Guide
association rather than anything to do with the organisation's own activities. Nevertheless,
in most societies, such changes would be considered significant.
In practice, most users of MSC have focused on changes that are a direct or indirect result
of what their organisations are doing. There is, however, an argument for saying that many
organisations already see the world too narrowly, and that it would be healthy to identify
SCs arising from any cause.
These options do not need to be mutually exclusive. It should be possible to track both
types of changes through the one application of MSC. One or more domains could be
about changes caused by the organisation's work, while another could ask specifically
about changes not caused or influenced by the organisation.
Doing so would help provide what Chris Roche (1999) has described as a 'project out' and
a 'context in' perspective.
Oxfam CAA - lessons from impact assessments
“Related to these specific findings wasjthe broad observation that even the most
successful community development programs should be aware of the context in which
they are situated ... For example, a group of waste-picker women who had enormous
success in empowerment that led to significant changes to their working and living
conditions, were having their livelihoods threatened by a proposed privatisation of
waste management in their city. This change in urban waste management policy had
the potential to undermine the strong empowerment results that had been achieved.
A broader advocacy campaign on waste management may be required to address the
I rights of these women.” (Linda Kelly, Patrick Kilby, Nalini Kasynathan, 2004)
Who should be involved in determining domains?
In some organisations, existing commitments to the pursuit of specific objectives are likely
to lead to the use of domains based on program objectives. Hopefully they will already be
well-known and owned bv the staff and clients Where there is more freedom to select the
domains, using a participatory process to identify appropriate domains is likely to encourage
all participants to take a more active interest in the MSC process and its products.
D^SSia IS-to .i- n t
i
Different techniques for developing domains
Domains can be identified by a top-down or bottom-up process, i.e. by the senior managers
of an organisation or by its beneficiaries, or though a wider process encompassing other
stakeholders as well. In the case of CCDB, the four domains were identified by Rick
through consultations among the five most senior staff. In the case of Target 10, Jess used
the Delphi technique1 to identify four domains of change through consultations with 1 50
program stakeholders The Delphi technique is a form of interactive (postal) surveying that
utilises an iterative questionnaire and feedback approach to provide participants with an
opportunity to revise earlier views based on the responses of other participants until some
desired level of consensus is reached.
1. See ‘Prioritization Process Using Delphi Technique" by Alan Cline at
http://www.carolla.com/wp-delph.htm
Implementing MSC
'
’ .
21)
:
Chapter 2 * Ten Steps To Implementing MSC
Most Significant Change Guide
Victoria: a bottom-up approach for developing domains
“In Landcare in the North Central Region ofVictoria, Australia, over 140 SC stories
were collected without domains through an interview process. These stories were then
screened by a steering committee that had been set up for this process. This group
firstly eliminated any stories that were not about change, or had insufficient detail.
They then categorised the remaining stories into piles of similar outcomes (domains).
We ended up with seven domains of change.” (Jess Dart, observations macle in 2004)
-------- :------ i---- 2-- ---- ----- :—J-------- 2______ J_______________________________________
Domains can be identified before SC stories are collected or afterwards by sorting SC
stories into meaningful groups (see above). This depends on the extent to which the
organisation wants to be open to new experiences rather than continuing to be guided by
past experiences.
Any documentation about the MSC process and its products should explain, albeit briefly,
how the domains were selected. This helps other people reading about the results to put
them in context.
Step 3: Defining the reporting period
Most applications of MSC have been as a form of monitoring. Monitoring involves periodic
collection ol information, but the frequency of monitoring varies across programs and
organisations. The same applies with uses of MSC. The frequency ol collection of SC stories
has varied from fortnightly to yearly. The most common frequency has probably been threemonlhlv, coinciding with the prevalence of quarterly reporting in many organisations.
Low frequency reporting, such as the yearly reporting used by VSO, runs the risk of staff and
project participants both forgetting how the MSC process works, or why it is being used. At
the very least it means there is likely to be a slow process of learning how to use MSC, ana
an equally slow process of organisational learning that is being stimulated by MSC. On the
other hand, a yearly cycle might require less time and resources and may be appropriate
in certain contexts.
With higher frequency reporting, all the participants in the MSC process are likely to learn
more quickly how to best use the process. However, frequent reporting will soon lead to
the exhaustion of known cases of longer-term significant change and a focus on the shorterterm significant changes that can be identified. Frequent reporting will also increase the
cost of the process, in terms of the amount of participants' time taken up by the process.
Each organisation using MSC has to make its own decision about the most appropriate
reporting period, balancing the costs and benefits involved, and taking into account the
reporting gaps that any existing M&E systems may be ignoring.
Our experience suggests that organisations tend to start MSC with more regular reporting
and decrease the frequency as the process continues. In the Bangladesh (CCDB) case,
SC stories were selected every two weeks for the first two months. This was followed by
monthly selection, which was changed to three-monthly at the end of the first two years. In
the Victorian case (Target 10), the initial monthly selection process eventually evolved into
a three-monthly selection.
Implementing MSC
22
AZosr S joi'cant Change Guide
Chapter 2 » Ten Steps To implementing MSC
When you first introduce the MSC process, there may be a whole backlog of SC stories
that people are keen to document. As implementation proceeds, these historical stories are
exhausted and subsequent SC stories tend to refer to more recent events. This change may
be accompanied by a decrease in the quantity of stories that are generated.
Step 4: Collecting SC stories
Eliciting SC stories
I'he central part of MSC is an open question to participants, such as:
‘Looking back over the last month, what do you think was the most
significant change in the quality of people’s lives in this community?’
This example is taken from CCDB, which was the first organisation to use MSC monitoring,
in Rajshahi, Bangladesh, in 1994. The question has six parts:
/.
'Looking back over the last month../- It refers to a specific time
period.
2.
'...what do you think was...'- It asks respondents to exercise their
own judgment.
3.
'... the most significant...' - It asks respondents to be selective, not
to try to comment on everything, but to focus in and report on one
thing.
4.
'...change...' - It asks respondents to be more selective, to report a
change rather than static aspects of the situation or something that
was present in the previous reporting period.
5.
'...in the quality of people's lives../ - It asks respondents to be
even more selective, not to report just any change but a change in
the quality of people's lives. This tag describes a domain of change
and can be modified to fit other domains of change. For example,
another one of CCDB's MSC questions referred to a change 'in
people's participation'.
6.
'...in this community?' - Like the first part of the sentence, this
establishes some boundaries. In this particular case we are not
asking about people's lives in New York or Alaska, but in Rajshahi.
This part can also be adjusted.
Implementing MSC
23 '
Chapter 2 • Ten Steps To Implementing MSC
Most Significant Change Guide
How to capture SC stories
There are several ways in which SC stories can be identified, then documented. The choice
of method depends in part on how actively the organisation wants to search for new SC
stories, versus its need to tap into the existing knowledge of its field workers through
retrospective inquiry. Active searching is likely to be more demanding in terms ol the
amount of the participant's time that is required, unless their time is already available
via existing processes of stakeholder participation (see below). Active searching through
purposive interviews also runs the risk of producing 'expected' accounts of change by the
respondents.
Fieldworkers write down unsolicited stories that they have heard
In this case, fieldworkers document unsolicited stories they have heard in the course of
their work. This technique was used in the CCBD example. The implicit assumption here
was that good CCDB fieldworkers should come to learn about change stories in the normal
course of their work because they have daily and close contact with their beneficiaries. If
they cannot find such SC stories this itself may signal something about the quality of their
work. Here MSC is incidentally monitoring the staff as well as the lives of the organisation's
beneficiaries (see the section on meta-monitoring below)
By interview and note-taking
Some organisations encourage nominated people to 'interview' beneficiaries and write
comprehensive notes by hand. To strengthen this method, interviewers read their notes
back to the storyteller to check they have captured the essence of the story. The story is
more valid if it is recorded in the storyteller's own words. The technique can be improved
by using a semi-structured interview guide such as provided in Appendix 2. Such interviews
can be a useful way of generating many SC stories in a short time through the efforts of a
group of people who are dedicated to the task. Stories may also be captured using a tape
recorder and then transcribed. This proactive method of identifvinq SC stories n..i. ‘ •
especially useful when MSC is being used for evaluation rather than monitoring processes
(see Chapter 7).
During group discussion
Rather than having one person interviewing another, a group of people can share their
SC stories. In the case of Target 10, sharing stories at committee meetings often triggered
additional stories from other farmer stakeholders who were present. It is a very human thing
to respond to a story with a second one! For this reason, a tape recorder was used at these
meetings to record spontaneous SC stories. This can be a very fruitful and enjoyable way
of collecting stories. Stories collected in a group situation can also be documented using
pen and paper.
The beneficiary writes the story directly
Another technique is for beneficiaries to document their own stories. On several occasions
in the Target 10 program, farmers brought pre-written stories to meetings. However, it was
more common for farmers to come with the stories in their minds - to be documented
during the meeting. As with the use of group discussion, the use of this method depends on
the presence of a pre-existing mechanism for stakeholder involvement in the monitoring
process.
Implementing MSC
24
Chapter 2 • Ten Steps To Implementing MSC
Most StgnSicani Change Guide
Nicaragua - testimonies rather than stories
“In English the used term is story, which means cuento or historia in Spanish.
In both Spanish and English the term implies a sense of something invented,
it is more associated with fiction than reality, which can cause confusion in the
application of the MSC method. People could'try^to invent a story illustrating
the change that the interviewer is seeking instead of a story from real life. For that
reason I decided to use the term testiniony/narrative, because it implies a’sense
of an experienced event from real life” {Gill Holmes, Lisbeth Petersen,Karsten
Kirkegaard, Ibis Denmark, 2003)
What information should be documented?
Information to be documented should include:
1.
Information about who collected the story and when the events occurred
2.
Description ol the story itself - what happened
3.
Significance (to the storyteller) of events described in the story.
Documenting who collected the story and when helps the reader put the story in context
and enables any follow-up inquiries to be made about the story, if needed.
The SC story itself should be documented as it is told. The description of the change
identified as the most significant should include factual information that makes it clear who
was involved, what happened, where and when. Where possible, a story should be written
as a simple narrative describing the sequence of events that took place.
Implementing MSC
I
25'
Chapter 2 • Ten Steps To Implementing MSC
Most Significant Change Guide
The storyteller is also asked to explain the significance of the story from their point of view.
This is a key part of MSC. Some storytellers will naturally end their stories this way, but
others will need to be prompted. Without this section, people reading and discussing the
story may not understand why the story was significant to the storyteller. For example, a
woman may tell a story about going to a community meeting and sitting at the back and
asking a question. 'So what?' you may think. She then tells you that this story was significant
because she had not previously had the confidence to go to a community meeting, and
that the program helped her gain the confidence to express her views in front of the village
elders for the first time.
Optional things to document
A useful addition to an SC story is a headline or title similar to what might be used in a
newspaper article. This can be a convenient handle for participants to use to refer to the
story when comparing it to others. It can also help the writer distil and communicate the
essence of what happened.
In the case of CCDB, the question 'Why is this significant to you?' was followed by an
additional question 'What difference has this made now or will it make in the future?'
Asking at the end of the story about recommendations or lessons learned can help to draw
out the implications of the story. Responses to these additional questions can be placed in
the section that describes the significance of the story.
How long should the stories be?
Most MSC stories we have seen are a page or less in length, with some being up to two
pages. Shorter MSC stories are quicker and easier to read, but they should not be so short
that vital information is left out. Different organisations tend to favour different lengths
of stories, depending on their culture. Some organisations value short and to-lhe-point
accounts of change, while others favour epic accounts told in an engaging manner The
^election process will favour Mories that lit with organisational values, and this is to be
encouraged as long as the stories are detailed enough to allow for some verification.
Reporting forms
Several organisations have developed standard formats for documenting stories. Some
examples are provided in the Appendices. This helps to ensure that important details are
not omitted. However, it is important that the form is not loo complex. The more complex
the form, the harder it is to motivate people to use and appreciate MSC. The essence of
the technique is to ask a couple of simple open-ended questions - you do not require a
structured questionnaire.
It is important though to capture sufficient detail. People who tell MSC stories often assume
that other people reading their stories will have all the background knowledge. Watch
for assumptions about background knowledge and encourage the writer to make it more
explicit. When people give hazy or unspecific answers, this may be because they think
their readers will know all the background, or they may simply not have all the details. The
more specific and detailed the MSC account is, the more credible it will be, partly because
it will be easier to verify.
Fortunately, even when people tell stories that are hazy, incomplete or totally off the track,
the process self-improves through repeated practice, selection and feedback. If you do
Implementing MSC
26
Chapter 2 * Ten Steps To implementing MSC
Most SgniTcant Change Gmo'e
encounter a hazy story, you could choose not to select that story and advise storytellers that
stories need to be more specific. This will give storytellers a better idea of what is required.
In this way the stories can become clearer and more focused with every round of collection
and feedback.
Papua New Guinea - whose voices?
“Papua New Guinean culture is an oral culture, and most Papua New Guineans are
far more comfortable with verbal expression than they are with written expression. If
such stories are to be treated seriously and the cultural environment respected, then
every effort must be made to ensure that .the authentic narrative voice of tlie speaker, is
preserved. When an English (or Australian) speaker transcribes.a story told by a Papua
New Guinean, it can easily lose that authenticity of voice, unless’ great effort is made
to ensure literal and exact transcription. Similarly, the use of'forms with sections (for
examples see Rowlands 2002; Dart and Davies 2003) or any but the lightest possible
editing can skew the storytelling.” (Elizabeth Reid, Deceinber 2004)
Whose stories to collect?
Deciding which people to ask to tell SC stones depends on the organisational context
and the subject matter ot the domains. For example, for a domain concerning changes in
people's lives, appropriate people to ask for stories would be the beneficiaries themselves,
or the people who interact with them, such as grassroots workers.
However, for a domain about 'changes in partnerships and networks with other NGOs', the
best storytellers are likely to be program staff and staff from partner organisations who are
in a position to comment.
The context of the project or program will also affect whose stories should be collected. If
the organisation is community-based and accountable to donors, it may be most appropriate
for their members to run the MSC process themselves, i.e. to share SC stories, select the
most significant ones and document them along with the reasons for their choice.
Experience suggests that stories narrated by beneficiaries are especially valuable but are
often the most difficult to elicit. Ideally, beneficiary groups would be trained in sharing
and selecting SC stories, and would report their selected story along with the reasons for
their choice. However, in some contexts this is not practical, and the storytellers by default
will be the fieldworkers. (See Step 6 for a discussion about the benefits and risks of having
beneficiaries involved in the feedback process.)
Even when the stories are to come directly from the communities, it often helps to start off
by first collecting stories from fieldworkers. This helps to ensure that staff understand the
process before introducing it to others.
Individual stories versus situational stories
We are often asked whether situational or group stories are permitted in MSC. A situational
story describes a change in a community or group, rather than being focused on an
individual. Any form of SC story is permissible in MSC. The choice will depend on what the
organisation using MSC is looking for: individual changes, group changes or institutional
changes. These options were discussed in Step 2 above in connection with choice of
Implementing MSC
27
Chapter 2 • Ten Steps To Implementing MSC
Most Significant Change Guide
domains. Because beneficiaries may not be aware of changes that are occurring in more
than one location, it is useful to also seek stories from field staff as well.
In one UK aid organisation, middle-level managers were allowed to submit their own
SC stories, which could be about larger scale and program-level changes. After a review,
however, it was realised that these staff tended to use the MSC process as just another
reporting channel. They wrote the same way as they did in their normal reports and did
not describe or explain any significant events in detail, preferring to offer bullet points and
general discussions of changes taking place. The lesson from this is that those who are
closest to where the changes are occurring or intimately involved in the program are more
likely to be able to narrate useful stories that tell us things we don't already know.
Ethics of collecting stories
Attention must be paid to the ethics of collecting stories from individuals. We suggest that
you develop processes to track consent right from start. When a storyteller tells a story, the
person collecting the story needs to explain how the story is to be used and to check that
the storyteller is happy for the story to be used. The storyteller should also be asked whether
they wish their name to accompany the story. If not, names need to be deleted from the
story from then on.
If a person or group is mentioned or identifiable within a story not told by them, ask the
storyteller to consult with the third party to check whether they are happy for their name
to be mentioned in the story. If a storyteller wants to tell a story about a third party without
naming that person, the identity of that person should be protected.
It is also worth noting that in some countries, including Australia, children under a certain
age cannot be interviewed without parental consent.
If a storyteller believes that their story is onlv going to be used for monitoring and
evaluation purposes, it would be unethical to publish the story in the local paper without
consulting the storyteller. Even when consent has been given, it is good practice to check
with storytellers before placing any stories in external media such as newspapers.
One way of making sure that ethical considerations are observed is to have a 'tick box'
on the reporting form to prompt the person recording a story to ask for the consent of the
storyteller. Appendix 2 gives an example.
Step 5: Selecting the most significant of the stories
The MSC approach uses a hierarchy of selection processes. People discuss SCs within their
area and submit the most significant of these to the level above, which then selects the most
significant of all the SCs submitted by the lower levels and passes this on to the next level.
The diagram below (Figure 2) illustrates this process.
The iterative process of selecting and then pooling SC stories helps reduce a large volume
of locally important stories down to a small number of more widely valued stories. The use
of multiple levels of selection enables this to happen without burdening any individual or
group with too much work. The process has been called 'summary by selection'.
Implementing MSC
28
Chapter 2 • Ten Steps To Implementing MSC
Most Significant Change Guide
Figure 2. Flow ofstories andfeedback in MSC
Flow of stories
Flow of feedback
Stories from story tellers
Stories from story tellers
Stories from story tellers
This hierarchical process can be structured in different ways. One way is for the structure to
'ride on the back' of the existing organisational structure. Another way is to set up specific
structures for selecting SCs.
Most organisations have a hierarchical structure with lots of field staff and one chief
executive. It makes practical sense to use this existing organisational structure to organise
the selection process. SC stories can be examined in the course of meetings already
scheduled for related purposes (such as quarterly and annual review meetings held in local
and head offices) rather than having to plan special events specifically for the analysis of
SC stories. This also helps ensure that staff al all levels of the organisation are involved in
analysing SC stones. MSC can also make use of pre-existing mechanisms for engaging
with other stakeholders. For example, the Target 10 MSC process used the pre-existing
stakeholder steering committees at regional and statewide levels.
A second reason for using existing structures is that the process of selecting SC stories can
iielp reveal the values of those within the organisation's authority structure and open these
up to discussion and change.
On the other hand, creating new structures for selecting SC stories can be useful where a
broader perspective is needed, or where the perspectives of different stakeholder groups
need to be highlighted. VSO brought senior staff members from different sections (e.g.
marketing, finance, programs) together in a single SC selection group. In CCDB, the annual
roundtable meeting with donors made use of five different SC selection groups representing
beneficiaries, junior staff, senior staff and two donor groups.
Before planning a complex SC selection process, we urge you to trial the technique in a
small way. Once you have trialled the technique and are ready to design an organisation
wide structure, there are several things you may need to consider.
•
How many levels of selection will there be above the field staff who
initially document the SC stories? This usually depends on the number
of layers of management that already exist within the organisation.
Implementing MSC
Most Significant Change Guide
Chapter 2 • Ten Steps To Implementing MSC
•
At each of these levels, how many separate selection processes
will there be? This will depend on the number of separate offices at
each level (based on location or specialisation).
•
In each of these levels, how many SC stories can be managed
by the staff involved? It is unrealistic to expect staff to meet and
work on the selection of SC stories for more than two hours at the
most. If there are four domains of change to review, this means 30
minutes for each. Within each domain, aim to read through and
discuss no more than 10 SC stories.
•
Who should participate in each selection process? This aspect is
covered in more detail below.
•
How often should selection occur? Normally this choice would be
dependent on the frequency with which SC are collected (see Step 3).
While the initial SC stories might be identified by individual fieldworkers, the selection
processes at each level in the hierarchy normally involve groups of people, not individuals.
The selection process should involve open debate rather than solitary decision-making.
Who should be involved in the selection process?
At a minimum, it should be people with line management responsibilities in relation to
the people who have forwarded the SC stories. It would be preferable to also include
people with advisory responsibilities in relation to the same staff as well as others who
would normally make use of information coming from the people who forwarded the
stories. The uppermost level would ideally involve donors, investors and other stakeholder
representatives.
Although there are many reasons to involve beneficiaries in the selection and iccdback
process, there are also some risks to be considered. Firstly, beneficiaries' time may not
be paid for in the same way as field staff and so asking beneficiaries to collect and select
stories could be seen as an unethical imposition.
It is also worth considering which field stall’ to involve in the selection process. Things can
become uncomfortable when field staff are involved in selecting stories written largely
Implementing MSC
30
Most Significant Change Gwc'e
Chapter 2 • Ten Steps To implementing MSC
by themselves. Selection appears to be easier when the stories have been written by
different people. The acceptability of self-selection seems to depend on the culture of the
organisation. When in doubt, it may be better to design a structure so that most of the SC
stories are selected by people other than those who wrote them.
In some cases, including CCDB, the people involved in documenting SC stories have also
been involved in the subsequent selection process at the district level, along with their
managers. But at the Dhaka level, the next level up, only the senior staff were involved in
the selection process.
Ghana - discomfort with story selection
“The discussion of the stories in a forum including the authors of the stories proved in
some ways uncomfortable. It appeared that the Facilitators’ work (or the effectiveness
of their input to their district) was on) trial. However we tried to overcome this
discomfort, and with whatever humour the exercise was carried out, it was never
adequately dealt with. It seems unfair? to ask the authors of the stories to step back
from their own perceptions and look at all the stories from a broader, perspective to
identify what is most significant for the project rather.-than for their own context. This
became particularly awkward when the selection team was smaller and the authors
were forced either to ‘lobby’ for their own stories or be generous to colleagues and
appear to let their own district down!” (Johnston, 2002:8)
isri>iTi;n
How to select stories
Story selection usually involves a group of people sitting down together with a pile of
documented stories that may or may not be assigned to domains. The task is to reduce the
pile ol stories to one per domain. For each domain the group will select a story that they
believe represents the most significant change of all. If the stories have not been assigned
to domains, this is the one of the first jobs to be done.
The selection process invariablv begins with reading some or all of the stories either out loud
or individually. We tend to prefer reading the stories aloud, as it brings the stories to life, but
the effectiveness and practicality of this may depend on the context. If the stories have already
been allocated to domains, then all the stories from one domain are considered together.
Various facilitated and unfacilitated processes can be used to help groups choose the most
significant of the stories. Then the reasons for the choice are documented. We encourage you
to experiment with different selection processes to find what best suits your cultural context.
While various processes can be used, the key ingredients to story selection are:
•
everybody reads the stories
•
the group holds an in-depth conversation about which stories
should be chosen
•
the group decides which stories are felt to be most significant
•
the reasons for the group's choice(s) are documented.
Implementing MSC
Most Significant Change Guide
Chapter 2 * Ten Steps To Implementing MSC
Criteria for selecting SCs
One choice that must be made is whether to identify criteria for selecting stories before
or after reading them. If the criteria are agreed beforehand, the process of learning (via
selection of SCs) will be significantly influenced by what the organisation already thinks it
knows. When the selection criteria are not discussed until after the stories have been read,
the process becomes much more open to new experiences. Personal preferences may also
be relevant. People vary in their degree of personal comfort about making judgments with
or without predefined criteria. Although there is a choice here, we believe that if MSC is
being used to aid organisational learning, the selection criteria should not be decided in
advance but should emerge through discussion of the reported changes.
There are several ways of reaching a decision about which stories to select.
Majority rules
A simple way of coming to a decision is to read the stories, make sure everyone understands
them, and then vote by show of hands. The main risk is that a choice will be made without
any substantial discussion. Arguments about the merits of different SCs are important
because they help to reveal the values and assumptions behind people's choices. Only
when this is done can participants make more informed choices about what is really of
value.
Iterative voting
In iterative voting, after the first vote, people discuss why they voted as they did. This is
followed by a second and then a third vote, ideally with some movement towards consensus.
In some cases, the participants who disagree with the majority view will eventually decide
to agree. Where they are unwilling to do so, their dissenting views can be recorded as an
important caveat to the group's main judgment: for example, about an aspect of the story
that was unclear or contradicted the main point of the story. Where groups remain mn:r
evenly split in their opinions, two stories may need to be chosen. Iterative voting can be
time-consuming, but it fosters good quality judgments.
Scoring
Instead of voting, participants can rate the value of a SC story. The ratings for each of the
stories are then aggregated and the story with the highest rating is selected as the most
significant. This is a more discriminating way of summarising judgments than a simple
show of hands. It is also a method that can be used remotely, as well as in face-to-face.
meetings. The downside is the limited opportunity for dialogue, although explanations for
ratings can be given at the same time as the ratings. Explanations are especially important
when a participant rates an SC story much higher or lower than other participants.
Pre-scoring then a group vote
This method is suitable for groups who are short of meeting time. Prior to the meeting,
participants are asked to read SC stories and rate their significance. These ratings are
summarised in a table and presented to the participants when they meet face-to-face.
Participants discuss the scores and cast their votes. Prior scoring ensures that participants
have read the stories before the meeting, and can lead to a shorter and more focused group
discussion at the meeting. The disadvantage is that all stories must be sent to participants
some time before the meeting.
Implementing MSC
32
Chapter 2 • Ten Steps To Implementing MSC
Most Sgnificant Change Gu'de
Secret ballot
It is also possible to cast votes anonymously. Each person writes their choice of SC story on
a slip of paper, and then the total votes are presented. This should be followed by an open
discussion of the reasons for the choices. This process can be surprisingly useful, especially
if there are power inequalities in the group, or if people are initially reluctant to cast their
votes publicly.
It is important to remember that in MSC, transparency is an important way of making
subjectivity accountable. Therefore, it is very important to add the second step of capturing
and discussing the reasons for choice.
To facilitate or not?
Facilitation can speed up the story-selection process and ensure equal participation by
group members. In some situations, an outside facilitator can be very useful. In the Target
10 implementation ot MSC, all the story sessions were run by trained facilitators. The
facilitation process used by Target 10 is described in Appendix 4.
It might not always be possible or appropriate to facilitate story selection. In small, informal
groups, it may not be necessary.
Documenting the results of the selection process
The reasons for selecting an SC story as the most significant should be documented and
attached to the story following the explanations given by people who initially documented
the story. The SC and the explanations for its selection are then sent on to the next level of
the selection process, if there is one. The results of the selection process should also be fed
back to all the people who provided SCs for review. Explanations that are not attached to
the stories the)' apply to will make less sense to the reader.
Because documenting the reasons for selection is usually the last task in a selection meeting,
there is a risk that this will be done too hastily and that what is written will not do justice to
the depth of discussion or the quality of the judgments made. Explanations should be more
than a tew key words, such as 'more sustainable' or 'gender equity'. Full sentences should
i.cwrl in express what was seen as significant in the selected SC storv. If multiple criteria
were used to justify selection of a story, these should be listed along with an explanation of
their relative importance.
Myanmar - forgetting to record the reasons for selection
“I had asked senior staff to sit with the small groups when they read the stories and
discussed their significance, but there were very few notes / feedback from the senior
staff on this; they got too caught up in listening to the stories to be able to step back
and identify values.” (Gillian Fletcher, 2004 (Advisor to CARE HIV/AIDS program)
The documentation attached to the most significant SC story should also record the process
used to select the story. This will provide other users of the SC stories with important
contextual knowledge, and explain the origin of the SC they are reading.
Implementing MSC
:33‘
Chapter 2 • Ten Steps To Implementing MSC
Most Significant Change Guide
What happens to the stories that are filtered out?
Stories that are filtered out should not be thrown away. They should be kept on file so that
they are accessible to others within the organisation using MSC, for as long as they continue
to use MSC, and arguably even for a while after that. This is to enable some systematic
content analysis of the full set of documented SC stories. See Step 9 in this chapter.
It is also worth noting that the SC stories that are not selected at higher levels in the
organisation still have some local value. Each story is important to the person who originally
documented it, and possibly to others at higher levels even though it was finally decided
that a different SC was more significant. It may be worthwhile following up all such stories
later on to see how they were used, or whether they had any influence on what people did.
This is discussed in Step 6.
Step 6: Feeding back the results of the selection process
The role of feedback in MSC
Feedback is important in all monitoring, evaluation and learning-oriented systems, and
MSC is no exception. The results of a selection process must be led back to those who
provided the SC stories. At the very least, this feedback should explain which SC was
selected as most significant and why. It would also help to provide information on how the
selection process was organised. In some cases, including CCDB, participants provided
more comprehensive feedback in the form of tables showing who gave which rating to
what SC story.
There are several reasons why feedback is useful. The most important of these is that
information about which SC stories were selected can aid participants' searches for SCs in
the next reporting period. Knowing that a particular type of change is valued can lead to
further searches lor similar changes in that area. The focus of the search can move to where
Implementing MSC
■ .;?
34
Most Significant Change Gude
Chapter 2 8 Ten Steps To Implementing MSC
it seems to be most needed. Feedback about why a selection was made can expand or
challenge participants' views of what is significant. Feedback about the selection process
can help participants to assess the quality of the collective judgments that were made.
Feedback also shows that others have read and engaged with the SC stories - rather than
simply filed them, which is the unfortunate fate of a lot of monitoring data.
Providing feedback about what was selected, and why and how, can potentially complete
a communication loop between different levels of participants in an organisation. In doing
so, it can create an ongoing dialogue about what is significant change.
Ibis Denmark - feedback or downward accountability?
In an MSC training workshop in October 2004,an Ibis staff member commented
“Downward accountability is called feedback—you are lucky ifyou can;get it”. Perhaps
one way to address this problem more directly would be to rename this stage in the
MSC implementation process “Downward Accountability”, to create and assert rights
to knowledge about decisions (about MSC) made by others, rather than treating
“feedback” almost as an optional item. (Rick Davies, 2004)
Different ways to provide feedback
Feedback can be provided verbally or via email, newsletters and formal reports. In the
CCBD-case, formal reports were provided after each selection meeting. In Target 10,
feedback was provided verbally at the regional level and by email to the program team; a
formal report produced after one year included funders' feedback. Some MSC users have
placed the selected stories and the reasons for their choice in community newsletters
circulated to all participants. The results of the selection process could also be disseminated
via CD-ROM, the Internet or by means of artistic activities such as pictures, videos or
dramatic re-enactment.
Benefits of feedback to the community
Placing feedback in wider forums such as community newsletters produces a range of
People can be motivated by leading stories of success and participants can gain
ideas about how they may reach their goals. As a form of celebration for what has been
achieved, it can lift the morale of staff and participants. It can also make the process more
transparent, especially if the stories in the newsletters are accompanied by the reasons that
these SC stories were selected.
Risks of giving feedback to the community
While fieldworkers have an obligation to try to achieve the stated objectives of a program,
beneficiaries may not. Giving feedback to the community about which changes the program
team does and does not value might be interpreted as the program trying to tell individuals
and communities how they should develop.
One way of overcoming this risk is to involve some beneficiaries in selecting the final stories. Then
the feedback about selected stories will come from beneficiary representatives as well as program
staff. For example, in the CCDB case, alongside the panel of funders who selected the 'winning'
stories was a panel of beneficiaries who examined the same stories and selected what they felt to
be the most significant changes. The two panels then exchanged their choices. In a similar way in
the Target 10 case, a panel of farmers selected stories in parallel with the funders.
Implementing MSC ‘
35
■
! >L.
Most Significant Change Guide
Chapter 2 • Ten Steps To Implementing MSC
Step 7: Verification of stories
Why verify?
In the right context, verification can be very useful. There is always a risk, especially in
larger organisations, that the reported changes may not reflect what has actually happened,
but instead:
be deliberate fictional accounts, designed to save time or gain
recognition
describe real events that have been misunderstood
•
exaggerate the significance of events.
A reported change may be even more important than is initially evident from the way
in which the change was documented. Important details and wider implications may lie
hidden until further investigation of the reported event.
When participants know that there are procedures for verifying SC stories, this can have
several consequences. Contributors of SCs are more likely to be careful about the way they
document their SCs and this can help improve the overall quality of the SCs. The existence
of a verification process may also give external parties more confidence in the significance
of the findings of the MSC approach.
On the other hand, undertaking some verification of SC stories may have negative consequences
if not managed properly. Participants may feel they are not trusted, and may be discouraged
from reporting anything other than what they think is expected. It may be useful to describe
follow-up inquiries as 'exploration' or another less-threatening term. Using the newspaper
metaphor to explain the MSC approach; follow-up inquiries can he explained in terms oi doing
a 'feature article' on the most significant news story of the week (month, quarter).
Implementing MSC •
36
Most Sgnificant Change Guide
Chapter 2 • Ten Steps 7b Implementing MSC
Choosing not to verify
Verification may be unnecessary in some instances. When stories are selected, they are
vetted to some degree for accuracy by those who selected them. Where most of the people
selecting the stories have background knowledge of the events described in the stories, it
may be sufficient to accept their 'vetting' as verification. This situation might arise in smallscale projects or in larger programs where the beneficiaries are represented in the selection
process.
Who verifies the stories?
It is in the interests of whoever selects a SC story as the most significant to make sure they
feel confident with the accuracy of both the SC story and the interpretations made of it.
Their judgments will normally be included in the documentation of the SC story and made
visible to other participants in the process and to users of the results.
Verification is also likely to be of concern to the most senior levels of any organisation using
MSC. The SC stories they select as most significant will be the subject of attention from both
staff and funders. CCDB gave responsibility to a staff member from their monitoring and
evaluation (M&E) unit to carry out three-monthly field visits to follow up the SC stories
selected at the Dhaka headquarters level. ADRA Laos contracted an external evaluator to
assess the SC stories and the process that generated them.
What types of MSC stories should be verified?
We do not recommend making random checks of reported changes as a method of
verification and we don't know of any organisation that has used random checks.
The best verification method is to check those changes that have been selected as most
significant at all levels: at the field level and by middle and senior management. Given
the weight of meaning attached to these reported changes, it is wise to ensure that the
loundations are secure - that the basic facts are correct.
There are points in the MSC process where verification might be given a high priority. One
is when a slorv is first accepted into the organisation: for example, when a fieldworker
documents a change reported to them. Another is when a story is communicated beyond
the organisation: tor example, to donors or the general public. A further instance is where a
story is used as the basis for recommending important changes in an organisation's policies
or procedures. This could happen at any level within an organisation using MSC, but is
more likely at the senior levels.
What aspects of MSC stories should be verified?
Both the description and interpretation aspects of MSC stories can benefit from verification.
With the descriptive part of a story, it is useful to consider whether any information is
missing and to ask how accurate the facts are. Is there enough information to enable
an independent third party to find out what happened, when and where, and who was
involved?
Il is likely that most stories will contain some errors of fact. The question is the extent to
which these errors affect the significance given to the events by the people involved or the
observer reporting the event.
Implementing MSC
37' •
Chapter 2 * Ten Steps To Implementing MSC
Most Significant Change Guide
With the interpretive part of a story, it is useful to ask whether the interpretations given to
the events are reasonable. It is often impossible to disprove an interpretation, particularly
when some information, especially about future consequences, may not be available. As
in everyday life, we can look for contradictions within the story, or with other accounts ot
the same event. It is also worth asking whether what the reporter did after documenting the
story is consistent with the contents of the story.
Mozambique - follow-up preferred
“Verification of stories was not done in the pilot study. However, many of the stories
had a character that immediately asked for further investigation. The curiosity of MS’s
program officers was awakened, and it is expected that follow-up will be done. We
found that die word ‘verification’ should not be used in external communications to
refer to such further investigations. The word was too much connected with control.”
(Sigsgciard, 2002:11)
Example
In the late 1990s, the main verification work for CCDB was undertaken by a member of
the Impact Assessment Unit at the direction of the most senior selection committee in the
Dhaka headquarters. A report based on field visits was written up and circulated to all
participating CCDB staff.
Step 8: Quantification
MSC places a strong emphasis on qualitative reporting of change, using stories rather
than numbers to communicate what is happening. However, there is also a place for
quantification of changes.
Within MSC. (here arc three ways in which quantitative inlormation can be
'
analysed. The first is within individual stories. It is possible, as with any news story, to
indicate how many people were involved, how many activities took place and to quantity
effects of different kinds.
The second method can be used after the selection of the most significant of all stories,
possibly in association with the feedback stage. For example, if the most significant of all
stories referred to a woman buying land in her own name (as in Bangladesh), all participants
could then be asked for information about all other instances of this kind of change that
they are aware of. This one-off inquiry does not need to be repeated during subsequent
reporting periods.
The third means of quantification is possible during Step 9. This method involves examining
the full set of collected SC stories, including those not selected at higher levels within the
organisation, and counting the number of times a specific type of change is noted.
Step 9: Secondary analysis and meta-monitoring
Both secondary analysis and meta-monitoring refer to an additional level of analysis that
complements the participatory selectipn of SC stories. Step 9 is not a critical step in MSC,
but in our experience it can be very useful and it adds further legitimacy and rigour to the
process.
Implementing MSC
38
Chapter 2 * Ten Steps To Implementing MSC
A/ost S-gn^cant Change Guide
Secondary analysis involves the examination, classification and analysis of the content
(or themes) across a set of SC stories, whereas meta-monitoring will focus more on the
attributes of the stories, e.g. the origins and fate of the SC stories, including who identified
them, who selected them, etc, Meta-monitoring can be done continually or periodically.
Because secondary analysis is a more in-depth look at the contents of all the stories it tends
to be done less frequently, such as once a year.
Both techniques involve analysing a complete set of SC stories including those that were
not selected at higher levels. Unlike the selection process in MSC, Step 9 is generally done
in a less participatory way, often by the person in charge of monitoring and evaluation, or
a >| ax. id I i bl.
Record keeping
In order to do either meta-monitoring or secondary analysis, all documented SC stories
need to be kept on file, regardless of how far they progressed up the hierarchy of selection
processes. In our experience, the best place to keep the SC stories is probably at the first
point within the organisation where they are documented: for example, in the field offices
of an organisation, where field staff who interact with beneficiaries are based. Some
organisations, such as MS Denmark, have gone a step further and entered their SC stories
into a text database. This would be useful for those planning to do secondary analysis at a
later stage or wanting to make the SC stories widely accessible within their organisation,
but it is not essential.
In preparation for both meta-monitoring and secondary analysis, it is also useful to develop
a supporting spreadsheet containing data about each of the SC stories, one per row. Each
column entry can provide the following types of information:
.Implementing MSC
39
.
T'
■
■
Chapter 2 • Ten Steps To Implementing MSC
Most Significant Change Guide
•
a serial number for each story
•
the title of each story
•
the date it was recorded
•
the name of the person who documented the story
•
some details about the storyteller: job, gender, region, etc
•
the date of the first selection process
•
the outcome of the selection process
•
the date of the second selection process
•
the recommendation made for follow-up action
•
what action was taken on the recommendations that were made.
Meta-monitoring
Meta-monitoring is relatively simple, it does not require expert knowledge and we strongly
recommend it. There are four main types of measures that can be monitored.
•
The total number of SC stories written in each reporting period
and how this changes over time. A larger number of SC stories
might be expected at the start of MSC as participants 'mine' all
the SC stories they can remember. A continually diminishing trend
over a long period of time might reflect disenchantment with the
use of MSC or a mistaken view that only really big changes should
be reported (see Chapter 3: Troubleshooting).
•
Who is writing stories and who is not, and how the membership
of these groups changes over time. This analysis can include
attention to differences such as men versus women, old versus
young participants, those belonging to different ethnic groups or
classes, and different locations. This may provide us with insight
into the performance of different parts of the project both in terms
of participating in MSC and in terms of achieving valued results.
For example, low numbers within some regions may reflect a
lack of understanding of MSC, or resistance to its use, but it
could also reflect real differences in what has been achieved on
the ground (the impact of the organisation's activities). Which of
these explanations best apply can be usefully discussed in staff
workshops.
•
Whose stories are being selected and whose are not. Again, this
analysis can be done in terms of gender, age, ethnicity, class and
location, according to local concerns.
•
What has happened to those SC stories. How many generated
recommendations, and how many of these recommendations were
then acted on. Again, this analysis can be done in terms of gender,
age, ethnicity, class and location, according to local concerns.
Implementing MSC
40
'■'.?sf S c-? aca.nt Change Guide
Chapter 2 * Ten Steps To Implementing MSC
Who is going to use this analysis?
There are two likely user groups. One is the staff member(s) charged with responsibility
for managing the use of MSC within their organisation. Having someone in this role can
be useful. CCDB assigned a person to be in charge of MSC and kept a person in that role
throughout the 1990s. Their responsibilities included organising verification visits to field
offices to follow up SC stories that had been selected by middle and senior level selection
processes.
The other potential user groups are boards of trustees and the organisation's donors who
receive the SC stories that come out of the top of the selection processes. These groups
need contextual information that tells them where the stories come from. This can be in two
forms. One is a short account ot how the MSC process works, in abstract. The other is some
information about how MSC worked in practice: how many stories were collected, by what
percentage of the expected participants, who was involved in the identification and then
the selection of SC stories. This is where meta-monitoring data can be very useful. Both the
CCDB and Target 10 applications made use of published annual summaries of SC stories
that included some meta-monitoring data about numbers of stories and participants.
Secondary analysis
Once you have some experience of implementing MSC, you may want to do some deeper
analysis of all the stories together. This is one means of using MSC as a component of
summative2 evaluation. However, we believe that MSC can still be a rigorous and useful
process without secondary analysis.
Secondary analysis is easier if you already have some research and analysis skills. Rick
and Jess have both experimented with various forms of secondary analysis, and it is fertile
territory lor research students. Secondary analysis is generally done in a non-participatory
wav by a researcher or a monitoring and evaluation (M&E) specialist. Some recent
innovations are described in Chapter 9.
Analysis of the range of changes described in the SC stories
There are many different ways to analyse and describe the range of changes or themes
. .. led ii. u
oi SC stories. You can find out more about these options in publications
that explain how to do qualitative analysis. In the following paragraphs we provide a brief
overview of some ways of conducting secondary analysis with a set of SC stories.
Thematic coding
One basic method of thematic coding is to search all the stories for different kinds of change.
Note every new type of change on a piece of paper and attach it to the story to remind you
what sorts ol change it refers to. Once you have examined all the stories and have no more
new types of change, remove the notes and sort them into categories that represent similar
types ol change. You can then go back through all the stories and work out which stories
reler to each type of change. This is a bit like domains, but much more specific; you may
2. Summative evaluation is generally conducted after completion of the program (or when a program
nas stabilised) and for the benefit of some external audience or decision-maker. The findings from
a summative evaluation could be used to decide whether to continue a program or not, or to justify
program spending. Formative evaluation is conducted to provide program staff with judgments useful
m improving the program. The aim of a summative evaluation is to report on the program, whereas a
formative evaluation reports to the program (Scriven 1994).
Implementing MSC
4.1
Chapter 2 • Ten Steps To Implementing MSC
Most Significant Change Guide
have listed 30 or more types of change. You can document your results in a table with the
categories of change as column headings and one row for each SC story. Each cell contains
a simple yes or no (1 or 0; tick or cross), and these can then be aggregated into totals and
percentages.
Analysing the SC stories for positive and negative changes
The incidence of negative versus positive changes is one issue that many users of MSC
are likely to make a high priority for analysis. At first view, this could be seen as a meta
monitoring task, because negative SC stories should be simple to identify and count. But
this task can be more complex than appears at first glance, and more care needs to be
taken. SC stories that appear positive may have negative aspects and vice versa. Individual
stories about successful resolution of credit repayment problems, when seen lime and
time again, also seem to signal negative developments - the growing incidence of such
problems. Participants may insert negative comments into their SC stories in quite subtle
ways. Identifying negative SC stories can be especially difficult in MSC applications that
involve translation of SC stories across languages and cultures.
Analysing the changes mentioned in MSC stories against a logic model
Stories can also be analysed by using a hierarchy of expected outcomes (i.e. a program logic model)
and scoring each story against the highest level of the hierarchy that is referred to in the story.
Bennett's hierarchy (Bennett, 1976), which describes a theory of voluntary behaviour
change in seven steps, is an example of a generic outcomes hierarchy. The first level
is inputs (1), which are the resources expended by the project. The inputs are used in
activities (2) that involve people (3) with certain characteristics. Level 4 relates to the way
these people react or respond (4) to their experiences, which can lead to changes in their
knowledge, attitudes, skills, aspirations and confidence (5); level 5 is often abbreviated to
KASAC. If these changes occur, people may then instigate practice change (6) that achieves
an end result (7), which is expressed in terms of social, economic or environmental change
level 7 is often abbreviated to SEEC. Level 6 represents the short-term impact of a project.
Level 7 represents the longer-term results.
The 'logical framework' used in planning development and aid programs is similar lo
Bennett's hierarchy, only shorter.
We have found that participants in the group selection of SC stories tend to use informal
hierarchies on an unplanned basis. For example, stories about impacts on people's lives tend lo
be rated more highly than stories about program activities that are precursors to those impacts.
If you are interested in this approach, you may need to do some research on program logic and
outcomes hierarchies, jess used this form of analysis for the Target 10 project (Dart 2000).
Analysing the genre
Content analysis can also focus on the genre people use to write MSC stories. A genre is a
large-scale categorisation of experience and includes such forms as drama, tragedy, comedy,
satire, farce and epic. These forms can tell us something about the overarching beliefs of the
organisation using MSC, and the morale of the people who work there. Rick did some analysis of
genre in his doctoral thesis, which can be found at: http://www.mande.co.uk/docs/thcsis.htm
Implementing MSC
42
■/osi Significant Change Guide
Chapter 2 * Ten Steps To Implementing MSC
Mozambique - cultural effects
“In the beginning respondents often told their stories in a very flowery, formal and
roundabout way. This was especially marked in Mozambique, and maybe due to the
Portuguese language inviting such diversions. It may also be due to a tradition of being very
‘formal’ when you report to officials or odier like persons.” (Peter Sigsgaard, 2002:11)
Analysing differences between selected stories and those not selected .
Some very interesting findings can be made by examining the differences between the
stories that were selected and those that were not. You can examine differences in many
aspects, including:
j
the types of changes
•
the storytellers
•
the long-term or short-term nature of the changes described in the story.
This type ol analysis can reveal things such as an unrepresentative proportion of stories
selected (or not selected) from a particular region. This may reflect differences in the quality
of SC stories coming from different people and locations, especially if this ratio is stable
over time. It can also indicate real differences in what has been happening on the ground.
As well as reflecting the comparative performance of different parts of the organisation, it
may also provide insight into what the organisation values.
Victoria - what secondary analysis revealed
“For example, in the Target 10 case, secondary analysis revealed several differences
between stories that were and were not selected. Stories narrated by a beneficiary were
more likely to be selected and stories that concerned higher-level outcomes (against the
logic model) were more likely to be selected.” (Jess Dart, 2000)
Similarly, looking at the differences between stories selected by different stakeholder
d::lc:cik c.s 'n dcMicd oukunies and values.
Bangladesh - preference for long-term changes
“In CCDB, the SC stories that were selected in the final selection process (an Annual
Roundtable Meeting with donors) involved changes that had taken place over a long
period of time. This seemed to be connected with both CCDB’s and the donors’
concern to establish evidence of longer-term impact. While this is understandable it
can be at the cost of not seeing short-term changes that the organisation can respond
to quickly, and thereby change the incidence of.” (Rick Davies, 1998c)
Analysing the activities or groups mentioned in stories
You can analyse SC stories to find out how often different types of beneficiaries are
represented within the full set of stories. If there is good coverage of all types of beneficiaries,
Implementing MSC
43
Chapter 2 • Ten Steps To Implementing MSC
Most Significant Change Guide
you can be more confident that the findings represent the whole population. In the case of
CCDB, the total number of beneficiary groups referred to in stories grew month by month,
so that after 10 months, more than 70 per cent of all the village groups had been the subject
of at least one story.
Analysing the length of time participants were engaged in the project
Further insight can come from analysing how long the beneficiaries (or communities)
experiencing the changes described in the story have participated in the program. In
many rural development programs, there is an expectation that longer-term participation is
related to increased positive impacts. On the other hand, there is evidence in some savings
and credit programs that the most dramatic impact on people's lives takes place shortly
after they join the program.
Analysing the selection criteria
As well as analysing the story itself, it is possible to analyse the criteria that different
groups use to select SC stories. Questions to ask include, 'Do the criteria vary across
time?' and 'Do different groups of stakeholders use different criteria to judge the stories?'.
Because the MSC process documents the criteria used by groups to select one story over
another, it provides insight into what the organisation values at any given time. It can also
be interesting to compare the criteria used by different organisations. For example, there
is tension in many organisations between concern about having an impact on people's
lives and ensuring the sustainability of the services that create impact. Tension can also
arise when there are different views of the relative importance of the social and economic
impacts of program activities.
Step 10: Revising the system
Almost all organisations that use MSC change the implementation in some way, both during
and after the introductory phase. This is a good sign, suggesting that wnic organisational
learning is taking place. Not having any revisions would be more worrying, suggesting that
MSC is being used in a ritualistic and unreflective way.
Some of these changes have been noted already in the descriptions of Steps 1 to 9. In order
of incidence, the most common changes are:
•
changes in the names of the domains of change being used:
for example, adding domains that capture negative changes, or
'lessons learned'
•
changes in the frequency of reporting: for example, from fortnightly
to monthly or from monthly to three monthly in CCDB
•
changes in the types of participants: for example, VSO allowing
middle management to submit their own SC stories
•
changes in the structure of meetings called to select the most
significant stories.
Many of the changes made by organisations using MSC arise from day-to-day reflection
about practice. In a few cases, organisations have undertaken or commissioned meta
evaluations of the MSC process. A recent example is the meta-evaluation of ADRA Laos's
use of MSC by Juliet Willetts from the Institute for Sustainable Futures at the University of
Implementing MSC
44
Most Significant Change Gu de
Chapter 2 • Ten Steps To Implementing MSC
technology, Sydney, New South Wales. Juliet's meta-evaluation examined four aspects of
the use of MSC, described as follows:
*
efficiency: how well MSC was implemented using the resources
and time available, and how the benefits of MSC compared with
the cost
•
efficacy: to what extent the purposes of using MSC were achieved
•
effectiveness: to what extent the use of MSC enabled ADRA Laos to
facilitate program improvement
•
replicability: to what extent differences in context, staffing,
programs and donors might limit the ability of other organisations
to replicate ADRA Laos's use of MSC.
Mela-evaluations of the use of MSC involve extra costs. These are most justifiable where
MSC has been implemented on a pilot basis with the aim of extending its use on a much
wider scale if it proves to be successful. This was the case with the ADRA Laos meta
evaluation.
Implementing MSC
Most Significant Change Guide
Chapter 3 • Troubleshooting
Chapter Three: Troubleshooting
he most common of the problems and concerns that people encounter when introducing
and using MSC are listed in this chapter. Some of these are also addressed in other
sections of this Guide, such as Chapter 6 on validity.
T
Concerns expressed by participants
Problems with the question
Many people have commented on the difficulty of eliciting good stories. This is often associated
with how the question has been translated - and particularly the word 'significance'.
Eliciting good stories requires some research skills - as does community development in
general. You must be able to engage with people and elicit their views. If the question isn't
working, then you may need to re-phrase it carefully. Once you find a good way of phrasing
the question in the local language, stick to it. In Bougainville, Jess found it helpful to go
through a stepped questioning process as shown in the example below.
Bougainville - issues with how to phrase the question
“I did not find it easy collecting the stories using the MSC question technique; people
did not immediately understand what I was trying to get at. This may be much easier
in Tok Pisin, but in English it needed more prompts to get at an in-depth answer. In
the end, I used a modified version of MSC where I asked the following four questions.
•
How have you been involved in the project?
•
What are the important changes that have resulted from this project for you?
•
What are the important changes that have occurred in the community
as a result of this project?
•
What problems were there?
The story seemed to emerge from any of these four questions, depending on the
experience of the participants.” (Jess Dart, workingfor Oxfam New Zealand, 2004)
Nothing has changed, so what can we report?
This response may suggest that respondents are looking for changes that can be defined
as significant in some sort of absolute sense. It helps to ask respondents to look for any
changes at all and then to identify those they think are the most significant, in relative
terms, of all the changes they have noted. For those more philosophically inclined, it may
also be worthwhile quoting Heraclitus, who reportedly said 'It is not possible to step into
the same river twice', meaning that change is taking place all the time, so it is never true
to say that nothing has changed. The idea is that if no change can be seen, the person
concerned should take a closer look.
Troubleshooting
46
Most Significant Change Guide
Chapter 3 • Troubleshooting
- JJtfl'll' (T
Covl-P 7&H' yw
~%r /Ms '
What do you mean by significant - compared to what?
Yes, it a group is to come to a judgment about what they think is most significant, it must
be by reference to some common concern. With many applications of MSC, the common
concern will be the objectives of the program, no matter how vaguely they may be defined
at that stage.
Ghana - whose perspective?
“When trying to agree on the ‘most’ significant changes during Period III, two
examples were given of changes in one district that would NOT have been significant
in another. 1110 introduction of two health insurance schemes in Jaman (eventually
selected as the most significant change in rural livelihoods) would NOT have been
significant in Asunafo, where such schemes already exist. Similarly, the bank ‘susu’.
scheme that was identified as the most significant change in service delivery in Asunafo
v.uuld not have been significant in jaman, where it already operates. This discussion
led to the conclusion that we should not be comparing the RELATIVE significance
10 THE BENEFICIARIES of the changes, but rather the relative significance
from OUR perspective. No amount of discussion could change the perfectly valid
statements that the ‘susu scheme or the health insurance schemes were of great
significance in each of the districts where they had been introduced. This was not
something that could be ‘traded’, however persuasive the arguments put forward for
either. Were we approaching the selection of‘most’ significant change with the wrong
criteria in mind?”. (Johnston, 2002:9)
This is totally subjective!
Some participants may not be comfortable with the idea that they have to exercise their
own judgment about what is a significant change, rather than make choices according to
pre-defined and agreed rules.
Troubleshooting
'
47, ” • ■ ,
Most Significant Change Guide
Chapter 3 * Troubleshooting
We suggest two ways of responding to this concern. One is to explain that by asking
people to write down their explanations for their choices of what is most significant, we are
making their subjectivity accountable. Their judgments become open to scrutiny by other
participants. Knowing that fact may also encourage participants to think carefully about
how they explain their choices. On the other hand, this very possibility may be a source of
anxiety, especially where people are not normally asked to make judgment calls!
The second response is to say that in making a judgment there is no response that is
objectively correct or incorrect. We are asking for people's interpretations. Built into those
interpretations are arguments about what values are important in a given situation. One
person may argue for the significance of a specific change on the grounds of sustainability,
another because of improvements in gender equity. Choices between these interpretations
involve choices about priorities given to different values, and this is a matter of negotiation
rather than calculation of truth.
Ghana -what is significance?
“Significance IS ‘subjective’, and the successive ‘selections’ of the process arc not meant
to invalidate the previous ones, only to reflect the varying perspectives from which the
‘selectors’ interpret the stories.” (Johnston, 2002:9)
Why do we have to select only one SC story?
For a range of reasons, participants may express concerns or even dislike about having
to choose one SC story from among the many SC stories in front of them. If they don't
understand the MSC technique, then you can explain that the process of having to make a
choice, especially in a group setting, can stimulate debate and encourage people to think
more deeply about what is involved in each story. Reluctance to choose can also have a
more social and cultural basis. Participants may want to avoid conflict or being seen as
critical of others. If this is the case, then consider different ways of structuring the selection
process. Some of the options, including voting by secret ballot, are outlined in Step 5. It
may be necessary to allow participants to select more than one story. In other difficult
settings, participants have been asked to identify SC stories that can be eliminated (i.e. to
select the least significant rather than the most significant). We have not experienced any
situation where it was impossible to devise some form of selection process.
This is too time-consuming!
Time can be a significant problem in large organisations with large numbers of beneficiaries
and staff. Selection processes should be structured so that no meeting called to select SC
stories takes more than two hours. Try circulating stories to be read before meetings or
having a facilitator at the selection meetings. Established procedures for reading, discussing,
scoring or voting then documenting agreed choices can also help. Organisations can change
the frequency of reporting SC stories. Only a few do it fortnightly, many do it monthly and
some have changed to three-monthly reporting.
The documentation of SC stories by individual participants may be considered time
consuming for a number of reasons. The process may be new and unfamiliar or participants
may not be familiar with narrative reporting. In this case, time needs to be taken to build the
capacity of people to collect stories. For example, provide positive feedback about good
practice, give examples of good practice from elsewhere, and offer refresher training (as has
been done by CCDB).
Troubleshooting
48
Most Significant Cnange Guide
Chapter 3 • Troubleshooting
Documentation of stories may be'time-consuming because staff have insufficient knowledge
of what is happening in the field and have to pay special visits to the communities to elicit
stories. This may be symptomatic of wider problems within the organisation and need
attention by managers.
Complaints about MSC being time-consuming could also be associated with poor
motivation. Participants may not see sufficient benefit from the time they spend on MSC
or they may be experiencing other pressures on their time. If participants are not seeing
benefits, then find out whether feedback is being sent and received. Have any of the stories
nominated by these participants been selected at higher levels? If not, why not? Can helpful
advice be given as to reasons why they are not being selected? If there are other pressures
on participants' time, these should be identified and addressed by their managers.
This is too competitive!
Selecting the most significant story may go against the non-competitive ethos of some
organisations and cultures. Complaints about the competitive nature of the selection process
may also reflect individual anxieties about personal performance. While a sense of competition
can often be healthy, one way of responding to these concerns is to switch the focus so that any
apparent competition is between stories rather than individuals, or between the values behind
the choice of stories being made rather than between the stories themselves.
Selection processes can also be designed to control competitive tensions. For example,
in one Australian implementation, some participants felt that the selection process was
building competition between staff. Some staff disliked the pressure that this generated. The
selection process was changed so that staff no longer voted for stories and the selection was
done by a stakeholder steering group. This seemed to solve the problem.
None of the SC stories really represent what we are doing!
This may reflect awareness of a worrying gap between expectations and reality, or between
head oft ice and held office views of reality. It may also reflect field staff reporting what
they think is expected instead of something more realistic. One pre-emptive way to
respond is during initial training in MSC. Give a clear message that factual accounts of
significant changes of any kind, both expected and unexpected, are required, and that
repeated instances of the same kind of significant change are unlikely to be selected as
>i^;’.ilicunl each consecutive reporting period. Another way to respond is through
informative feedback attached to the stories selected as most significant. This can point out
the positive features of the story and also suggest what is still missing. If the frustration is
being expressed by field staff, rather than middle or senior managers, get people to spell
out what it is that they think is missing and give an example, which can then be converted
into a story.
There is not enough detail in the story to make a judgment!
Taken at face value, this suggests that the story in question should not be selected as most
significant of all. Or that it should be sent back to the provider for more details to be
included. The same complaint may also mask anxieties about making choices, which is an
issue dealt with earlier in this section.
Why hasn't our SC story been selected as most significant of all?
In many cases it takes too long to provide feedback on the merits of all stories that were
subject to selection. In these situations, participants have to guess how their stories were
Troubleshooting
49
Most Significant Change Guide
Chapter 3 * Troubleshoo’ing
judged, on the basis of the feedback provided to them with the SC story selected as most
significant. Some participants may be better at doing this than others. The quality of the
feedback provided on the most significant story may not be as good as it could be. One
way to address the problems associated with using single case feedback is to ensure that
comments about this story are expressed in comparative terms, even referring to other
stories if necessary. Another response is to listen to signs of dissatisfaction and respond to
them case by case where this seems worthwhile.
Complaints about the results of selection processes may relate to perceived biases in the
process. For example, participants may feel that stories are being selected on the grounds
of who provided the story, rather than on a story's contents. This is a real risk, especially in
larger organisations. Respond by making the selection process as transparent as possible:
for example, by showing each participant's ratings for each story that was examined. This
risk can also be addressed through secondary analysis. With CCDB, Rick extensively
analysed whether individual Dhaka headquarter participants' choices related to the source
of stories (by location and staff member). No correlation was found, suggesting no personal
biases towards specific sources.
This sentiment can also be aroused when a region identifies a story that represents a change
they believe is particularly relevant in their region but this story is not selected at the next
level of the organisation. This occurred in one case in Australia, and led to discussions
concerning the relationship between regional and statewide priorities.
What about the negative changes? These arc being ignored!
This is a valid statement in many applications of MSC. In Chapter 2, Step 2, we outline
some different ways of responding to this concern by the use of domains.
Concerns expressed by others
Nicaragua - big changes over a period of time
“The participants in the exercises were able to identify vague or general changes over
periods of time, but were not immediately able to pinpoint the moment of change
except in the case that it was a big formal event (Grupo Consultivo, October 2003,
Nicaragua). I have found that when a general situation or change is offered, it helps to
ask the person about the first time they realized there had been a change, where were
they, what were they doing, what was the date?” (Gillian Holmes, Ibis, 2004)
Participants' explanations of their choices of stories are being ignored
In our experience in the SC selection processes, many participants focus on the description
of the SC story, and only make passing reference to (or in some instances ignore) the
explanation given by the writer for why they selected that particular SC story. The same
often goes for the other selection explanations that are later attached to the same story, as
it progresses through a number of levels of selection.
This is worrying for two reasons. Firstly, it is neglecting a specific opportunity for secondorder learning (Bateson, 1979): not the opportunity to learn how to belter achieve a
goal, but the opportunity to question and adjust the goals that are being pursued. This
questioning often does take place by participants in the selection process, during the lively
Troubleshooting
50
Most Significant Change Guide
Chapter 3 • Troubleshooting
discussions about candidate SC stories and the reasons for selecting them, but it is often not
well documented. Ignoring the attached explanations is the equivalent of leaving the SC
storywriters out of the debate and almost assuming that they had no significant opinion.
The second reason for concern is that the focus on the description of the SC change
suggests a focus on finding out what happened, as distinct from finding out who knew
what happened, and what they thought about what happened. In organisations that have
decentralised decision-making power, local actors have more autonomy and as a result
their knowledge and attitude towards what happened is of major importance. It can affect
the sustainability and replicability of the successes, and their culpability for the failures.
We therefore argue that these explanations should in fact be given special attention.
Il’ an important change is identified but misinterpreted by field staff, this could have
major consequences for the organisation. Implementing a large-scale program requires
a substantial degree of decentralisation and delegation of authority, and it is important
to monitor the quality of the judgments made by those with delegated power. Similar
concerns apply with programs implemented through sub-contracted partnerships with
other organisations. It is the participants' explanations that tell us how well their views are
aligned with ours. If views are not well aligned, a partnership may fall short of its joint goals
- or fail altogether.
1 here is no easy solution to this problem. The people who introduce MSC to an organisation
should take the earliest possible opportunity to highlight the potential for problems of this
nature to arise - and watch out for it during implementation. Leadership by example is also
important, particularly involving the most senior of the staff who are participating in MSC.
Feedback is being forgotten
In many monitoring and evaluation (M&E) systems, feedback of analysis and
recommendations is an afterthought or may even be neglected altogether. This situation
rellects (he power differences between the people who are supplying and using M&E
Troubleshooting
oo
Chapter 3 • Troubleshooting
Most Significant Change Guide
data. Field staff who supply data may not be able to demand information about how their
superiors interpret and respond to that data. This is an aspect of MSC use that needs to be
watched closely.
In Chapter 2, Step 6, we outline the characteristics of good quality feedback: clear explanations
of choices made and a transparent selection process.
The optional meta-monitoring stage (see Chapter 2, Step 9) could include investigating the
frequency with which participants reported receiving feedback, and what they thought of
the quality of that feedback.
Some possibilities for further research on possible innovations in feedback mechanisms are
described in Chapter 9.
What about gender issues?
Since Rick first wrote about CCDB's use of MSC, questions have been raised about how
MSC treats gender issues. There was nothing in the design of the MSC process for CCDB
that suggested any special attention to or concern with gender issues. For example, none of
the domains referred to changes in gender equity. Despite this, the most actively debated
aspect of the stories brought to the Annual Roundtable Meeting with CCDB's donors in
late 1994 was the nature of the relationship between a man and his wife in a story about
a successful small enterprise development. Who contributed the most and who benefited
the most? Peter Sigsgaard cites (below) a similar development with MS Denmark's use of
MSC in Tanzania.
Mozambique
the Review Team was clearly looking'for measures to satisfy their indicator
(money), mirroring the objective of income generation. The team came to appreciate
(through the SC story given to them) that this objective was not so important to the
production group, but that gender equity was in focus and had been facilitated by the
development intervention.” (Peter Sigsgaard, 2002:9)
These examples show how gender issues can be mainstreamed, within overall monitoring
of program developments, through the use of MSC - rather than being subjected to special
attention via domains or indicators. However, it cannot be assumed that gender issues
will automatically be documented, or recognised when documented, when MSC is being
used. This will depend on the participants' values and the feedback from the selection
processes.
A more interventionist option that has been tried is to use separate men and women's
groups during the SC story selection process, so that the choices of men and women are
clearly visible, and their value differences (and areas of agreement) are also clear. It is also
possible to have domains of change that ask specifically about gender issues within assisted
communities or elsewhere.
We believe there is a place for specifically directing attention to gender issues at the level
of meta-monitoring and content analysis. Il is at this stage that attention needs to be given
to the gender mix of participants and how their respective stories are treated in the selection
process. Content analysis of the stories can include coding of the different types of gender
Troubleshooting
52
Vos? S'-gnifcant Change Guide
Chapter 3 • Troubleshooting
issues that arise and how selection processes treated these issues. Were they ignored
or attended to? Were recommendations made, and if so, were the recommendations
appropriate? This analysis should be fed back to the participants to inform their subsequent
contributions.
Badly written stories
It is not uncommon for participants in selection processes to express concerns about
differences in the quality of the SC stories being examined and compared. Some are better
written than others, some are more engaging than others. If there are noticeable differences
like this, it is important that they are openly recognised; then the group can decide how
to deal with these differences. In our experience the quality of the story is rarely the main
reason for rejecting or selecting an SC - unless the SC is so totally lacking in detail that
there is nothing to go on. Instead, what participants tend to do is to weight their judgment
of the quality of the story by the apparent importance of the content of the story. A poorly
written story about important developments can get selected. But a woefully written story
will not.
One option is to ask the person who documented the story to re-write it to better convey
the changes that are of central concern. This has been done where there is evidence of
something important happening, but not enough detail. However, care needs to be taken
here. A re-write could easily lead to confusion as to whose story it really is.
Troubleshooting
53 •
Chapter 4 • Building Capability for Effective MSC
Most Significant Change Guide
Chapter Four: Building Capability for Effective MSC
"phis chapter looks at the resources an organisation may need to implement MSC.
We consider the following strategies:
•
building the capacity of the champions
•
building the capacity of the staff
•
consideration of costs and time.
Building the capacity of the MSC ‘champions’
In Chapter 2, Step 1 (How to get started and raise interest), we discussed the benefits
of having people to actively champion or promote MSC within an organisation. These
champions can:
•
excite and motivate people at the beginning
•
answer questions about the technique
•
facilitate selection of SC stories
•
encourage people to collect stories
•
ensure that feedback occurs
®
ensure that the stories are collected and organised and sent to
review meetings
•
develop protocols to ensure confidentiality of informants where
necessary
•
conduct secondary analysis.
While it helps if the champions already have some knowledge of qualitative methods and
participatory techniques, our experience suggests that the most important attributes lor
champions are enthusiasm and interest in MSC. Good facilitation skills are also useful.
Champions need to develop a sound understanding of MSC so they can address the
inevitable questions. This knowledge can be acquired in various ways. An inexpensive
approach is to read some of the existing documents on MSC (such as this Guide) and
to experiment with MSC on a small scale. Many of the organisations that now use MSC
began this way. This learning can be accelerated with some basic training in MSC. Rick
and Jess are currently planning a 'train the trainers' course for MSC (see MSC website
ht(p://groups.yahoo.com/group/MostSignificanlChanges/ for dates and locations).
Another option is to have a consultant visit the program office and work with the champions
to introduce MSC to the organisation, as well as helping the champions to build their
Building Capability
54
Most Sign+cant Change Guide
Chapter 4 • Building Capability for Effective MSC
knowledge base. Other options include staff going on secondments to other organisations
that are more experienced in using MSC.
If one person assumes leadership for MSC in an organisation, we strongly recommend
building the MSC capacity of a second or third person as well. We know of several
instances where implementation of MSC has fallen through when a champion has moved
to another job.
Building the capacity of the staff to participate fully in MSC
Getting project staff to understand MSC is a frequent stumbling block. While MSC is
relatively simple in practice, for many people it is a radically different way to monitor
and evaluate. It is often implemented in cross-cultural and bilingual contexts, where
communication of the simplest things can be a challenge. To overcome this hurdle, give
some thought to how MSC may best be communicated in your program context, and how
participants can acquire enough knowledge and skills to be able to participate. Chapter 2,
Step 1 listed some metaphors that can help to describe the role of MSC.
There are probably two main options available for building the capability of program
teams in MSC: one is through training, and the other is through mentoring and practice. In
most cases, one person has taken an active role in disseminating the technique across the
organisation. This may or ma)' not involve that person training the program staff.
How lo train people in MSC
In our experience, training generally consists of one to three days of in-house training
led by an external consultant or an internal monitoring and evaluation (M&E) specialist.
Although there are no foolproof recipes for how to train people in MSC, we can offer you
some tips.
1.
Use plenty of hands-on exercises. Jess often invites groups, early
in the training session, to take part in a role-playing exercise
where they read through some stories from a different program
context and select those that they think are most significant. Many
people find it easier to understand the process when they see it
used in a different context - otherwise people tend lo focus more
on the content of the stories. Having a go at selecting stories for
themselves helps people to get a feel for MSC.
2.
Ask participants to document their own stories in the training
session. An effective training technique is to put participants in
pairs and encourage them to interview each other to elicit their
MSC stories. Choose a topic that everyone will relate to, such as
'the most significant change in beneficiaries' lives'.
3.
Compare MSC with other techniques such as case studies and
conventional monitoring systems to help participants understand
the differences.
4.
Explain how MSC fits into the project or organisation monitoring
and evaluation framework; it is not a stand-alone technique and is
unlikely to satisfy all the accountability requirements of funders.
Building Capability
Most Significant Change Guide
Chapter 4 • Building Capability for Efleelive MSC
5.
Offer plenty of opportunity for questions and discussion. People
often need time to absorb MSC.
6.
If possible, run the training in conjunction with a facilitator who
can focus on how the participants are feeling.
7.
Once the initial training has been conducted, it helps to have a
refresher session after the first stories have been collected and
selected. This might be just a few hours long.
Laos - training is essential
“Developing interviewing skills of field staff was not part of the goals of using MSC,
but is a prerequisite for sound process and this aspect is one that requires further
attention.” (Juliet Willetts, external evaluation ofMSC use by ADRA, 2002)
Practice and improvement
If training is not an option, it may be possible to implement MSC by trial and error for
example, asking staff to document stories and providing (hem with feedback about how
they went, along with examples (the selected stories), will give them a good idea of how
to proceed. MSC has an in-built improvement cycle, so it can work without training.
However, initial training can avoid much of the confusion and frustration that program staff
sometimes feel when they are thrown into MSC without being fully orientated.
If you choose the path of practice and improvement rather than training, it helps to have
someone with a very good understanding of MSC who can answer questions, address any
confusion and design systems to minimise frustration.
•
7
T
7
“
I
Victoria, Australia - stories improved gradually over time
“Staff were given no training in MSC. We ran a number of short presentations to staff
so that they understood what MSC was, and ran a pilot - but no-one was trained. The
first stories were not so good: many were more like testimonials. But over time, the
feedback helped staff have a clearer idea about what it was all about and the stories
gradually improved, until they were all change focused.” (Jess Dart, 2000)
Considering resources and time required
There is no doubt that MSC is time-consuming. As well as the time required to collect
the stories, regular meetings must be held to select the most significant stories. This is the
most frequently voiced concern at the start of MSC implementation. Once MSC is going
smoothly, it should become quicker and more streamlined. Organisations often choose to
lengthen the reporting period after a year or so, which also reduces the amount of time the
process consumes.
In 2004, ADRA Laos became the first organisation to analyse the amount of time taken to
conduct MSC. The following text box and two tables present a picture of a time-intensive
venture. However, our experience is that MSC is not always so time-intensive. The time
taken depends on the nature of the program and the intended uses of the MSC process.
Building Capability
56
/*ost Significant Change Guide
Chapter 4 • Building Capability for Effective MSC
Laos - estimation of the time spent on MSC
“Tlie time costs of MSC were primarily in the investment in training for staff and
secondly in monthly meetings and translation. The actual collection of stories
themselves is not particularly time-intensive. The table below demonstrates the time
resources consumed during the pilot project period. It documents the total person-days
and person hours for office staff in Vientiane and project staff in the field. In addition
to the time documented below, most ADRA Australia staff attended a one hour
selection meeting and spent time reading the set of stories.” (Julia Willets, 2004)
Table 1. Time-costs for MSC activities in pilot project expressed as total person-hours or person-days
(modifiedfrom originalformat)
Aspect of MSC
ADRA Vientiane
Office Staff
Luangnamtha
Field Staff
Attapeu
Field Staff
Managing the process (days over 9 month pilot)
Training [include eval w'shopj (days over 9 month pilot)
Travel for training (days over 9 month pilot)
60 days
46 days
8 days
6 days
45 days
9 days
6 days
31 days
9 days
TOTAL DAYS
114 days
60 days
46 days
Monthly activities (conducted for six months)
Collecting stories (hours/month)
Translation and typing stories (hours/month)
Selecting stones (hours/month)
0 hrs
3 hrs
22 hrs
24 hrs
10.5 hrs
42 hrs
12 hrs
12 hrs
28 hrs
TOTAL Monthly activities (DAYS over six months)
19 days
57 days
40 days
TOTAL DAYS SPENT OVER ENTIRE PILOT PROJECT
133 days
117 days
86 days
Building Capability
Chapter 5 « MSC Within a Monitoring and Evaluation Framework
Most Significant Change Guide
Chapter Five: MSC Within a Monitoring
and Evaluation (M&E) Framework
MSC within the program cycle
ithin most organisations there are cyclic processes of planning, implementation,
review and revision. This is often referred to as the program or planning cycle. Within
this cycle a further distinction is sometimes made between monitoring and evaluation.
Distinctions can also be drawn between different forms of monitoring and different forms
of evaluation. MSC can be used for monitoring and for evaluation, and for different forms
of monitoring. All of these options are reviewed in this chapter.
W
MSC as monitoring and evaluation
MSC has been conceptualised as a monitoring tool and an evaluation tool. The distinctions
between monitoring and evaluation are blurred, and both terms can be defined in various
ways. In this Guide, we refer to monitoring as an ongoing process of information collection
primarily for the purpose of program management. As such, monitoring lends to locus
on activities and outputs. We refer to evaluation as a less-frequent process of information
collection that tends to focus more on outcomes and impacts. Both processes involve
judgments about achievements, but evaluation tends to take a wider view of an entire
program and encompass a longer period of time, often from the inception of the program
to the present.
In our view, MSC sits on the line that differentiates monitoring and evaluation, which could
help to explain why it is so difficult to describe. Like monitoring, MSC provides ongoing
data about program performance that assists program management. But MSC goes further
than most conventional forms of monitoring in that it also focuses on outcomes and.impact,
involving people in making judgments about the relative merits of different outcomes in the
form of MSC stories. In this way, MSC contributes to both monitoring and evaluation.
MSC as a specific type of monitoring
When Rick first documented MSC, he looked at the types of outcomes that could be
monitored, and noted how different forms of monitoring were needed to track these
different types of outcomes. These factors are summarised in the table below.
Table 2: Types ofoutcomes
Outcomes are
Of agreed significance
Of disagreed significance
Unexpected
Expected
Predefined indicators are
most useful
Indicators are useful and
MSC is useful
MSC within an M&E Framework
58
MSC is useful
MSC is most useful
t/.ost S-gnrfcant Change Guide
Chapter 5 • MSC Within a Monitoring and Evaluation Framework
Note that we do not consider MSC to be a substitute for more conventional monitoring of
activities and outputs against predetermined indicators such as the number of meetings held
or the number of participants within a program. Instead, MSC provides a complementary
form of monitoring and one that fills an important gap. We do not believe that MSC should
be used as the only technique in a monitoring and evaluation framework. However, where
there is no existing framework, MSC is an excellent place to start as it builds staff capacity
to capture outcomes.
The next section summarises the ways in which MSC is a complementary form of monitoring
and the gaps that it fills.
MSC tells us about unexpected outcomes
Conventional quantitative monitoring of predetermined indicators only tells us about what
we think we need to know. It does not lead us to into the realm of what we don't realise
we need to know. The difference here is between deductive and inductive approaches.
Indicators are often derived from some prior conception, or theory, of what is supposed
to happen (deductive). In contrast, MSC uses an inductive approach, through participants
making sense of events after they have happened. So a key gap that MSC fills within a
monitoring and evaluation (M&E) framework is that it helps us to monitor the 'messy'
impacts of our work - including the unexpected results, the intangible and the indirect
consequences of our work. By getting this information on a regular basis, and taking time
to reflect on what this means, groups of people can alter their direction of effort so that they
achieve more of the outcomes they value.
Ghana - changes outside the logical framework
The recognition that changes take place distinct from those anticipated as indicators in the
project logframe seems important. In die particular example of BADSP, it is highly unlikely
that many of the indicators will be met, and yet die project has seen considerable change
occurring in die districts in which it operates...” (Johnston, 2002:11)
MSC within an M&E Framework
59
Most Significant Change Guide
Chapter 5 • MSC Within a Monitoring and Evaluation Framework
MSC encourages and makes constructive use of a diversity of views
In many monitoring systems, the events of concern are defined by people distant from
where the events happen and are monitored. Indicators are often identified by senior
executive staff or senior staff specialist research units. Some organisations have tried to
improve the situation by taking the indicator identification process down the hierarchy. In
some cases this has meant using Participatory Rural Appraisal methods to obtain (he views
of the beneficiaries themselves. The problem with such an approach is the difficulty (he
organisation then finds in summarising the information produced by a diversity of locally
identified indicators.
MSC gives those closest to the events being monitored (e.g. the field staff and beneficiaries)
the right to identify a variety of stories that they think are relevant. These are then
summarised by selection when other participants choose the most significant of all (he
stories reported. Here diversity becomes an opportunity for the organisation to decide what
direction it wants to go.
MSC enables rather than directs participants
With monitoring systems that use predefined indicators, the nature of the information
and its meaning is largely defined from the outset. Data must then be collected in as
standardised a way as possible. With MSC, participants are actively encouraged to exercise
their own judgment in identifying stories and selecting stories collected by others. This
involves the use of open-ended questions such as: "From your point of view, what was the
most significant change that took place concerning the quality of people's lives in this'"
This freedom is especially important in the case of beneficiaries and fieldworkers, whose
views might not reach senior management, often as a result of day-to-day managemenl
procedures.
MSC enables broad participation
The events documented bv an organisation's monitoring svstcm are often nnalvcrd ■>
a centralised basis at senior levels of the organisation. Typically, field-level workers do
not analyse the data they collect, but simply pass the information up the hierarchy for
others to analyse. With MSC, information is not stored or processed centrally, but is
distributed throughout the organisation and processed locally. Staff do not only collect
information about events, they also evaluate that information according to their own local
perspective.
Ghana - MSC shows a richer picture
"... the wealth of material collected would never have been gathered without the
explicit attempt to monitor significant change. In itself, it provides a picture of the
context in which BADSP operates that is quite different from any that might be
developed from traditional project documentation.” (Johnston, 2002:11)
MSC puts events in context
Normally when quantitative monitoring data is analysed, it is stripped of context. Central
office staff who analyse tables of statistics sent from field offices are usually well removed
from the field sites Typically, few text comments accompany statistics sent from fieldworkers.
MSC makes use of what has been called 'thick description': detailed accounts of events
placed in their local context, where people and their views of events are visible. In the world
MSC within an M&E Framework
60
S gnAcant Change Guide
Chapter 5 * MSC Within a Monitoring and Evaluation Framework
of ordinary people, these often take the form of stories or anecdotes. In MSC monitoring,
stories are also accompanied by the writer's interpretations of what is significant.
MSC enables a changing focus on what is important
In most monitoring systems, indicators remain essentially the same each reporting period:
the same questions are asked again and again, and the focus remains the same. There
is limited scope for independent (constructive or subversive) staff adaptations of the
monitoring system. With MSC, the contents of the monitoring system are potentially far
more dynamic and adaptive, although in practice this will of course vary from organisation
to organisation. Participants choose what to report within specified domains and, less
frequently, can change the domains themselves. MSC stories can reflect real changes in the
world as well as changing views within an organisation about what is important.
MSC as program evaluation
Patton (1997) suggests that program evaluation findings can serve three primary purposes:
'rendering judgments, facilitating improvements and/or generating knowledge'. MSC can
be used for all three purposes.
Rendering judgments
As far as we know, MSC has not been used as the sole technique for producing summative
judgments of the overall success of a program. We would have serious reservations about
attempting to use MSC in this way. Most evaluations benefit by using a mix of methods (e.g.
participative and expert, deductive and inductive).
MSC can be used as an activity built into a summative evaluation or as an activity
preceding a summative evaluation. In both cases, MSC can provide a wealth of mini-case
study material to support and illustrate arguments that are developed during the evaluation.
Records of the selection processes can also provide a wealth of success criteria that should
inform the criteria being used by evaluators and any other participants in the evaluation
process (Dart and Davies, 2003).
MSC can also play a more central part in the evaluation process as a means of identifying
and aggregating the views of different stakeholders on a large scale. Rick used MSC for
ihis nurpnse in a series of evaluations of DFID-funded NGO programs in Africa and Abia.
Compared to using MSC tor monitoring, this involved a longer reference period (i.e.
changes in the last three years) and paid greater attention to obtaining MSC stories from
identifiably different stakeholder groups.
MSC can also be combined with a theory-led (deductive) approach to evaluation. Most
programs have an expectation (i.e. a theory) about when the most significant impacts
oi program activities will be most evident. In many programs, more impact is expected
to occur towards the end rather than the beginning of the program. However, in others
such as savings and credit programs, the maximum impact can occur at a different time.
For example, this could be within three months of members joining a savings and credit
group tor the first time. These predictions can be tested by collecting data on pre-defined
indicators and examining trends in the SC stories collected over the lifetime of a program.
CCDB participants were asked to examine the stories selected over the past 10 months
MSC vyithin an M&E Framework
61
Most Significant Change Guide
Chapter 5 • MSC Within a Monitoring and Evaluation Framework
and identify the most significant of all. This process could be extended to cover a longer
period of time and strengthened by asking participants to rank the stories rather than simply
selecting the most significant.
Programs also vary in the extent to which they are expected to equitably affect a large
number of beneficiaries or affect only a small number of beneficiaries. Most programs
that aim to improve service delivery expect some degree of comprehensive and equitable
coverage of beneficiaries. In contrast, programs involving research into new technologies,
such as improved rice productivity, will expect a significant number of failures - and hope
for some outstanding successes. One outstandingly successful research result will have
the potential to affect large numbers of beneficiaries when applied b) farmers nationwide.
MSC, with its focus on the 'edge of experience', may be better suited to evaluating programs
that focus on research rather than service delivery.
Generating knowledge
Patton's third purpose relates to knowledge generation via evaluation, especially knowledge
that can be exported beyond the program of concern to others that might be able to use
this knowledge. This is a typical aim of theory-led evaluation, of the kind Pawson and Tilley
propose in their 1997 book Realistic Evaluation. On the surface, MSC does not seem well
suited to this purpose, and we have not seen it used in this way. However, if we see MSC
stories as mini-case studies, it is quite conceivable that the stories could be a rich source
of hypotheses about how things work in programs. MSC could be used, in part, to identify
causal relationships between particular activities and outcomes in stories and to then
recommend systematic surveys of the incidence of these activities and their relationship to
the outcomes. This usage is an extension of Step 8: Quantification (See Chapter 2).
Facilitating improvements
MSC was originallv designed for, and seems most obviously suited to, Patton's second
evaluation purpose: to facilitate improvements. MSC can enable organisations to locus
their work towards explicitly valued directions and awa> num less valued dncutiuns. uwh
within the many positive SC stories, there are choices to be made about which ones to
respond to and which to leave aside for the time being. These choices are available through
the diversity of stories identified by participants.
Several factors affect the extent to which the use of MSC leads to actual program
improvement. SC stories are sometimes selected as most significant of all because they
confirm existing views of what the organisation should be doing. These may not lead to any
observable improvement, except perhaps in the form of greater organisational coherence
and clarity in an organisation's views about where it is going. This type of outcome might
indicate a poorly functioning MSC: the process has failed to identify a significant change,
a difference that makes a difference. This is more likely when stories are very brief or
explanations are poorly documented. In contrast, some stories do identify or imply follow
up actions that need to be taken in order to make a change. Some MSC users have tried
to capture these by including a recommendations section at the end of the reporting form
(e.g. BADSP in Ghana).
The types of changes (hat participants focus on may also be important. During
implementation of MSC, choices are made, though the selection process, about what
duration of change is of most interest. Senior staff can reinforce an organisation's focus on
long-term change by selecting appropriate stories or they can select shorter-term changes.
MSC within an M&E Framework
62
Most S.gn-.'cant Change Guide
Chapter 5 * MSC Within a Monitoring and Evaluation Framework
Il seems likely that the longer-term changes will be more difficult to quickly influence
through responses to recommendations, simply because they are long term. Conversely,
short-term changes should be easier to influence. This is a view that could be tested through
further evaluations of the use of MSC.
Frequency of reporting is another factor that affects the ability of the MSC process to
influence program improvement. In theory, the more frequently changes are tracked, the
more opportunities there are to identify whether follow-up actions are having any effect
- and to identify and respond to newly emerging issues. Equally importantly, collecting
stories more frequently enables participants to more quickly learn how to make the best use
of MSC. VSO has faced the biggest challenge in this area. Not only does VSO collect and
select stories on an annual basis, the main source of its stories is VSO volunteers working
in developing countries for an average term of two years.
Another adjustable setting that may affect how program improvement takes place is the
choice of domains. Domains can be defined in advance, applied at all levels and focused
on existing organisational objectives. They can also be defined more loosely, only applied
after significant changes are identified and include 'any other change' domains. ADRA
in Laos may be moving from domains focused on objectives to wider categories relating
to positive and negative changes. The consequences of such a change would be worth
tracking.
MSC can also affect program performance by influencing the definition, and even the
choice, of a program's objectives - as distinct from the achievement of these objectives.
While many program evaluations may benefit from examining unexpected outcomes, MSC
plavs a pivotal role in evaluating programs with less predictable outcomes. For example,
some extension programs have deliberately loose outcomes and participatory design,
often yielding a multitude of complex and diverse outcomes. These types of programs are
ideally suited to evaluation techniques that involve searching for and deliberating the value
of significant outcomes. In such programs, the refinement of MSC domains over time, as
quasi-objective statements, could be seen as a product of the process, not just as part of the
MSC technique.
fvlSC ?.nd organisations! learning
MSC can have a formative influence on organisations beyond the realm of program-specific
activities and performance. Perhaps most importantly, MSC has the potential to influence
what can be called the 'population of values' held by staff within an organisation, and
mavbe even within its associated stakeholders. In the selection process, designated people
such as funders, program staff and stakeholder committee members deliberate about how to
judge MSC stories. This involves considerable dialogue about what criteria should be used
to select winning stories. Questions like: 'Is this change sustainable?', 'Did women benefit
from this event?', 'Will donors like this outcome?' all embody views about priority values.
The choice of one story over another reinforces the importance of a particular combination
of values. At the very least, the process of discussion involved in story selection helps
participants become aware of and understand each other's values. Analysing the content
of selected stories, as discussed in Chapter 2, Step 9, can help identify the extent to which
organisational learning is taking place in terms of changes in the prevalence of particular
values.
. MSC within an M&E Framework
.
'
63
Most Significant Change Guide
Chapter 5 • MSC Within a Monitoring and Evaluation Framework
The process of dialogue has horizontal and vertical dimensions. The horizontal dimension
is between a group of participants engaged in discussing and selecting the most significant
of a set of stories. Vertical dialogue involves exchanges of views between groups of
participants at different levels, e.g. field staff, middle managers, senior managers and
donors. The vertical dimension is very important if the MSC process is to aid organisational
learning throughout the organisation, but it is also the slower of the two processes and the
most vulnerable to failure. It depends on good documentation and communication of the
results of one group's discussion to the next. The downward link is most at risk, because
those at the lower levels of an organisation rarely have authority over those above.
Other uses of MSC within programs
In addition to its monitoring and evaluation functions, MSC can also assist in:
•
fostering a more shared vision
•
helping stakeholder steering committees to steer
•
building staff capacity in evaluation
•
providing material for publicity and communications
•
providing material for training staff
•
celebrating success.
Fostering a more shared vision
Regularly discussing what is being achieved and how this is valued can contribute to a
more shared vision between those involved in MSC (e.g. the people who collect, select and
receive feedback aboqt the stories). In this way, MSC helps groups of people to make sense
of the myriad effects that their interventions cause, and to define what it is that they want
to achieve. Unlike a vision statement, the shared vision that accompanies MSC is dynamic
and can respond to changing contexts and times.
MSC within an M&E Framework
64
Most Significant Change Gude
Chapter 5 « MSC Within a Monitoring and Evaluation Framework
Helping stakeholder steering committees to steer
Especially in developed economies, many social change programs have stakeholder
steering committees. However, the task of steering a program without delving too deeply
into management issues can be challenging. MSC enables a stakeholder committee to act
as a sounding board to a program team, advising what committee members think valuable
and not so valuable in terms of the outcomes represented in SC stories.
Building staff capacity in evaluation
MSC can help to build the capacity of program staff to identify and make sense of program
impacts. Busy organisations tend to focus on what needs to be done next, rather than
searching for the impacts of what has already been done. Many organisations struggle to
demonstrate the impact of their work. MSC is an excellent way to encourage a group of
people to focus on the impact of their work. The feedback loops within MSC can ensure that
people continuously learn and improve their skills in articulating instances of significant
impact.
j wMw n we wsr
ruuviding material for publicity and communications
After several rounds of filtering, the stories that emerge from the selection process are
generally very strong, powerful accounts of program impact. These stories make excellent
material lor publicity and communications activities. An added bonus is that these stories
have been bought into by a whole group of people.
While this is a very attractive way to use the stories, care must be taken that publicity does
not drive the MSC process, which at its worst could become a propaganda machine. If an
organisation just wants success stories for publicity purposes, it would be far more efficient
to hire a reporter to go out and collect these.
It is also worth considering the ethics of using stories for publicity or communication
purposes. If a story is to be published outside an organisation, the storyteller and the people
mentioned in the story must consent to this use.
MSC within an M&E Framework
? 65-j
Most Significant Change Guide
Chapter 5 • MSC Within a Monitoring and Evaluation Framework
Providing material for training staff
The stories themselves can also be used to show new staff how the program works, and
what things yield desired results. In some schools of business management, case studies are
used as the primary teaching tool, as the focus of problem-solving tasks. Students can be
asked about how they would respond if they were working in the situation described in the
case study. Many SC stories could be converted into simple case studies, especially if they
were followed up by verification visits, which would generate more story detail.
Celebrating success
Sharing success stories can form part of a celebration process. In some programs, large
groups of beneficiaries have come together and shared SC stories and celebrated what has
been achieved. A good story can be incredibly moving and form a human and enjoyable
way of acknowledging achievements.
MSC within an M&E Framework
66
Chapter 6 » Validity and Voice in MSC
Most Significant Change Guide
Chapter Six: Validity and Voice in MSC
eople engaged in implementing MSC sometimes express concerns about validity. Like
many qualitative approaches, MSC does not rest on conventional measures of validity
such as statistical tests to determine the significance of differences. This chapter explains
why we believe MSC can be considered a valid way of drawing conclusions about such
work. We then tackle two of the more controversial aspects of MSC: the sampling technique
and the issue of bias.
P
MSC: a valid technique
The mechanisms employed by MSC to ensure validity include:
•
thick description
•
systematic process of selection
9
transparency
•
verification
9
participation
*
member checking.
Thick description
In qualitative approaches, validity is ensured by presenting solid descriptive data or thick
description (Geertz, 1973) so that there is enough internally coherent information in order
that others can attach their own interpretations. Thick description consists of closely textured
accounts of events, placed in their local context; the observer's role and subjectivity are
visible. In the world of ordinarv people, these accounts often take the form of stories or
anecdotes. SC stories are accompanied by the reviewers' reasons for selection as well as the
storyteller's reasons for telling the story. This is an even thicker level of description (a metalevel, peThaps?), which gives readers an opportunity to attach their own interpretations to a
story - and to interpret the reasons why others have selected the story.
Systematic process of selection
Validity is enhanced in MSC through a systematic process of selection. All stories are
analysed by a panel of designated stakeholders, who attach their interpretations to the
sTory. TheseTected stories may~be passecTon to another group for selection, which must also
attach jtsjnterp.reitat ions to the stories. This process is far more systematic and disciplined
(and inclusive) than the way most information would be captured from an organisation.
Transparency
This is a cornerstone for rigorous qualitative analysis. Regardless of how analysis is done,
analysts who use qualitative approaches have an obligation to monitor and report dieir own
analytical procedures and processes, as fully and truthfully as possible. The MSC process
Validity and Voice
■
: -tl
Chapter 6 • Validity and Voice m MSC
Most Significant Change Guide
emphasises transparency by systematically recording the interpretations and making them
transparent tor all to see.
This point can be highlighted by comparing MSC.with a case study approach. In a typical
case study approach, an expert researcher will decide which information is presented
in the case study"and which" is not. They will describe the methods used to capture the
data ancTth’e process of interpreting the data, but the success criteria that underpin their
interpretations are generally not transparent. With many case studies, it is difficult (o tell
if they were purposively selected (and if so, on what basis) or randomly selected. Without
this information, it is difficult for a reader to know what value to put on the events in the
case study.
Verification
This is a key step to ensure the validity of SC stories (see Chapter 2, Step 7) and can
occur at several levels. Firstly, many stories are collected by fieldworkers who regularly
observe what is happening in the field; they may choose to investigate more fully if they
are suspicious that a story is untrue or inaccurate. Secondly, mosisimes_aiiLaccQj^pan_iccl
by the names ot thosejnvolyed_in_ the .event and. the Location ol the-event - therefore
making”their origin transparent. Thirdly, during the selection process, all stories are vetted
by panels of designated people who will often have J nrdepth knowledge about the project
and will cross-check the accuracy of the stories while considering them; stories that seem
implausible or factually incorrect wilf not be selected. Finally, a selection of stories (usually
the 'winning' stories selected at the highest level of an organisation) can be externally
verified to determine whether they are accurate, in addition to following up the events that
have transpired~sTnce the story was first told.
Participation
MSC is particularly valid in the context of participatory programs. It promotes the
involvement of a wide range of stakeholders, and emplo\s_mc.lhpdb that encourage
expression of views and sharing of lessons.
One of the major challenges facing the field of international development in the last 15
years has been how to nneasure the impact of participatory projects in a manner that is in
keeping_wi.th the philosophy of these projects (Oakley et al, 1998). The overriding concern
is for the process of monitoring and evaluation to reinforce, rather than inhibit, participation
and empowerment of the program participants. External evaluation based on outside
values about what constitutes success is not appropriate in this context. In many cases,
participatory projects require participatory monitoring and evaluation (M&E) approaches
that allow stakeholders and beneficiaries to state their views about which changes are
important and which should be measured.
Member checking
This provides an additional way of adding to the validity and accuracy of the SC stories.
This involves cross-checking the documented version of the SC with the original storyteller
and the people named in the story. When one person collects a story by 'interviewing'
another, we encourage the person documenting the story to share their notes and to allow
the storyteller to edit and re-word the story until satisfied that it reflects what they were
attempting to convey. This tan simply be a matter of reading back the story after it has been
documented.
Validity and Voice
68
'/.ost
Cnange Gude
Chapter 6 • Validity and Voice in MSC
Purposive sampling
The MSC sampling technique is selective rather than inclusive. Instead of providing
information on the 'average condition' of participants, it provides information about
exceptional circumstances, particularly successful circumstances. This is referred to as
purposive sampling. Some would argue that the information that this sample technique
produces is not a reliable basis on which to make judgments about the performance of a
program.
Nevertheless, purposive sampling (or purposeful sampling) is a legitimate form of data
mquirv m qualitative research and forms a dominant part of the logic of qualitative
research. Patton states that: "The logic and power of purposeful sampling lies in selecting
information-rich cases for study in depth. Information-rich cases are those from which one
can learn a great deal about issues of central importance to the purpose of the research,
thus the term purposeful sampling."(Patton, 1990:169)
Patton describes several different strategies of purposive sampling that serve particular
evaluation purposes. The 'extreme or deviant case sampling' approach focuses on cases
that are rich j_n information because, they _are_unus.ua I. or special in some way. The MSC
sampling system uses this approach in capturing significant instances of success or failure.
The purpose is to learn from these extreme stories, and ultimately to move extension
practices more towards success and away from failure. Therefore the strategy is to select
those stories from which the most can be learned.
If the purpose of monitoring and evaluation is to precisely document the natural variation
among outcomes for beneficiaries, and you want to be able to make generalisations
about the experience of all participants, then you need a random sample that is large
enough to be representative. However, Patton (1990:170) suggests that, "... in many
instances more can be learned from intensively studying extreme or unusual cases than
can be learned from statistical depictions of what the average case is like". Another
popular option is to combine approaches so that you gain an understanding of the normal
distribution of participants as well as the extreme cases. In CCBD and Target 10, MSC
was combined with other approaches that captured the normal distribution of farmers
attending programs.
There is some evidence that extended use of MSC can lead to reporting from a much
wider range of participants than a randomly sampled survey. In CCDB, the number of
shomities (participants groups) that were the subject of SC stories grew progressively
month by month as staff continued to search for SC stories to report. After one year, more
than 70 per cent of the shomities in the MSC pilot area had been the subject of a story.
By contrast, a typical random sample survey would probably not aim to reach more than
10 per cent at the most. This suggests that in any MSC application it is worth tracking the
extent to which SC stories are being sampled from an increasing range of sources, versus
remaining concentrated on a small subset. The former trend would be more supportive
of claims of widespread impact. However, as noted above, in some programs such as
agricultural research, a dramatic finding in one of many funded research activities can
be more significant, in the longer term, than many smaller scale achievements across a
range of funded research activities.
Validity and Voice
Chapter 6 • Validity and Voice in MSC
Most Significant Change Guide
Bias in MSC
Bias towards success
MSC often tends to favour success stories rather than 'bad news'. In Target 10, about 90
per cent of stories concerned positive outcomes. The proportion in ADRA Laos ranged from
80 to 90 per cent. However, this is not necessarily a failing, because identifying what the
program can achieve when it is at its best should help move the program towards achieving
more of these positive outcomes. Designation of a specific domain to capture negative
stories (Chapter 2, Step 2) can be done if this is desired.
Subjectivity in the selection process
The MSC selection process is subjective in that it is an expression of the values of the
people on the selection panels. It is therefore important to be aware who is and who is
not represented on the selection panels. However, unlike other research approaches,
this subjectivity is another source of data about organisational values. The reasons for
selecting SC stories are recorded and documented along with the stories themselves. The
inclusion of these interpretations as another form of evaluative data affords a high level of
transparency.
Bias towards popular views
Another criticism of the MSC selection process (and all methods that strive for consensus)
is that particularly harsh or unpopular views may be silenced by the majority vote. This is a
real issue that needs to be considered. However, in our experience, the inductive process
of story selection (voting first, then identifying the criteria) is more likely to identify and
record the less-popular views than other techniques of monitoring and evaluation. Being
required to choose one significant story over another seems to encourage surprisingly open
and frank discussions.
At a broader level, MSC maintains a diversity of views rather than striving for consensus. The
risk of one story type dominating is mitigated by the fact that at each selection level new
MSC stories are introduced from other sources. Even after the most significant changes from
each domain have been selected by the most senior staff (or the donor), some branches of
the organisation will still view other stories as more significant. MSC does not produce an
absolute consensus. It is based on contending stories and ongoing debate about their merits.
“The Wisdom of Crowds”
“Diversity and independence are important because the best collective decisions
are the product of disagreement and contest, not consensus or compromise ...
Paradoxically, the best way for a group to be smart is for each person to think and act
as independendy as possible.” (The Wisdom of Crowds, Suroweicki, 2004: xix)
Bias towards the views of those who are good at telling stories
Like all monitoring and evaluation techniques, MSC favours some types of data over others.
MSC has the unusual bias of favouring the views of people who can tell a good story.
This is another good reason for not seeing MSC as a stand-alone tool for monitoring and
evaluation. However, we have seen cases where participants in the selection process were
aware that storytelling skills could have an undue influence, and so they adjusted their
assessment of stories accordingly.
Validity and Voice
70
.V'ost S
.’leant Change Gude
Chapter 6 • Validity and Voice in MSC
Issues of voice and power in MSC
In MSC, many staff, donors and other stakeholders (including participants in some cases)
can become actively involved in collecting and analysing data. MSC is one of the most
participatory monitoring and evaluation techniques available. However, in terms of who
gets a voice, it can be argued that MSC favours the inclusion of some stakeholders over
others.
The story selection process is inherently biased in favour of those people who attend the
story review sessions. The people attending the review panels may not be fully representative
of the wider voice of staff or beneficiaries. This can be offset to some extent by having a
representative spread of people involved in selecting stories, or having parallel selection
panels representing different interest groups.
Nonetheless, MSC is embedded within the discourse of the project staff and members of
the selection panels. It does not deliberately attempt to capture the opinions of those who
choose not to participate. This is a real issue, especially when using MSC for summative
evaluation, but we deal with this by combining MSC with other techniques, such as semi
structured interviews that seek the views of non-participants or critics. Another possibility
is that a researcher could seek out stones from antagonists and include them in the MSC
review process.
However, MSC does employ some mechanisms for balancing unequal voices in
organisations. As the process usually sits in a highly visible power structure, all judgments
are made much more public than they might otherwise be. Those at the top of the hierarchy
have to choose from menus of options created by those below them. Finally, the optional
'any other changes' domain opens up the breadth of change that can be placed on the
menu. Although the choices are never entirely free, because they occur in an organisational
context, MSC gives a greater voice to those at the bottom of the organisational hierarchy
than is the case with many conventional monitoring and evaluation systems.
In VSO, SC stories that involve and have been validated by other parties in addition to the
volunteer writing the story are frequently rated as more significant than those written by
the volunteer without reference to other stakeholders' views. This suggests that the process
•’ - I
stories if managed properK, can help address the risks of participating staff
reporting stories that are, intentionally or unintentionally, self-serving.
Validity and Voice
7i-
Most Significant Change Guide
Chapter 7 * MSC Compared To Other Approaches & Epistemologies
Chapter Seven: How IV1SC Compares To
Other Approaches and Epistemologies
e believe that MSC can be successfully implemented without a strong understanding
of the theory. If you just want to know about the practicalities, you may not need to
read this chapter. But for those readers who enjoy a foray into theory, this chapter examines
validity in MSC and how it fits with other approaches and epistemologies.
W
Appreciative inquiry
MSC has been likened to 'appreciative inquiry' (Hammond, 1996). Ford and Ashford (2000)
describe MSC as an example of how appreciative inquiry (Al) can be used in monitoring
and evaluation.
Appreciative inquiry is essentially a package of approaches used to study organisational
change and community development. It has a complex philosophy that engages the entire
organisational svste_m in an inquiry about wTiaT works. A central part of Al—and a facet of
’MSC—is to look at what works’and deTermine how to do more of what works. In principle,
MSC looks at positive and negative changes, but in practice the bias towards the positive
may mean the differences between MSC and Al are not so pronounced. The principles ol Al
could equally be applied to MSC.
Unlike MSC, Al is not necessarily a continuous process, although it can be ongoing. Al
involves a visioning process about ..the.future, and MSC docs not .MSC use4' si; u I
selection processes and Al does not. In terms of the program management cycle, Al is
more relevant to the planning stage, whereas MSC is more relevant to the monitoring and
evaluation stages.
Participatory monitoring and evaluation
MSC could be considered a form of participatory monitoring and evaluation: an umbrella
term that describes various types of participatory M&E conducted in the development
sector. However, MSC differs from many other forms of PM&E in that MSC data is in the
form of text-based accounts of reported changes. The way the MSC approach involves
participatory analysis and selection of stories appears to be unique. MSC also differs in the
extent to which it uses existing organisational power structures instead of trying to reach
conclusions through the use of more ad hoc and egalitarian processes.
Case studies and vignettes
Like case studies, anecdotes and vignettes used in reports and evaluations, MSC generates
data in the form of text. All these methods are similar in that they often involve thick
description (description that is rich in context) or stories.
MSC Compared
72
Vos: Sgi.rcant Change Gj'de
Chapter 7 • MSC Compared To Other Approaches & Epistemologies
However, in most evaluations that use case studies and vignettes, the reader does not
know:
•
who selected the story, and what their role was
*
how they selected the story - from how many others and over what
period
•
why they chose this story over other stories.
The MSC approach generates thick descriptions about change on a systematic, transparent
and accountable basis.
Evolutionary epistemology and MSC
Rick's writings about MSC within the discourse of_deyelopment studies and organisational
jearning have been informed by a body of theory known as evolutionary epistemology
(Campbell, 1969). Within evolutionary epistemology, evolution is seen as a learning
process, and learning by individuals is seen as a subset of this process. Learning is defined
as the selective retention of information, and information is defined as 'differences that
make a difference' (after Bateson, 1’979).
The core of the evolutionary learning process is what is known as the evolutionary
algorithm, which involves the reiteration of variation, selection_and retention processes.
This can be seen in both organic and cukuraLevpJutiQil^Populations of animals contain
diverse characteristics; some of these confer survival advantages to the animals concerned
and thus are selectively retained over time. Those animals reproduce, and a diversity of
characteristics will again emerge among their descendants. Similarly, in cultural evolution,
the meaning of a given event (e.g. circumcision) may be interpreted in a variety of ways
by people. Some of those interpretations may have a better fit with the world view of the
people concerned, and thus become more prevalent than other views held in the past.
Within this newly dominant view, further variations of interpretations may emerge, and so
on.
The MSC process within CCDB was an attempt to design a structured social process
that embodied the three elements of the evolutionarv algorithm: variation, selection and
icleimon, reiterated through time. The entities subjectT6‘selection were events, and the
associated interpretations of _these events. The environment in which the processwas taking
place was the organisation using MSC. Selection took place when field staff selectively
identified what they saw as significant changes. These changes (and interpretations of them)
were then retained through being documented and communicated to others further up the
organisational hierarchy. When grouped together at that next level, these accounts (and
interpretations) recreated a diversity, which was then subjected to further selection, and
then the retention ot the most significant of all these significant changes. The SC stories that
survived through this iterated process were those that fit the organisation best, in terms of
its values, concerns and aspirations, at that moment in time.
This process involved two levels of selection processes that relate to the concept of first
and second order learning, which was originally developed by Bateson. First order learning
involves the selection of those changes which have the best fit with a given organisational
value, or set of values, such as the importance of increasing beneficiaries' control over the
use of development aid. Second order learning involves the selection of some values from
MSC Compared
73 .
Most Significant Change Guide
Chapter 7 * MSC Compared To Other Approaches & Epistemologies
among others, which have the best fit within the organisation, according to any higher
order concerns - for example, the very survival of the organisation. There can also be an
element of self-organised selection here as well, as some values may be more consistent
with each other than others, and therefore more likely to be retained over time.
As with organic evolution, there is no guarantee of 'good' outcomes from this type ol
learning process. The 'difference that makes a difference' is that unlike many evolutionary
processes, the MSC process is transparent. By increasing the visibility ol existing processes
of organisational observation and judgment, there is more room for participants to make
conscious choices about change. And these choices become available to a wider section ol
the organisation than might normally be the case.
MSC under a constructivist lens
With a background in program evaluation, Jess tends to frame MSC within the constructivists'
subjective epistemology, focusing on the process of increasing stakeholder understanding
of the program and the way others view it._Fpr example, in MSC, stakeholders interpret their
experiences with’a program and record stories about instances they consider to represent
the niost significant change. They'a’fso record why they consider this change significant.
Therefore when a beneficiary tells a story of significant change, she or he interacts with
the world and draws meaning from it, and it is in the telling of the story that meaning is
constructed. Then when reviewers read ancFevaluate the story, they engage with it and
construct a further new meaning. When this is done in a group, construction is shared. In
MSC, the criteria used to interpret a story are clearly documented, made transparent and
kept with the story. This transparency makes the whole process even more open to new and
more sophisticated constructions of meaning, because in MSC we know who selected each
story, in what context, and for what reason.
However, MSC also includes a verification stage in which the stories can be amplified and
checked to see if the events they describe real I) occuriccl. This suggests that MSC -.m-w
conceptualised under a radical constructivist ontology, where 'facts' are considered to be
a function of multiple realities as much as values. For these reasons, Jess suggests that MSC
is best described as employing a constructivist epistemology and a realist ontology. Rick
concurs with the description, and adds an ironic twist suggesting that the MSC is a form of
'practical postmodernism'.
MSC has also been likened to some of the constructivist evaluation approaches referred
to in the international program evaluation literature. One of the best known constructivist
approaches is Fourth Generation Evaluation (FGE) (Guba & Lincoln 1989). FGE and MSC
both assume that program stakeholders have different values that need to be explored
during evaluation, but Ls_ed i ff_e ren t.methodologLes. While both are participatory, dialogical
approaches, FGE is not usually conducted as a ongoing process and does not explicitly
involve the collection of stories. It has also been argued that FGE tends to be more of
a theoretical position than a practical methodology (Fishman, 1992). MSC was and is
developing through practice; having been implemented numerous times throughout its
evolution, it is certainly practical in orientation.
MSC Compared
74
Most Syn;:.'canl Change Guxfe
Chapter 8 * The History of MSC
Chapter Eight. The History of MSC
History as process
I t is more than ten years since Rick first developed MSC in Bangladesh. Since then, a wide
I range of organisations have experimented with MSC and some have continued to use it.
In the process of applying MSC to different organisational contexts, the design has been
modified. Within each application, the details of MSC design have often been adapted in
the light of experience.
MSC has spread in a very decentralised way. For example, no single donor has said that
MSC must be used in any project designs that it will fund. Since Jess completed her PhD
on MSC in 2000, MSC has been actively promoted in Australia by her involvement in
the Australian Evaluation Society, publication of journal articles and the training she has
provided to a large number of interested organisations. More globally, the spread of MSC
has been facilitated by the establishment of the MSC mailing list, along with an associated
file repositor)' in 2001. The file repository now contains 1 8 folders detailing MSC uses in
10 countries. The usefulness of MSC was recently made evident in an excellent ADRA MSC
guide, which was developed by Robyn Kerr (Kerr, 2004) almost solely on the basis of the
documentation she found at ADRA Laos.
The Guide you are now reading is also a step towards more active facilitation, in that we are
trying to selectively summarise and disseminate some of the lessons from the last ten years.
We have tried lo_applv_a ligllDpucly, avoiding where possible the imposition of compulsory
practices and stressing the alternatives that are available when applying MSC.
Types of MSC uses
Table 3 provides a chronological history of the uses of MSC that we have been able to
identify so far. You can find documents describing many of these applications on the MSC
website at htipv/groups.yahoo.com/group/MostSignificantChanges/
The following section identifies some of the most important differences between these
applications, especially the types of settings in which MSC has been used. We have then
tried to identify the consequences or implications of those differences. The classification
process is a 'work in progress' because the information we have for many applications of
MSC is incomplete.
HistoryofMSC
’ ■ .
. .U
.
. 7’; 7....
Chapter 8 • The History ol MSC
Most Significant Change Guide
Table 3: Known applications ofMSC
Program/;/•_ •••
Year
r ■
nformant • ,
Philippines
USAID
1993
Malawi
Government of Malawi
1994
Bangladesh
CCDB
People’s Participatory Rural
Development Program
Rick Davies
1994
Ethiopia
SIDA
Community Empowerment
Program
Terry Bergdall
1996
Multiple countries
ITDG, UK
Global program
Helen Wedgewood
1996
India
Aga Khan Foundation
Aga Khan Rural Support Program | Barry Underwood
Australia
: Department of Primary
Industries, Victoria.
Target 10
1998
Mozambique
Oxfam
1998
Philippines
2000
Multiple countries
VSO
Global program
2001
Ghana
DFID
Brong Ahafo District Support
Program
Francis Johnston
2001
Pacific Islands
IDSS/AusAid
Pacific Children’s Program
Kerin Winterford
2001
Mozambique
MS Denmark
Country program
Peter Sigsgaard
2001
Zambia
MS Denmark
Country program
Peter Sigsgaard
2001
Australia
Department of Victorian
Communities (formerly
DPI)
Bestwool
Jess Dart
2002
Multiple countries
MS Denmark
Country program
Peter Sigsgaard
2002
Laos
ADRA
1997
I
2002
Rick Davies
Jess Dari
i
i
Safe Motherhood Program
Tanzania
Ferry Bergdall
Governance and Local
Democracy1 project
Leslie Dove
Robyn Keriger
Country program
Peter Sigsgaard
STREAM
1 Pat Norrish
2002
Thailand
2003
Papua New Guinea
Oxfam New Zealand
1
New Zealand Bouga'nv’lle '
| Program
i Ghana
I CARE
Country program
2003
Central America
Ibis Denmark
Country programs
2003
Australia
S.A Dept, of Education
Learning to Learn
Margot Foster
2003
Australia
Landcare
Landcare statewide
Jess Dart
2003
Multiple countries
Oxfam Australia
2004
Australia
Desert Knowledge CRC
National body
Jess Dart
2004
Serbia
SIDA
2004
Australia
Landcare
2004
Australia
I
2003
History of MSC
Pt"*
I r~
rs
, Fiona Percy
Silke Mason Westphal
Deb Elkington
Terry Bergdall
North Central Landcare
Jess Dart
Creatively Connecting
Communities
Jess Dart
76
Vosi S-oo-'cant Change Gude
Chapter 8 • The History of MSC
The ‘differences that make a difference’ in implementing MSC
Figure 3. The central differences in program context that we believe have affected the way MSC has
been implemented to date
Key difference 1: Use of MSC in ‘developed’ versus ‘developing’ countries
Some ot the most obvious differences occur between implementing MSC in a developing
economy and implementing it in a developed economy. The differences are vast, and
beyond the scope of this Guide, so we highlight only three: cross-cultural communication,
specialisation in the program context, and levels of power and social capital.
Cross-cultural communication
MSC has been used in a wide range ot developing countries including Bangladesh, Brazil,
Mozambique, Tanzania, Ghana, Philippines and Laos. A key challenge in these contexts was
the cross-cultural communication of what MSC is all about: the idea of monitoring without
History of msc
Chapter 8 • The History of MSC
Most Significant Change Guide
indicators, the meaning of 'significance7 and the need to_exercise individual judgment. In
many cases, there was also the challenge of translating guidance notes into the national or
local language and translating the documented MSC stories into English, Danish or another
language used by the foreign aid agency. In contrast, when MSC has been introduced into
organisations based in Australia and the UK, there have been no language problems and
relatively few cultural problems.
Specialisation of program focus
A common difference between the development context and the developed economy
context may well be the diversity or specialisation of focus of the programs themselves.
Implementation of MSC in developing economies has largely focused on development. In
developed economies, the programs implementing MSC have focused on such diverse things
as the way students learn, practice change in dairy farming, healthier desert communities,
natural resource management, increasing wool production, and strengthening communities
and employment.
Levels of power and social capital of the program participants
A central difference for MSC in developed and developing economies relates to the
levels of human and social capital of the program participants. In developed countries,
it has perhaps been easier to involve program participants in all levels of MSC than is
the case in developing countries. For example, in the Target 10 dairy extension project
in Victoria, Australia, community participants were represented at every level of story
selection, including the roundtable meeting of funders and key influencers. This is because
community stakeholders tend to have more power and voice in developed economies.
Key difference 2: The extent to which participants in developed economy
settings are involved in analysing stories
Within developed economies, a key variable in MSC applications has been the extent
to which community members have participated. In two Australian cases—the Be,;
wool project and some Victorian regions of the Landcare project—the first selection
of documented SC stories was done by community landholders in large group forums.
Both initiatives are highly participatory, being largely run by the landholder participants.
Landcare, for instance, is not an agency-controlled program, but a grassroots movement
of community land-managers who meet regularly, apply for government funding and
corporate sponsorship for projects, and sometimes employ facilitators.
This suggests that the degree to which participants can be involved in analysing stories is
strongly related to the extent to which the organisation or movement is participatory or
'bottom-up' in structure.
Key difference 3: Multiple-country applications versus single-country
applications
Some aid organisations have used MSC to monitor changes taking place across a range of
country programs (including VSO, ITDG, MS Denmark and Ibis Denmark). Others have
focused on monitoring changes within one country or a single program within one country
(e.g. CARE Ghana, ADRA.Laos and CCDB Bangladesh).
History of MSC
78
Mos'. Significant Change Gude
Chapter 8 • The History of MSC
The multi-country applications have been the most challenging of all uses of MSC,
particularly in the case of VSO, which works in 35 countries. VSO asks every volunteer to
complete a MSC report at the end of their two-year placement. These reports are reviewed
and selected annually at country, regional and global levels. Because the selection process
is annual, there are fewer opportunities for VSO staff to learn about MSC through repeated
practice than in most other applications, where three-monthly reporting and selection
cycles are common. A second complication is that each year a new group of volunteers
must be told about MSC and asked to complete a report at the end of their placement.
A third difficulty, which is shared by all organisations using MSC on a global scale, is
enrolling all country programs as voluntary and willing participants in the process. This can
be difficult enough with individual staff members of individual programs. At a global level,
country program managers often have substantial autonomy, so gaining their interest and
willing involvement is a special challenge. Not surprisingly, the level of compliance and
active engagement with MSC varies across countries within VSO.
Nonetheless, cross-country programs are likely to encounter these same problems
implementing any common monitoring and evaluation program. It could also be said that
MSC is extremely useful in these contexts as it allows value pluralism and the chance to
talk about these differences and make sense of them. Indeed, despite difficulties, VSO has
four years experience with MSC and continues to use it.
Key difference 4: Times of relative peace versus conflict and post-conflict
In developing countries, most single-country applications of MSC have been in the context
of development aid programs. We know of only one application in more of an emergency
context. This is the use of MSC by CARE Afghanistan, as introduced by Jon Kurtz, whose
Master's thesis includes chapters on his experimentation with MSC in Afghanistan.
The introduction of MSC was used as a basis for understanding the influence of both
organisational and contextual factors on organisational learning within the humanitarian
emergency sector. According to Kurtz (2003:73), 'experimentation with the method yielded
insights into staff perceptions of the overall purposes and value of M&E - a critical factor
affecting the abilitv of M&E to generate learning'. Kurtz concludes that 'the MSC method
appeared to build well on, and provide some much needed structure, to our previous
ciforts to improve qualitative M&E'. You can find the^e chapters in the MSC file repo^itorv
‘»1 i mp-z/groups.yanoo.com/group/MostSignificantChanges/files/
We argue that MSC should be useful in emergency contexts because what it can provide is
close to a 'real time' impact assessment (for example, in terms of effects on people's lives).
It can also provide more frequent opportunities to steer an intervention in the appropriate
direction, through periodic feedback on the stories identified as most significant of all. In
an emergency context, a shorter reporting period such as weekly or monthly would be
preferred to the quarterly cycle often used in development projects.
We know of one use of MSC in a post-conflict context, by Jess in Bougainville. Bougainville
was subject to a prolonged civil war in the late 1990s. The number of casualties is unknown,
but it is estimated that deaths and injuries were proportionately among the highest in
the world. Jess used MSC as part of an impact assessment of Oxfam New Zealand's
interventions during and after the war. Every post-conflict situation is unique, and most
tend to be extremely dynamic in the first decade of peace. Therefore interventions need to
evolve as the context becomes better understood and the situation changes. A responsive
History of MSC
79 •< •
Chapter 8 * The History of MSC
Most Significant Change Guide
program design that can adapt to these changes requires regular reflection and course
correction. In the Oxfam New Zealand Bougainville program, the objectives changed five
times over a five-year period! MSC seemed particularly appropriate because it is a form of
monitoring and evaluation (M&E) that is not based solely on checking to see whether the
original objectives have been met. The judgments made in MSC are more about whether
significant changes are being achieved; the criteria used to determine the significance of the
changes can themselves change over time.
Key difference 5: Degree of sensitivity of the program focus
As well as the more typical implementation in rural development context, MSC has also
been applied to more 'socially sensitive' topics. For example, it has been used on a Safe
Motherhood project in the Philippines, a child welfare project in the Pacific and an HIV/
AIDS program in South East Asia. The use of MSC in the latter two contexts has raised
important questions about privacy, confidentiality and appropriate methods for eliciting
stories in sensitive subject areas. For example, asking for SC stories about 'other people' that
the respondent knows about could in fact elicit stories about the respondent. There are also
concerns about the most appropriate means of verifying MSC stories in these contexts.
History of MSC
80
■Most Significant Change Guice
Chapter 9 • New Directions for MSC
Chapter Nine: New Directions for MSC
SC is still evolving and while we make some suggestions for improving the technique,
you may well find other useful ways to improve MSC or adapt it to different contexts.
We invite you to join us in exploring how MSC can be further developed and creatively
combined with other techniques and approaches in evaluation.
M
This chapter outlines some possible future directions for MSC. We begin by considering
some ways to fine-tune MSC, discuss how you could creatively combine MSC with other
approaches, and look at some innovations to the MSC process.
Fine tuning
In our experience, MSC can be fine-tuned by developing methods for:
*
incorporating insights into program planning
*
eliciting the views of program critics
’
participatory analysis of stories en masse
•
improving the feedback process.
Incorporating insights into program planning
MSC can lead to bigger improvements to programs when there is a formal process for
incorporating the lessons learned from the stories into both long-term and short-term
program planning. You can encourage this in the short term by asking those selecting MSC
stories whether they can offer any recommendations for action as a result of the story they
have selected. If the SC stories contain critical information (i.e. information about differences
that make a difference), the best SC stories will have made differences that continue on into
the future. To date, only one or two MSC report formats have included a recommendations
• < nun. We now believe it should be used more widely, if not in all MSC applications.
Another way to enhance the impact of MSC on program improvement is to have periodic
'reflections' that lead into formal program revisions. In 2004, Oxfam Australia held a series
oi annual reflections across all programs to focus on examining what significant changes
had occurred.
Including a process to elicit the views of critics
MSC does not deliberately set out to capture the opinions of community members who
chose not to participate in a program, and may not give voice to critics of the program.
Combining MSC with a process that sought out program critics would offset this bias and
provide a more comprehensive evaluation outcome. The selection process could include
stories collected from critics or have critics participate in the selection panels.
Another option is to expand or modify the range of groups who select winning stories.
There is no reason to restrict story selection to those with program responsibilities (e.g. staff
groups, steering committees and investor groups). It would be possible, for example, to
New Directions
;
\81
’
Chapter 9 * New Directions for MSC
Most Significant Change Guide
involve members of the public in discussion about which stories they did and did not value
Conducting part of the MSC process on the Internet would enable many more people to be
involved in voting for stories and explaining the different reasons behind their views. Some
organisations that use MSC (including VSO, CWS Indonesia) have started to place their SC
stories on the Internet. This process could take place either in parallel or alter the process
has been completed within the implementing organisation.
Participatory analysis of stories en masse
Nominated stakeholders could periodically analyse the stories en masse, in addition to
making judgments about the relative merit of selected groups of stories. In other words,
secondary analysis could be conducted in a participatory manner. For example, designated
stakeholders could help to identify major themes arising from the whole spectrum of stories,
including those not selected. This could form the basis of a whole program 'reflection' with
documented recommendations that lead directly into program planning.
Improving the feedback process
This process could be done better by ensuring that someone always accompanies the results
back to the story providers at the level below, rather than just sending them by document
alone. At these meetings, the story providers could be asked to guess which SC was
selected as most significant of all. This will raise their interest immediately. The messenger
should then inform them which SC was actually selected, and why. If this is different from
the prediction, then it is highly likely that there will be some animated discussions about
the differences in perspective between these providers and the people who selected stories
at the next level. The messenger will be able to feed back the essence of this discussion
to that group. These suggestions are based on Rick's one positive experience with feeding
back results of an impact assessment survey, on the same basis: asking for predictions of
expected responses, revealing the actual responses, then discussing the differences.
Combining with other approaches
MSC has different strengths and weaknesses to conventional methods of monitoring and
evaluation. It is therefore a good tool to combine with other methods and can be used
effectively as one of several methods chosen to offset different biases and meet the full
evaluation requirements. Evaluation approaches that would complement MSC could
include those that provide:
•
quantitative evidence of the spread of emergent outcomes
•
evidence of the achievement of predetermined outcomes (if these
have been articulated)
•
evidence of the 'average' experience of participants (or of
subgroups of participants) as well as exceptional outcomes
•
information on the views of non-participants and other 'victims' of
the program
•
improved knowledge with regard to the logic of the program
intervention
•
evidence of whether desired outcomes have been achieved, in
what situations and why.
New Directions
82
Most S gnricant Change Gu de
Chapter 9 • New Directions for MSC
UK-MSC can work with indicators
"The POEMS [read MSC] system and the intelligent use of indicators are not
contradictor}'. POEMS can suggest and highlight appropriate indicators of impact that
could then be employed in a more ‘formal’ impact assessment, or be built back into the
system as new domains.” (Wedgwood and Bush, 1996:5, ITDG)
Using MSC alongside program logic to create a comprehensive monitoring,
evaluation and learning framework
In the last two years, Jess has coached several organisations to integrate MSC alongside
'program logic' and reflections. Firstly, Jess facilitates program staff to develop a program
logic model to help them come to a shared understanding of who their programs are
targeting, and the underlying logic and expectations of their work with these people. The
program logic then guides the type of evidence they need to collect in order to tell the
story of their progress in achieving the intermediate impacts. This will establish a picture
of how the program has contributed to the ultimate outcomes. However, this is only one
side of the story - it only tells us the extent to which the program has achieved its expected
outcomes. MSC supplements this by helping program staff to search for emergent instances
of significant change (as seen by the participants) and come to an agreement on the value
of these changes. The third component of this model is to combine these two approaches
in a program of regular reflection.
Figure -1 shows the relationship between program logic, MSC and annual reflection. The
annual rellection examines whether there is alignment between the project-centric logic
model and what MSC reveals. It asks: 'What impact is our work having in general?' and 7s
it the right thing to do?' as well as 'Are we doing what we said we would do?' The annual
reflection is used to revise the program logic model and make recommendations for
changes in direction to be incorporated into the annual planning process.
Figure 4. How program logic, MSC and the annual reflection process work together
New Directions
Chapter 9 • New Directions for MSC
Most Significant Change Guide
Innovations
Network alternatives
Using a hierarchy of selection processes to summarise-by-selection a large range ol program
experiences fits reasonably well with the hierarchical structure of most organisations.
However, it is becoming more common to see development programs involving multiple
partners, and networks of stakeholders with various kinds ot linkages to each other. Many
have a voluntary membership and many do not have a simple hierarchy ot authority. In
these settings, summary-by-selection processes require more careful thought. When a
group selects a most significant SC story from those provided by its members, who should
it then feed the story to? In some cases there may be elected management structures that
could be used, but in many cases there will not.
The alternative, which seems to have been used in one application of MSC in Papua New
Guinea (Reid, 2004), is that results of different stakeholder group's selections are fed into
each other, for a second round of reflection, and possible readjustment of their original
judgments. This process can be repealed, until each stakeholder group's judgment stabilises.
This approach is consistent with some theoretical work on the nature of selection processes
in self-organising systems (Kauffman, 1995). The potential downside of this approach is that
it would be a more time-consuming process. In this context it is worth noting that the PNG
application was in the context of an evaluation, not an ongoing monitoring process.
A more radical use of SC stories is being proposed within (he ADB 'Making Markets Woik
Better for the Poor' (MMWB4P) project in Vietnam. A communications strategy has been
developed to ensure that research findings are communicated to and used by policy makers.
Project office staff will collect SC stories from the funded researchers they are in contact
with, and from participants in dissemination workshops. These stories will be used lor two
purposes. Firstly, to develop a better understanding of the relevant policy-making process
(this will be an MSC domain). Secondly, the contents and sources of (Imsc MS^
may shed light on the network of connections that exists between policy makers and the
project. In the original use of MSC by CCDB, a structure was deliberately set up in advance
to enable filtering of SC stories. In the ADB project, the SC stories that become available
will be used to uncover existing structures. One of the first SC stories to be documented
is shown in the box below. This is one part of a wider jigsaw puzzle, with the surrounding
parts yet to be found.
Vietnam - SC stories as jigsaw pieces: how do they connect?
“The MMWB4P Project Office received a fax copy of a page of Hansard covering some
parliamentary Q&A dated 29 November. There is a section on Vietnam with Mr
Alexander, a representative of the Foreign and Commonwealth Office (FCO), being
quizzed about what the UK is doing to help Vietnam become a market-based economy.
Mr Alexanders reply has a whole paragraph on the MMWB4P project ending with ‘For
further information on this intervention, I refer my hon. friend to the Malting Markets
Work Better for die Poor website: www.markets4poor.org”. (Rick Davies, 2004)
(The MMWB4P Project Office does not yet know how the FCO representative knew
about their project.)
New Directions
84
.‘.'JS;
-Tcant Change Guide
Chapter 9 * New Directions for MSC
MSC for process monitoring
A recent innovation is to use MSC to monitor changes in management processes within
aid programs. In Bangladesh, the Social Investment Program has contracted a third party
to monitor the processes used to help plan and fund community-level development
initiatives. MSC is one of the methods the contractor will use. Instead of significant changes
in beneficiaries' lives, however, the MSC participants (including program beneficiaries) will
be asked to identify stories about changes in the way the program is being implemented:
for example, how program staff work with villagers to develop annual development plans,
or how grants are disbursed to fund those plans.
MSC and large groups
Jess has experimented with using MSC in large group contexts in a short timeframe, as
an alternative to stories being generated and selected by small discrete groups of people.
Storytelling is conducive to large group settings and feedback from participants indicates that
the forums have been well received. This is also a good way to encourage beneficiaries to
be more centrally involved in the selection and collection process. However, it may not be
appropriate in every cultural or programmatic context, as it does tend to be very public.
For example, in 2002, Jess facilitated MSC at a forum with 90 wool producers sitting in
groups of around seven people. Each group was asked to discuss any changes that they
felt had resulted from the program. They then selected the most significant of all these
changes, and recounted this story to the large group along with the reasons why their group
selected that story above the others. A microphone was used to ensure everyone heard the
stories, which were also recorded. The atmosphere was very excited, and people embraced
the opportunity to tell their stories to the whole group. That night the stories were hastily
transcribed and documented. A stakeholder steering committee re-read the stories the next
day, selected the most significant and fed the results back to the participants in the minutes
of the workshop. These forums were conducted in three different regions, and were to be
repealed in following years.
MSC in strategic planning
In 2004, Jess experimented with a combination of appreciative inquiry and MSC for
oilategic planning, using large group processes. The process had two positive outcomes. The
resulting strategic plan was realistic and grounded in experience to a greater extent than
the average strategic plan. The other positive outcome was the high degree of ownership of
the strategic plan.
For example, MSC was used to help develop a strategic plan for the Landcare Support
Program in the North Central Region of Victoria. Around 70 volunteers (half of whom were
'beneficiaries') went into the community and interviewed a wide range of people that they
felt had important views about Landcare, including young people, mayors, agency staff and
landholders. The resulting 140 stories were screened by a steering committee before being
analysed at a two-day community forum attended by 80 people, mainly beneficiaries.
Participants were divided into groups of around eight people and asked to read a unique
pile of stories, with al least one story per domain, and to distil from each story 'what
Landcare is when it is at its very best'. They attached removable self-adhesive notes to each
story in response to this question. Each group then chose the most significant story from its
pile and read this out to the large group, along with the reasons for the choice.
New Directions
:85..
’
-
Most Significant Change Guide
Chapter 9 * New Directions for MSC
The facilitators then grouped the self-adhesive notes that distilled 'what Landcare is when it
is at its very best' into 11 key success factors and an artist drew a picture to represent each
of these. Together with the eight examples (i.e. stories) of what was valued, and the reasons
why, the key success factors were used to ensure that success factors were included in the
strategic plan. This involved developing a vision and identifying actions along the lines of
a more typical Appreciative Inquiry approach. The story analysis component of the summit
took around three hours and was well received by all participants.
MSC as a participatory component of summative evaluation
MSC can be used to ensure participatory values are included in summative evaluation
Summative evaluations typically involve an external evaluator interviewing a range of
people, collecting secondary evidence and making observations. The external evaluator
then considers the evidence and makes judgments about the extent to which the program
was worthwhile and how it could be improved. Ultimately the process depends on the
evaluator using their best judgment (based to some degree on their own values; they are
human after all) to assess the merit and worth of a program. But in highly participatory
programs, is it appropriate for an external evaluator to judge what constitutes success?
MSC can help extract local success criteria that may be more appropriate than the success
criteria developed by an outsider.
For example, in 2004, Jess conducted an external evaluation of the Oxfam New Zealand
Bougainville Program (ONZBP), which was on the verge of becoming Osi Tanala, an
independent Bougainvillian NGO. Because Jess felt that it would be inappropriate to
conduct an entirely external evaluation of what is now an autonomous local institution,
she recommended that the evaluation include some elements of participatory evaluation
based on the values of the local team. Program staff collected 1 7 significant change stories
and the evaluator (Jess) collected eight as a cross-check.
All stories were collected with the help ol an interview guide, with notes being •.r:,‘^:'
down and then read back to the informant for verification. The staff and a small number of
beneficiaries selected the most significant of the 25 stories. The evaluator used the success
criteria identified by staff as the main themes in the evaluation report. In addition to the
MSC process, the evaluator interviewed 23 key informants and 1 2 community members in
Bougainville, including some critics of the program. The final evaluation report used extracts
from the SC stories and quotations from the interviews to illustrate the key findings.
Modifications to sampling process in MSC for use in summative evaluation
A potential limitation for MSC in summative evaluation is that it captures the most significant
stories - the most successful cases. Summative evaluation generally requires data on the
spread of impact across different participant groups. With ONZBP, this limitation was
addressed by first classifying the projects as very good, good, not so good, etc. Projects
were then selected at random from each category, and staff collected MSC stories from
these project locations. In other words, the evaluation used 'stratified purposive sampling'
rather than random sampling.
Future research areas
We know of two completed PhD theses (Jess and Rick) and two Master's degree theses (Jon
Kurtz, 2003; Bettina Ringsing, 2003) that deal with MSC. We believe MSC offers plenty of
scope for further research, particularly in the following areas:
New Directions
86
Chapter 9 • New Directions for MSC
•
the proportion of MSC applications that are really about unexpected
changes, and what factors most influence this proportion: for example,
cultural, organisational, program design and MSC design factors
•
what factors have the most influence over the percentage of negative
stories that are reported, and how controllable these factors are
•
how to strengthen the feedback loop, which is known to be a weak
link in the MSC process
•
how to strengthen the link between the MSC dialogue and program
planning
•
how to strengthen MSC for use in summative evaluation
•
combining MSC with deductive approaches that develop a
program logic.
An invitation to innovate, review and communicate
Every organisation that uses MSC introduces some innovations. Every application inevitably
requires fine tuning and adaptation of the MSC process to the local context and specific
program objectives. Some of these changes will make MSC more useful, some will not. The
value of these experiments will be magnified if the methods and results can be documented
and shared with other users of MSC.
We encourage you to:
*
join the MSC mailing list to learn more about other people's
experiences. Please introduce yourself and explain who you are,
what you are doing and what you are interested in
•
document your planned use of MSC. This could include noting the
rationale for its use, and recording the guidelines for how it is to be used
*
review your actual use of MSC and document your conclusions,
especially after a pilot period and preferably at regular intervals
•
make your MSC documentation available via the MSC mailing list
and through other means such as your website.
In return, we will try to condense the lessons learned from this growing body of experience,
and produce a revised version of this Guide within the next three years.
Happy trails
Rick and Jess
February 2005
New Directions
i
87'>tT ■
:
Most Significant Change Guide
Bibliography
Bibliography
References to all sources quoted
Bateson, G. (1979), Mind and Nature: A Necessary Unity. E.P. Dutton. New York.
Bennett, C. (1976), Up The Hierarchy. Journal of Extension, March/April, USA. See also
http://www.lhccrems.nsw.gov.au/pdf xIs zip/pdf env report/Attach5-BEN NETTS.pdf
(site visited: 26 September 2004).
Campbell, D.T. (1969), Variation, Selection And Retention In Socio-Cultural Evolution.
General Systems, Vol 14, p 69-85.
CCDB (2000), Roles and Functions of the CCDB PME System. Internal CCDB Memo.
Dart, J.J. [1999a], 'The Tale Behind The Performance Story Approach', Evaluation News
and Comment, 8, No.1, pp 12-13. Link available at: http://www.clearhorizon.com.au
(site posted April 2005).
Dart, J.J. (1999b), 'A Story Approach For Monitoring Change In An Agricultural Extension
Project', Proceedings of the Association for Qualitative Research (AQR), International
Conference, Melbourne, AQR, www.latrobe.edu.au/www/aqr/offer/papers/IDarLhtm link
at: http://www.clearhorizon.com.au (site posted April 2005).
Dart, J. J. (2000a), 'Stories for Change: A new model of evaluation for agricultural
extension projects in Australia', PhD, Institute of Land and Food Resources, University of
Melbourne, Melbourne. PDF available at: http://www.clearhorizon.com.au (site posted
April 2005).
Dart, J. J. (2000b), 'Stories for Change: A systematic approach to participatory
monitoring', Proceedings of Action Research & Process Management (ALARPM) and
Partkipalor\ Action-Research 'P^R' World C'ongress Rallar.it ^uctralia
www.ballarat.edu.au/alarpm/docs/Dart, I - Paper.doc (site visited: 26 September 2004).
Dart, J.j. (2000c), Target 10 Evaluation stories, Department of Natural Resources and
Environment, Victorian State Government, Melbourne. PDF available at:
http://www.clearhorizon.com.au (site posted April 2005).
Dart, J. j. & Davies, R.j. (2003), A dialogical story-based evaluation tool: the most
significant change technique, American Journal of Evaluation 24, 137-155.
Dart, J. J., Drysdale, G., Cole, D. and Saddington, M. (2000), 'The Most Significant
Change Approach For Monitoring an Australian Extension Project', in PLA Notes, Vol. 38,
International Institute for Environment and Development, London, 47-53. PDF available
at: http://www.clearhorizon.com.au (site posted April 2005).
Davies, R.J. (1996)*lAn Evolutionary Approach To Facilitating Organisational Learning: An
Experiment By The Christian Commission For Development In Bangladesh/
http://www.mande.co.uk/docs/ccdb.htm (site visited: 26 September 2004).
Davies, R.J. (1998a), 'An Evolutionary Approach To Organisational Learning: An
Experiment By An NGO In Bangladesh', In Mosse, D., Farrington, J., and Rew, A.,
(1998), Development as Process: Concepts and Methods for Working with Complexity.
Routledge/ODL London.
Bibliography
88
r.'jsr Sgn^cant Change Gu.de
Bibliography
Davies, R.J. (1998b'), 'An Evolutionary Approach To Facilitating Organisational Learning:
An Experiment By The Christian Commission For Development In Bangladesh.' Impact
Assessment and Project Appraisal, Vol. 16. No. 3, September 1998, pp 243-250.
Davies, R.J. (1998c), Order And Diversity: Representing And Assisting Organisational
Learning In Non-Government Aid Organisations. PhD Thesis. University of Wales Swansea, http://www.mande.co.uk/thesis.htm (site visited: 26 September 2004).
Davies, R.J. (2004a), Practical Postmodernism, Or The Systematic Use Of Anecdotes. The
Evaluator. Summer Issue. UKES. London.
Davies, R.J. (2004b), 'The Emerging Communication Strategy: Engaging and Connecting
People', Consultancy report to the Making Markets Work Better for the Poor Project, ADB,
Vietnam.
Fishman, D. B. (1992), 'Postmodernism comes to program evaluation: A critical review of
Cuba and Lincoln's Fourth Generation Evaluation', Evaluation & Program Planning, 15,
no.3, 263-270.
Fletcher, G. (2004) MSC and Myanmar. Email to Rick Davies. 20 August 2004.
Ford, N., Ashford, G. (2000), Appreciative Inquiry, E-views, Issue 1 - February 2000, eds
H. MacKinnon, C.McMillin, E. Morin, The Participatory Development Forum, Ontario,
Canada, http://www.pdfoium.oig/eviews1 .html (site visited: 02 February 2004).
Geertz, C. (1973), The Interpretation of Cultures. New York: Basic Books, 1973.
Guba, E. and Lincoln, Y. (1989), Fourth Generation Evaluation. Newbury Park, CA. Sage.
Hammond, S.A. (1996), The Thin Book ofAppreciative Inquiry. Thin Book Publishing Co. Plano, TX.
Holmes, G. (2004), Ibis Monitoring and Evaluation pre-course material. Managua, Nicaragua.
Holmes, G., Petersen, L., Kirkegaard, K. (2003), Proposal: How to use the methodology 'The
Most Significant Change' in the monitoring of Ibis' thematic programs in Central America. First
draft, July 2003. Ibis. Denmark. Available at MSC mailing list file repository at:
http://groups.yahoo.coin/group/MostSignificantChanges/ (site visited: 26 September
Jobs, S. (1994) Steve Jobs and the next Big Thing. Fortune, p 23.
Johnston, F. (2002), Tales Of The Unexpected: 'Most Significant Change' Monitoring Of
The Brong Ahafo District Support Project (BADSP), Ghana November 2000 to December
2001. Unpublished report. Available at MSC mailing list file repository at:
htip://grc)ups.yahoo.com/group/MostSignificantChanges/ (site visited: 26 September
2004).
Kauffman, S. (1995), At Home In The Universe: The Search For Laws Of Complexity.
Penguin. London.
Kelly, L., Kilby, P., and Kasynathan, N., (2004), 'Impact Measurement For Ngos: Experiences
From India And Sri Lanka', Development in Practice, Vol. 14, No. 5, pp. 696-701.
Kerr, R. (2004), MSC Guide: Based on the Experience of ADRA Laos. A guide to
implementing the Most Significant Changes (MSC) monitoring system in ADRA country
Bibliography
89 ’
Bib'iography
Most Significant Change Guide
offices. ADRA. Laos. Available at MSC mailing list file repository at:
http://groups.Yahoo.com/group/MostSignificantChanges/ (site visited: 26 September 2004).
Kurtz,J. (2003) Learning Amidst Crisis: Barriers And Incentives For Organizational Learning
In Post-Taliban Afghanistan. Published thesis, Wageningen University, The Netherlands.
Three chapters available at MSC mailing list file repository at:
http://groups.yahoo.com/group/MostSignificantChanges/ (site visited: 26 September 2004).
Le Cornu, R., Peters, J., Foster, M., Barrat, R., Mellowship, D., (2003), Exploring
Perceptions of 'Significant Change' in Reforming Schools. Paper presented to:
NZARE/AARE Joint Conference, Auckland November 30 - December 3, 2003.
Available at MSC mailing list file repository at: http://groups.yahoo.com/group/
MostSignificantChanges/ (site visited 10 October 2004).
Mason, S. (2004), Review of draft MSC Guide. Email to Rick Davies. 29 October 2004.
Oakley, P., Pratt, B., Clayton, A. (1998), Outcomes and Impact: Evaluating Change in
Social Development. INTRAC. Oxford.
Patton, M. Q. (1990), Qualitative Evaluation and Research Methods. Sage Publications,
Newbury Park, CA.
Patton, M. Q. (1997), Utilization-focused Evaluation: The Next Century Text. Sage. USA.
Pawson, R. and Tilley, N. (1997), Realistic Evaluation, Sage, London.
Rainey, J. (2001), Response to MSC inquiry. Email to Rick Davies, 11 April 2001.
Reid, E. and Reid, J. (2004), Story-Based Impact Assessment: Outline of a story-telling
methodology. Available at MSC mailing list file repository at:
http://groups.yahoo.com/group/MostSignificantChanges/ (site visited: 26 September
2004).
Ringsing, B., (2003)x Learning About Advocacy: Monitoring As A Tool For Learning In Ibis
South America. MSc Thesis. MSc programme Management of Agricultural Knowledge
Systems (MAKS). Available at MSC mailing list file repository at:
http://groups.yahoo.com/group/MostSignificantChanges/ (site visited: 26 September
2004)“
Roche, C (1999), 'Impact assessment for development agencies: Learning to value
change' Oxfam. Oxford.
Scriven, M. (1994). Evaluation thesaurus (5th ea.) Newbury Park, CA. Sage.
Sigsgaard, P. (2002), MSC Approach. Monitoring without Indicators. An ongoing testing
of the MSC approach. Evaluation Journal of Australasia. Vol. 2. No. 1. Available at MSC
mailing list file repository at: http://groups.yahoo.com/group/MostSignificantChanges/ (site
visited: 26 September 2004).
Suroweicki, J. (2004), The Wisdom Of Crowds: Why The Many Are Smarter Than The Few.
Little Brown. London.
Underwood, B. (1996), Report On 'Significant Changes' Qualitative Monitoring. AKRSP.
Ahmedabad. India.
Bibliography
90
Most S'gn^ant Change Gude
Bibliography
Wedgwood, H., and Bush, A. (1996), ITDG's Experience of Participatory Evaluation
-Oriented Monitoring System (POEMS) in the International Food Production Programme.
Paper presented at INTRAC's Third International Workshop on the Evaluation of Social
Development, November 1996, The Netherlands.
Willetts, Juliet (2004), 'Most Significant Change' Pilot Project. Evaluation Report Prepared
by Institute for Sustainable Futures For ADRA Laos. Sydney. Available at MSC mailing list
tile repository at: http://groups yahoo.com/group/MostSignificainChanges/ (site visited: 26
September 2004).
Winteri’ord, K. (2003), Sharing Stories - A Participatory Approach To Monitoring And
Evaluation In The Pacific. Pacific Children's Program. International Development Support
Services, Melbourne, Australia. Available at MSC mailing list file repository at:
http://groiips.yahoo.com/group/MostSignificantChanges/ (site visited: 26 September
2004).
Bibliography
.
,
91'. .
:
Sample Story Collection Formats
Most Significant. Change Guide
Appendix 1
Sample Story Collection Formats
1.1 Landcare (The Mallee Landcare Support Strategy)
Background
The Mallee Landcare Coordinators and Facilitators would like to capture stories of significant change
that may have resulted from their work with Landcare in this region. This will help us to improve
what we are doing, enable us to celebrate the successes together as well as being accountable to our
funders.
The stories and information collected from these interviews will be used for a number of purposes
including:
•
to explore what Coordinators and Facilitators together with Landcare groups in
the Mallee have achieved already
•
to help Facilitators and Coordinators understand what people in Landcare
value, and support more of these sorts of outcomes
•
to acknowledge and publicise what has already been achieved.
Contact details
Confidentiality
We may like to use your stories for reporting to our funders, or sharing with other people in the
region - such as other people in other Landcare groups.
Do. you, (the storyteller):
• want to have your name on the story (tick one)
• consent to us using your story for publication (tick one)
Yes
Yes
No
No
_______________ .____________________________ :_______________________________
Name of storyteller*
Name of person recording story
________________________________________________
____________________________________________
Locati on
Date of recording
__________________________________________________
* (If they wish to remain anonymous, don't record their name or contact details - just write
'landholder' or some similar description.)
Appendix 1
92
'■/lost Significant Change Guide
Sample Story Collection Formats
Questions
1.
Tell me how you (the storyteller) first became involved with Landcare and what your current
involvement is:
2.
From your point of view, describe a story that epitomises the most significant change that has
resulted from Landcare in the Mallee
Appendix 1
'
•
S3
'
‘
.
'
. '
Most Significant Change Guide
Sample Story Collection Formats
3.
Why was this story significant for you?
4.
How, (if at all) has the work of the Landcare Facilitators and/or Coordinators contributed to this?
Appendix 1
94
'Jp:: Sicmlpan: Cnange Gu'de
Sample Story Collection Formats
1.2 VSO (Volunteer Report Form)
VSO FINAL REPORT - Part 2 (Volunteer Report):
to be completed by the volunteer after the final Programme Office visit to the placement
Volunteer name:
Volunteer ref & country recruited:
Job title:
Job ref:
Employer:
Country:
Name of volunteer's line manager:
Volunteer start of service date:
□
1'lease use dark (preferably black) ink and use extra paper if you w?sh:
d
Thu main purpose of this form is to enable VSO lo understand and learn from volunteers' experience in their placements..
o
Thu report may also be’used tor briefing .by, future volunteers and'extracts may be given to funders or. used by. VSQ in publicity: ;
material (as explained in the volunteer handbook)
i
.
■
.. ,<
. ■'
;
'
• •’
o
li you wish to make confidential comments, please do so on a separate sheet of paper specifying from whom they should be .
confidential. VSO will ensure that they are treated with the confidentiality you require.'
- ’ .
1 . ’
'!
• '•
|-<
• '
•
’ .
•
z\ny issues arising from the placement should be discussed with your employer or programme office. If you feel’an.issuejhas not been
dealt with reasonably by the programme office you may write to.you(Regional. ProgrammeManager-at VSO'.UK; ’
I
□
•
I (a; Describe what you think is the most significant change that you contributed to, in some way, during your
placement.
-
Where possible, choose the most significant change after discussions with your employer, colleagues and
programme officer.
There may have been many changes, great and small, positive and negative. Choose the change that you
feel is most significant.
’
Describe who was involved, what happened, where and when.
-
Include enough detail to make it understandable by someone not familiar with your placement and to
make it possible to follow up later to see if the change has continued.
-
If choosing one change is too difficult, feel free to describe more than one (using further forms).
•
I'he significant change you choose can be in:
•
ihr lives of the beneficiaries of the organisation with which vou worked,
the lives of individuals in the community where you lived,
*
colleagues with whom you worked,
-
an aspect of the organisation with which you worked, or
»
the wider policy environment
(b) Explain why you chose this particular change.
•
What difference has it made/will it make?
•
Why do you think this difference is important?
(c) If anyone other than you was involved in choosing the story explain who and how.
(cl) Are there any lessons for VSO arising from change you have written about?
.e) Give your "news story" a headline, summarising it in a few words.
I-leadline:
Appendix 1
95
-
:
.
Sample Story Collection Formats
Most Significant Change Guide
(a) Description:
(b) Explanation:
(c) Who chose the story?
(d) Lessons or recommendations for VSO:
| Date report completed:
Appendix 1
96
Most Significant Change Guide
Sample Significant Change Stories
Appendix 2
Sample Significant Change Stories
2.1 Bougainville: Osi Tanata (NGO)
Significant Change Story
Do you the storyteller:
-
want to have your name on the story (tick one)
•
consent to us using your story for publication (tick one)
Yes
Yes
No
No
Contact Details
•Name of person recording story
Wilson during MSC training
Name of storyteller’
Sebastin Kakau
Project and location
O'Kerry organic project - cycle 3
Date of recording
23th of March
When did it happen?
Over 1 year?
Title ol store?
"Growing big"
Tell me how you (the storyteller) first became involved with Osi Tanata, and what your current
involvement is:
1 used to be a member of a community project. But I left the community project in anticipation of
disputes that might occur within the community project. However, upon hearing that Osi Tanata was
giving training to grass roots, I attended some of the Osi Tanata training of project management and
book keeping and TOT.
From your point of view, describe the most significant change that has resulted from your involvement
with Osi Tanata (training/support or funding)
After the training I went back to my village and mobilized my family members to venture into organic
gardening. I decided to set up my own family project on organic gardening. Despite not having funding
from any agency 1 ventured into setting up this small project with only the knowledge that I got from Osi
Tanata. We set up our organic garden growing cabbages, capsicums, greens, tomatoes, aibika, chillies
and other things.
Currently I am thankful for what I learned from Osi Tanata, and am using it. Today my project is
progressing well. We have sold many of their produces from their organic farm. For example, for a
bed of cabbage, he is getting around K100. Now they have spent the money to buy clothes and many
other basic needs. Apart from generating income the families and the surrounding villages have enough
surplus to feed their family and others. Also some of the money is being used to start other projects such
as a trade store.
Why is this significant to you?
It is significant to me because at first I had no knowledge to run a project. Today I have a good project
running and the income from this project is being used to sustain the livelihood of my family.
Domain:
Appendix 2
I
Changes in people's lives
i•
:97' .
Must Significant Change Guide
Sample Significant Change Stones
2.2 VSO - Philippines: Angie Bamgbose
Volunteer Story:
/\t the beginning of the gender profiling, the team (volunteer and Maradeca staff and volunteers) were
afraid that the communities would perceive the gender profiling activities as a process of Christian,
Western indoctrination on Maranao communities.
□tiring the gender profiling, in all communities, the team found very rigid gendered divisions of labour:
crudely men were seen as the breadwinners and worked largely farming rice or fishing, while women
worked managing the house in all communities. Women were not participative in communitv decision
making processes and their contributions to household income, household work, and role in peace
negotiating among other things were overlooked.
The community of Dansalan was slightly different in that men were to some extent involved in
childcare and women had a louder voice in community matters. This was a source of great shame tor
the women who cried when they described how they made decisions that "should be made by men"
They felt embarrassed that they were undermining men's roles that have been ascribed by Islam and
Maranao tradition. Dansalan has experienced many displacements over the last thirty years as a result
of flooding, rido (family feuds), martial law and the conflicts between the military and rebels in 2000.
Maradeca and Obaera continued to work in the community and after five months the team returned
to Dansalan to formally assess any changes since the gender profiling activities. Six men and eleven
women gave feedback on the changes they believed had occurred in the community as a direct result
of the gender profiling. This shows that the fear of gender profiling as being indoctrination ol forced
foreign values to divide the community did not come to fruition.
It was found that women have begun working with men to repair the boo (fishing cages). All activities
associated with fishing had previously been a totally male domain. Men continue to engage in
productive work and women here acknowledge that men are more actively involved in childcare,
cooking and laundrv. These are significant departures from the rigid gendered divisions nf labour seen
only a few months earlier.
Likewise, men said that they recognize that women's work is 'heavy' as women are involved in
both housework and productive work, working in stores, mat weaving etc. Men and women in the
community now work together in market gardening to augment the family income. It can be seen that
gardens have significantly increased in size now that men and women are working together, not simply
women alone.
Men said there have been many changes because they have gained many insights from the gender
profiling, specifically for the need for men and women to work together. The community has new ideas;
there is more motivation to work with women on income generating activities, e.g. carabao, gardening;
people generally have more motivation to work, as they have been able to see that if they don't work
(raise income) then they will not develop.
Both men and women acknowledged that women are included in community decision-making and
sometimes make more decisions than men. In terms of planning and ideas it is the women that are
ahead of men in making community developments. It should be remembered that in March women
were erving as they described their lead in decision-making and now they are proud ol it. In lad
women said that they are involved in community decisions because they are 'intelligent and wise'!
Appendix 2
98
'/os: Significant Change Guide
Sample Signincant Change Stories
\ concrete example of how these issues interplay can be seen through women's n£w ownership of a
carabao. A carabao is a water buffalo. Unless a woman is given carabaos as part of her dowry, she rarely
has any role in managing them. Even if she gets carabaos as part of her dowry, she doesn't actually get
the direct benefit as traditionally only men use these in their farm work.
In Dansalan, the women said that they telt more confident to organize themselves and develop after the
gender profiling. Following the profiling, women fell more able to organize their resources and consider
how to invest the profits from the Obaera store further. Obaera's cooperative has been so successful
that ihev warned to invest their capital in a small business that would provide additional income. The
women decided to buy a carabao. The carabao is owned and controlled by the cooperative. Women
use the carabao to help carry their goods from the road, which is about a 30-minute walk through rice
fields; there is no road and this a long way to carry a 50kg sack of rice and other produce for the family
and the store.
The men need to use carabaos in farming, and in this very poor community; they have to rent one at
high cost from outside the community. Now the men don't have to go outside the community, they can
simply ask their wife, sister, or daughter to rent the Obaera carabao. This represents a turn around of
traditional gender roles with women being in control of a carabao and men having to ask women to use
lhe carabao. In addition they have to pay the women rent.
Women use the income to further develop their cooperative, which then helps to meet their needs, lhe
needs of the family and therefore community. Money stays in the community rather than going outside
lo line another person's pocket. What creativity and self-assurance these women have shown!
Fhe next phase in this enterprising business is lo breed from the carabao. Their carabao is female and
ihev intend lo use her lo breed and they would either use this carabao for the Obaera members or lo sell
and gel further income. Each month the profits are around P5, 000 ($100), a carabao costs around PI 5,
000 and so after a further three months they intend to buy another carabao for lhe Obaera members to
continue lo help with the marketing and to rent to men for their farming activities.
This relates to the cross-cutting theme of gender within the strategic aim of participation and
governance.
Why did the volunteer choose this story:
The volunteer choose this story as I think that it clearly shows that gender profiling undertaken
with sensitivitv can create changes. Gender is not a static concept but rather a dynamic one that is
continually shitting. Men and women who are able to see that a more gender equal society is a stronger
one will embrace gender.
Many people are afraid of 'gender'; afraid to 'interfere' with culture and tradition, but a more equal and
balanced society does not compromise culture or tradition as seen here.
These changes were shared by 18 community members (six men and eleven women) of Dansalan
participating in a participatory assessment of lhe impact of the gender profiling. The outcomes of this
and all oilier aspects of the gender profiling have been recorded in the documentation of the impact of
assessment of lhe PRA and in the report written by the volunteer on the gender profiling, "Women in
Conflicts: Gender Dynamics in Maranao Communities, Southern Philippines".
The staff at Maradeca continue to believe that the most significant change is the trust of a foreigner
bv lhe communities as described above in the Final Report Part 1:3. If there were no trust then the
Appendix 2
<99;
;
•;
Sample Significant Change Stories.
Most Signtfcant Change Guide
communities would not share their experiences with the openness that they did. My colleagues
within Maradeca felt that my characteristic chatter, laughter and willingness to sit with the community
contributed to the trust building, as well as transparency in talking and explaining the work olI Toscadar
and the gender profiling. I think that most people will accept that with transparency and respect, trust
across cultural divides is possible. Instead I choose to concentrate on the impact of gender as I think that
there is an ongoing lack of belief that addressing gender is important and/or can result in real change
in communities in relatively short amounts of time as seen here.
Again, by focusing on one community I do not wish to devalue the contributions of the other
communities, or the hard work and steep learning curve that was undertaken by the staff and volunteers
of Maradeca who developed considerable knowledge and skills and confidence throughout the year
that we worked together.
Programme Office Comments/Explanation
Angie Bamgbose's positive experience in gender and development work in Muslim communities is an
excellent model for peace building initiatives that incorporate a strong gender perspective. This story
was also carried by the BBC.
Appendix 2
100
‘Ats1 Significant Cnange Guide
Sample Significant Change Stories
2.3 Australia: Learning to Learn Project (Le Cornu et al, 2003)
Title: “What’s different?”
To write this story I had a conversation with Angelica about the changes
she has experienced since 2001. She was very clear about the differences
she has experienced around managing and initiating her own learning and
how she feels about this. In writing this story I have not changed what
Angelica shared with me as I believe it demonstrates significant change.
Mv name is Angelica.
Today I am in Year 5 and my school day is very different to how it was in 2001 when I was in year 3.
In 2001 I was given difficult contracts to do with a short time span and I couldn't use other areas of the
school for my learning. My contract wasn't often completed because I found it too difficult. I felt very
uncomfortable and unhappy and would get into trouble. I cried a lot and felt worried in class. I didn't
really want to come to school.
Now I feel very safe, happy and the work, which we are given, is just right for me because I get to plan
my learning in a learning plan and 1 feel I can complete the work in the time I am given. It's still really
challenging and I learn a lot and the teacher knows this. An example of this is when 1 wanted to make
an i-movie I found it really challenging but I found out how to do it and it was still fun.
1 feel very comfortable and confident with the way my learning is going now. My teacher trusts me to
use any area of the school for my learning and I use the class system to go to other areas when I need
to.
I have made more friends and I think this is because of the way I am learning. I need to work with
(lifierent people and I am getting to know other people in the class much better. If 1 need a quiet place
lor in) learning I go to the Resource Centre.
I value the way my teacher lets us learn because we get to choose where we sit as long as we work on
our task. 1 feel that I am more in control of what I learn. I feel more mature.
’ !i‘:: the wav I am trusted to use the phone, computers, photocopier and cameras for my learning plans
and 1 am now an expert with computers, cameras and other technology. In Year 3 I didn't know how to
use these and 1 wasn't allowed to.
1 feel excited about coming to school, I love being at school and I don't want to stay at home if 1 am a
bit sick. 1 feel like 1 take more responsibility for my own learning.
Comment:
Angelica has experienced two quite different styles of teaching in the past 3 years.
The first a traditional one, familiar to us all, with the work and the environment determined by the
teacher.
Angelica now plans and manages her own learning.
Her sense of empowerment is obvious.
Appendix 2
ZsOPHEA-SOCHARA
Kora ma ng al a
< Bangalore - 34 y
Sample Significant Change Stones
Most Significant Change Guide
South West Learning Circle - Responses to Angelica’s Story (14/8/03)
Criteria indicating significant change
Shift in thinking/worldview
•
A shift towards the idea that every adult in a school has a duty of care for every child.
•
Shift in power/control away from teacher and shift in world-view.
•
Learning stems from student's experiences (as opposed to old teacher-imposed topics and
themes).
•
Teachers need to believe they have responsibility for all students' learning (not just own
class).
Whole School Level
•
Common understanding of teacher and learner.
•
Need for sound philosophy and theory around that (for learners and teachers).
•
For sustainability, need change in whole school culture and structures.
•
Learning environment that is comfortable and safe but challenging.
•
Student initiated learning requires a shared philosophy.
•
Management of resources needed, e.g. structural and cultural change to facilitate changed
practice.
•
Impact on how roles are interpreted, e.g. that of Teacher Librarian.
•
Importance of teaming structures in school so people are moved along by each other.
•
Talk, discussion, moderation, interaction among teachers to get change happening.
•
Self initiated learning can be inclusive of all students if managed well.
•
Affective dimension is powerful, e.g. relationships in which there is trust in the positive intent ol
others, feeling safe and happy, student feeling trusted by teacher.
Teacher Capacities
•
Student experienced a change in practice (so people need to have experienced change - has to
make a difference to students).
•
Student having control of learning and choice.
•
Teachers able to identify and communicate about what students can do.
•
Balance student initiated learning with explicit teaching in response to students' needs (not
laissez faire).
•
Self-awareness, critical reflection by teachers needed for growth.
•
Changed methodologies.
•
Change in student's engagement levels from year 3-5.
Student capacities
•
Student aware of own feelings, e.g. feeling empowered.
•
Student able to articulate (identify) what is different.
•
Students know about the development of thinking skills and learning styles (e.g. how to use
scaffolding to advance own learning).
Appendix 2
102
$:gn.hcani Change Gu.de
Sample Story Report Format
Appendix 3
Sample Story Reporting Format
3.1 DFID-funded Brong-Ahafor District Support Programme
Identifiers
Heading
NEW ASSEMBLY MEMBERS CAUSE A STIR
Who
District Coordinating Director (DCD)
When
22 June 2001
Location
Office of the DCD - Jaman District Assembly
Domain
Operations of DA, its sub-committees and sub-structures
Description - The Story
During an emergency senior staff meeting at his office, the DCD wanted to solicit the reactions of members
! present on the attitude of some ol the newly appointed assembly members who were demanding access to
certain documents in his custody. Judging this to be the first time such demands have been made, he was at a
loss as to what to do.
According to the DCD, it seems a new breed of assembly members have been appointed to the Jaman District
j Assembly who want to put undue pressure on assembly staff in the discharge of their duties.
He complained about some of (hem he described as "gurus" representing an interest group demanding access to
certain documents in the District Assembly. The documents they wanted include the District Assembly's Trial Balance
from January to May, the handing-over notes of the former DCE and a copy of the recent auditors' draft report.
• Aller a lengthy deliberation, members present agreed that it is within the law for assembly members to demand
I access to assembly documents at any time provided they are not personal documents. Therefore, there was
; nothing wrong with their demands except that the final report of the audit exercise should be given to them
• instead of the draft copy since changes could be effected in the final report.
I The DCD accepted this with some misgivings. He thought he should seek clarification from the Regional
; Coordinating Director since this was a new dimension in the assembly. However, this prompted him to advise
jail staff members to update their work and correct all anomalies in order not to be taken unawares. It was
I also agreed that there should be regular monthly meetings for staff to be well informed before all assembly
I meetings.
i Explanation/lnterprelation
I Demands for certain documents by some new assembly members have been viewed as a challenge to the
i 'authority' of assembly staff. Hitherto, assembly members virtually 'begged' to have access to documents and
services which were sometimes ignored with excuses.
However, the new breed of assembly members who are mostly professionals and experienced in their various
fields, could make assembly staff to sit up to put things in the right order.
I If things continue like this, the rights of assembly members would be respected. They can therefore make reasonable
I demands for the development of their communities. Quality discussions would take place at all assembly meetings
j for the right decisions to be taken. This would bring the needed change in the Jaman District.
I
' Recommendations
i All assembly members should be trained in various aspects of their work-roles and responsibilities in planning,
'community involvement in decision-making, financial administration of the DA, etc., in order to have the
i confidence to demand services and contribute to the overall development of the district.
Appendix 3
103:
■
'
?: 'f
■''
yp ■
J ■:
Facilitation Guide For Story Collection
Most Significant Change Guide
Appendix 4
Facilitation Guide for Story Collection
The facilitator writes all the titles of the stories on the whiteboard, grouped by domain They leave a
space next to each story for comments e.g
Domain
Title
Comments
4
My life is getting better
Strong, written by a beneficiary, but incomplete,
story not finished.
4
Feeling empowered
Moving story, beginning middle and end.
Attribution to project is questionable. Great story,
not sure if it is about the project.
4
Better decisions for the
family
Good solid story. Heard many times before. Small
change yet crucial. Not sure about the dates
mentioned.
4
Now I understand
OK, not enough information to really understand
what is going on.
The facilitator invites volunteers to read out all the stories belonging to the first domain of
change. After each story ask
1.
•
What is this story really about ?
•
What is your opinion of the story?
2.
The facilitator writes any comments next to the title on the white board as above.
3.
When all the stories have been read out for the first domain, ask people to vote for the
story that they find most significant. Voting can be done by a show of hands
4.
When the votes have been cast, if there is a range of scores, encouiage participant '< •
discuss why they chose the story they chose. Ask questions such as:
•
Why did you choose this story above all other stories?
•
But some of you chose a different story - can you explain why you didn't
choose this story?
•
What do you think of the stories in general?
5.
Next to each story makes notes of the reasons why they were and were not selected.
6.
Once everyone has heard why certain stories were voted for above others, the facilitator
may call a second vote, this time there may be more consensus.
If there is still no consensus about which story to choose, facilitate a discussion on the options with
the group and come to an agreement, for example:
7.
•
Choose two stories to reflect the range of views
•
Decide that none of the stories adequately represents what is valued
•
Choose one story but add a caveat explaining that not all people voted for
this story because...
Move onto the next domain.
Appendix 4
104
- Media
16713.pdf
Position: 701 (14 views)