6948.pdf
Media
- extracted text
-
NATIONAL WORKSHOP ON THE REPORT CARD APPROACH
October 25 -27, 1999
ORGANISED BY: PUBLIC AFFAIRS CENTRE, BANGALORE
RESOURCE KIT
PREPARED BY:
K. GOPAKUMAR
PRASANN THATTE
*
PUBLIC AFFAIRS CENTRE
BANGALORE
IMPROVING GOVERNANCE
HOW CITIZENS ' VOICE' CAN MAKE A DIFFERENCE!
he role of the State in the development and progress In developing
I
t
countries has often been significant and positive. There is,
however, a growing concern that the productivity of these
investments has left much to be desired and that the dominant role that
the government has played has not been matched by a high level of
public accountability and good governance. State monopoly has often
. < .
resulted in non-responsiveness to the people, inefficiency and
corruption. Though, the recent changes in economic policy
and reforms of government have introduced greater
competition in the economy, many sectors and activities of
JM
the government will become more accountable and perform
better only when citizens and customers of these services play a
watchdog role and challenge abuses.
A proactive civil society is a
strength of many developed countries.
Civil society institutions in
developing countries have, on the other hand, been less active in
challenging the abuses of public power with the exception of the
consumer movement and a few dedicated groups that have resorted to
public interest litigation in response to this problem.
Public Affairs Centre (PAC) was established in Bangalore in 1994 as an
independent initiative to address this gap. Its underlying premise is that
the quality of governance will improve only when civil society is aware
and active in demanding greater accountability from government and Its
agencies. It undertakes research in order to enable and strengthen the
institutions of civil society to better understand the issues to be
addressed and the options to be presented to government. Both research
and advisory services are provided by PAC in order to support collective
action by citizen groups to improve the responsiveness of service
NATIONAL WORKSHOP ON REPORT CARD APPROACH, BANGALORE,
OCTOBER 25-27, 1999.
providers. It networks with concerned organisations in India and abroad in
furtherance of these goals.
PAC’s strength and credibility lie in its
independence and commitment to strengthening civil society institutions.
In order to demonstrate what civil society can do in this regard, PAC
I
f
initiated a project to produce a "report card on urban public services” in
Bangalore by its citizens. This entailed a random sample survey of the
users of different public services (utilities) in the city, and the
aggregation of the public feedback as a basis for rating the service
providers. This represented a novel approach that received both national
and international attention.
^1?
»
The focus on urban public services stems
from the concern about the decay of our cities and the
1 realisation
that
improved
productivity
calls
for
an
enhancement of the quality of urban living. This exercise
generated much public and media interest in Bangalore and
w provided a stimulus to several public service agencies in the
city to review and improve their performance and attitude towards
customers. Report Card studies on the public services in the cities of
Bangalore, Ahmedabad, Pune, Chennai, Calcutta, Delhi and Mumbai have
been completed. A unique feature of these studies is the ratings by
.
citizens (users) of the major public service providers in the urban areas., ,,
inter-city comparisons of public services and agency ratings have been
published both in India and abroad. Media coverage of the study findings
has been impressive. Citizen groups have taken follow-up actions in
several cities. Dialogues with service providers for reform have been an
important outcome of these exercises.
PAC has brought out a study of the municipal budgets of Bangalore. The
findings confirmed the poor state of resource management in the city and
highlighted how this has contributed to the unsatisfactory municipal
services. The results of the study were discussed in a meeting with the
NATIONAL WORKSHOP ON REPORT CARD APPROACH, BANGALORE,
OCTOBER 25-27, 1999.
Mayor and her Finance Committee. Though city corporations receive loans
and grants from several financial institutions, their fiscal and accounting
systems and practices remain outdated and in disarray. Inter-city
comparisons of budgets are presently under way.
I
The Report Card studies seek to provide a benchmark on quality of public
services as experienced by citizens.
Hence, they go beyond particular
problems at hand, and place each issue in the perspective of other
elements of service design and delivery, as well as a comparison with
other services, so that a strategised set of actions can be initiated for
sustainable change.
Report Cards seek to capture citizens' feedback in simple and
unambiguous
dissatisfaction.
terms
by
indicating
their
level
of
satisfaction
or
For example, the most basic but clear feedback that a
citizen may give about power supply in Bangalore is total dissatisfaction.
To appreciate this feedback, we must relate it to the ratings given to
other public services by the same person. For example, water supply may
be rated worse than power supply. When we look at these two pieces of
information, we can conclude that power supply may be a cause of
dissatisfaction, but the priority for corrective action may be on water
supply. Hence measures of citizens satisfaction across different public
services constitutes the core of Report Card studies.
Report Card studies do not stop with measures of satisfaction - they go on
to enquire into specific aspects of interaction between the service agency
and the citizen, and seek to Identify issues that emerge in connection
with the same. In more simple terms, it suggests that dissatisfaction has
causes, which may be related to the quality of service enjoyed by the
citizen (like reliability of power supply, or availability of free medicines
in a public hospital), the type of difficulty encountered while dealing with
NATIONAL WORKSHOP ON REPORT CARD APPROACH, BANGALORE,
OCTOBER 25-27, 1999.
the agency to solve service problems (like excess billing or complaints of
power supply breakdown), and hidden Costs in making use of the public
service (bribes for getting the fuse connected or investments in filters to
purify "drinking water"). Therefore we can see that Report card studies
go into different aspects of performance in interfacing with citizens, to
r
provide indicators of problem areas in public services.
Report card studies are not merely a means of collecting feedback on
existing situations from citizens. They are also a means for testing out
different options
that citizens wish to exercise,
collectively, to tackle current problems.
Individually or
For example, the Bangalore
study enquired into whether citizens were willing to pay more or be part
of citizens’ bodies made responsible for managing garbage clearance.
Hence, Report Cards are also means for exploring citizens' alternatives
for improvements In public services. .
An important aspect of Report Cards is the credibility they have earned.
The conclusions in a Report Card are not opinions of a few persons who
think in a particular manner, nor the complaints of a few aggrieved
citizens.
The methodology involves systematic sampling across all
subsections or segments of citizens - including those who are satisfied as
well as the aggrieved - and presents a picture that Includes all opinions.
This is possible because the methodology makes use of advanced
techniques of
market
research,
for
selecting
samples,
designing
questionnaires, conducting interviews, and interpreting results.
As a
result, the report cards are able to provide reliable and comprehensive
representation of citizens' feedback.
Impacts
The concept of citizen feedback surveys to assess the performance of
public services is quite new to India. The responses and spin-offs from
NATIONAL WORKSHOP ON REPORT CARD APPROACH, BANGALORE,
OCTOBER 25-27, 1999.
such studies carried out by Public Affairs Centre in different cities and for
different sectors have been encouraging. From these experiences, one
could perceive the Impact at four levels:
i
I. CREATING PUBLIC AWARENESS
The Report Card findings are generally publicised prominently by major
newspapers. Needless to say, agency specific findings and the novelty of
the method used were in part responsible for this response. And, of
course, news about corruption always makes good copyl A good case in
point in this regard was the dissemination of findings in
Bangalore. A leading city newspaper, the Times of
| India, started a weekly feature with a graphic
| depiction of one of the study findings at a time. This
feature continued for about two months, thus keeping
the report card phenomenon In public consciousness.
Seminars and meetings are also organised in connection with the release
of Report Card findings, involving local activists in civic affairs,
representatives of residents’ associations and NGOs interested In the
problems of the urban poor. Most participants were familiar with the poor
performance of the city's public agencies, but did not have the necessary
information to grade them or highlight specific problem areas.
Report
Cards gave this critical segment a handy tool to focus on issues of concern
and stimulated them to move from anecdotal and subjective issues to an
objective database to put pressure on public service agencies.
II. STIMULATING AGENCY REFORMS
Report Card studies clearly brought to light a wide panoply of issues, both
quantitative and qualitative that send strong signals to public service
providers. The use of a rating scale permitted the respondents to quantify
NATIONAL WORKSHOP ON REPORT CARD APPROACH, BANGALORE,
OCTOBER 25-27, 1999.
the extent of their satisfaction or dissatisfaction with the service of an
agency. The scale was used not only for an overall assessment of an
agency but also for different dimensions of its service. The inter-agency
comparisons with respect to public satisfaction and corruption that a
report card permits also created a platform to stimulate agency Interest
I
in addressing the underlying problems. Quantification and rankings
demand attention in a way that anecdotes do not. They focus attention
on specific agencies and services that can be embarrassing to those in
charge especially because of the adverse publicity involved.
Many agencies used the Report Card findings as a diagnostic tool to
trigger off further studies and internal reforms. Inter agency comparisons
also acted as ’proxy’ market indicators to stimulate competition and
better responsiveness.
III.
STRENGTHENING CIVIL SOCIETY INITIATIVES
The findings and information provided by Report Cards have largely
succeeded in catalysing citizens to take proactive and creative steps.
Rather than existing as passive recipients of inefficient and unresponsive
services, more and more of them are today organising to involve as active
partners and participants. The Swabhimana Initiative in Bangalore,
launched in the aftermath of the release of the Report Card findings is
| one such example. The Initiative, mooted by the then
H
I Commissioner of Bangalore City Corporation, is a unique
state-citizen forum to Improve the quality of civic life in
city. This forum not only experiments with new approaches to
the
solving problems, but also disseminates information widely and performs
a watchdog function.
NATIONAL WORKSHOP ON REPORT CARD APPROACH, BANGALORE,
OCTOBER 25-27, 1999.
IV.
CAPACITY BUILDING FOR PUBUC INTEREST GROUPS, ADVOCACY
NGOs AND INTERNATIONAL ORGANISATIONS,
Public Interest Groups, Advocacy NGOs and International Organisations
are increasingly perceiving the potency of Report Cards In chalking out
1
effective strategies to make public service agencies more responsive and
accountable to the citizens. Most of the Report Card studies conducted in
various cities by PAC have been in dose association with public interest
groups and NGOs. For these organisations, the methodology has provided
a strong, coherent and credible database highlighting areas of concern
that help them to strategise their options and sharpen the advocacy
skills. The Report Card on Public Services for the Urban Poor
in Mumbai was conceptualised and organised by PAC In close
collaboration with the Rationing Krutl Samiti, an umbrella
organisation of 40 NGOs and local groups. FEDCOT, a large
federation of consumer organisations, associated with PAC In carrying out
an exploratory study of the Public Distribution System in the state of
Tamil Nadu, using the Report Card methodology; the entire survey was
conducted by volunteers of FEDCOT who were trained by PAC and Gallup
MBA.
This approach is also being integrated into Public Administration Reform
projects In Vietnam and Ukraine.
NATIONAL WORKSHOP ON REPORT CARD APPROACH, BANGALORE,
OCTOBER 25-27, 1999.
EVALUATING PUBLIC SERVICES
■RP^here is a growing awareness among people in government,
I academia and advocacy groups on the potential of evaluation for
I
improved decision making and positively influencing the policy
environment ( see Appendix for a list of major approaches to programme
evaluation). Evaluations affect policy decisions through:
• Developins new information about programme or policy effectiveness
• Explaining to the key players the implications of the new information
derived through evaluation
• Building a reliable data base around which effective advocacy efforts
can be designed
The days when development was conceived mostly in terms of centrally
planned investments with a clearly defined hierarchy of decision-making
levels, are long over. Today, development is increasingly seen as a
participatory process wherein governments are expected to create
market-friendly regulatory environments, develop human resources and
focus on capacity building. A major area where this rethinking is taking
place Is the public service sector where there is a definite shift in focus
from investment led growth strategies to an emphasis on the role of
policy, information and awareness, improved decision making and active
involvement of the civil society. This means that results on the ground
and sustainability are now the acid tests of performance. Issues
pertaining to accessibility, effectiveness and responsiveness have become
critical in this context. More attention is paid to impact assessments of
the services and greater stress is laid on feedback and dissemination
activities.
NATIONAL WORKSHOP ON REPORT CARD APPROACH, BANGALORE,
OCTOBER 25-27, 1999.
Public services in India, especially in urban areas, are widely believed to
be unsatisfactory and deteriorating. This unfortunate trend apart from
having severe consequences on the overall well-being of the economy,
affects the quality of life of the ordinary citizen in an adverse way. The
very fact that most of these services are monopolistic In nature provides
I
the beneficiary with no option to 'exit', but continue to tolerate them.
This monopoly power combined with lack of corrective actions by the
government and low levels of collective action by the citizens have stifled
the agencies’ responsiveness to the public and their motivation to
Improve services even within the limits of available resources.
Though a lot of studies have been carried out on the public sector in
general and public services in particular, the majority of them address
only economic and managerial perspectives. Seldom
does one come across perceptions on the public
services from the citizens’ perspectives. Because, as
Vg users or beneficiaries at the receiving end, their assessments of
the quality, efficiency and adequacy of the services and the
problems they face in their Interactions with the public agencies can
provide significant inputs for the improvement of service delivery and
management processes.
NATIONAL WORKSHOP ON REPORT CARD APPROACH, BANGALORE,
OCTOBER 25-27, 1999.
Major Approaches to Programme Evaluation
Key Values Promoted
Key Audiences
Efficiency,
High-level
accountability,
theoretical knowledge
Typical Questions
Preferred Methods
policy
and Quantitative: Systems Analysis, Cost Are desired outcomes attainable and
Benefit Analysis etc.,
decision makers
attributable to the programme? Is this
,
? t ■
Management/practicality,
Protyam
quality control, utility
administrators
i
managers,
Programme
Mixed: Structured and unstructured Which aspects of the Programme work
and surveys, questionnaires, interviews and well and which need improvement?
observations
decision makers
Shared Understanding
the most efficient alternative ?
directors,
How effective is the programme with
:Ir ■
___________ n <
respect to the beneficiaries’ need?
Qualitative: Case Studies, interviews,
How do the different stakeholders find
•! f
Staff and Beneficiaries
observations, document revIdM/.
the programme?
Emancipation/
Programme beneficiaries,
Participatory:
In what ways are the premises, goals,
empowerment,social change
their communities, other historical
’powerless’ groups
appraisals
Social
analysis,
Criticism,
participatory or activities of the programme serving
i
.
to
maintain
power
and
inequities in the society?
NATIONAL WORKSHOP ON REPORT CARD APPROACH, BANGALORE, OCTOBER 25-27, 1999.
resotrce
Citizen Surveys: Rationale 8t Applications
tHe Hndo^
arefully designed surveys can yield an abundance of useful information
on these and a variety of other topics and issues in public service
delivery. Having accurate information about what citizens think can
enable decision-makers to take informed decisions and policy choices and to
implement service improvements that respond to citizens,
needs and
preferences.
What is a Citizen Survey?
There are several methods for discovering what people think. One of the best
ways is to ask people directly about their opinions. But personal Interviews are
expensive and time consuming, especially In large populations or among
dlfficult-to-reach groups. A more practical method Is to ask a sample, or a
representative subset, of citizens about their opinions, attitudes, perceptions,
and behaviour. A citizen survey uses a systematic, scientific method for
Electing information from them, and making
selecting a sample of citizens, col
larger population that is usually too large to observe or
generalisations about a
interview directly.
Opinion surveys are an accurate, affordable way to determine what large
groups of people think. Many public administrators conduct such surveys
regularly to identify budget priorities; to obtain feedback from citizens,
customers, or clients on services and programs; and to acquire information on a
variety of Issues, problems, and choices that confront their organizations.
NATIONAL WORKSHOP ON REPORT CARD APPROACH, BANGALORE, OCTOBER
25-27, 1999.
The wide prevalence of opinion surveys is one indicator of their popularity and
potential for informing a variety of decisions that relate to management,
accountability, and resource allocation. However, not all surveys are equally
useful. Though some meet the highest standards of scientific rigor, others are a
waste of money and efforts. The latter are those that include
* r —
poorly written or misleading questions, omit
important
questions, or have flawed sampling designs. Only if a survey
instrument is properly designed and implemented can it yield
accurate information about who thinks what and why. Improperly
designed and executed surveys can misrepresent respondents’
views, and thus can mislead and confuse decision-making.
Hence, it is Imperative that those who conduct citizen feecfoack surveys
understand and apply the guidelines that will produce scientifically valid
and reliable survey data.
How do citizen surveys help?
Information surveys can help public officials to address issues In the delivery
and maintenance of critical services. It is helpful to begin the initial task of
drafting questions by deciding whether the information desired is related
mainly to policy formulation,
implementation, or evaluation.
Though,
inevitably there would be some overlap among these stages, they would
provide a useful initial framework for thinking about information needs and the
kinds of decisions that can be Informed by survey results.
It is also useful to consider how opinion surveys can help to broaden the scope
of citizen participation in the variety of decisions that confront officials in the
public arena.
NATIONAL WORKSHOP ON REPORT CARD APPROACH, BANGALORE, OCTOBER
25-27, 1999.
w Policy Formulation
Policy formulation Involves decIdins what to do. Surveys can help public
officials to determine what people need, want, prefer, or demand from their
I
government or for their rupees spent as tax. They can then use this information
to make choices, set priorities, change practices, and translate popular
demands into public policy. Questions that help to inform policy choices often
measure the preferences, popularity, or acceptability of singular or competing
ideas, actions, choices, or services.
Policy Implementation
Citizen feedback can also help public officials decide how best to deliver or
provide services. Useful questions in this area may concern the variety of
activities and decisions Involved in Implementing a policy, programme, or
service. These can be directed to one or more of the groups with a stake in the
issue, such as elected functionaries, management team, employee unions,
resident associations and public interest groups.
»<■ Policy Evaluation
Citizen surveys can also provide useful feedback for evaluation of public
policies and programmes. In the business of service delivery, the consumer’s
perception is the pertinent reality. One could safely assume that the agency
responsible for a particular service is not doing it's job well if citizens express
dissatisfaction with various dimensions of output effectiveness, such as the
quality, timeliness, level, accuracy, reliability, convenience, utility, and price
of the service. Survey questions can ascertain what citizens think about the
quality of services, who uses services, how frequently they use them, and
where specific improvements need to be made. For instance, survey findings
might suggest the need to publicize the availability of services that are under
utilized.
NATIONAL WORKSHOP ON REPORT CARD APPROACH, BANGALORE, OCTOBER
25-27, 1999.
Surveys as participation Mechanisms
Surveys can help to broaden the scope of citizens, participation in government
decision making. Practitioners know that the citizens who feel they play a part
or have some impact upon a policy or programme are more likely to feel they
have a stake in its outcome. Citizen surveys are one means of advancing a
process of deliberative democracy, where public officials address citizens’
concerns up front rather than later, In court. As a method of practising the
politics of inclusion, surveys have the potential to enhance the quality of
democratic governance. This potential can be realized when the objectives of
the survey are dear, when citizens have enough Information to make choices
and form opinions, and when the findings are publicized and discussed in
forums of community outreach.
NATIONAL WORKSHOP ON REPORT CARD APPROACH, BANGALORE, OCTOBER
25-27, 1999.
Planning a Report Card Strategy
A
s a first step in moving towards data collection, your organization
should define an issue focus and a framework for your study. Several
jf^^^questions should be asked to help provide clarity to your research
design:
•
WHAT DO YOU WANT TO KNOW?
■
What are the issues or problems that you find the most troublesome in your
community (e.g., continued power shortages, lack of access to public bank
loans, corruption in local government agencies, etc.)?
What does your community have to say about those Issues/problems?
Can this research add value to existing studies and/or current action on that
issue area(s)?
■
-
•
■
-
ABOUT WHOM?
Do you want to focus on a single public utility or service provider?
Do you want to gather comparative information from a wide range of
utilities?
•
■
-
HOW WILL YOU USE THE INFORMATION?
What is the purpose of this study?
Will It re-shape a current program In your organization?
Will it be used to consider developing new programs?
Who will see the results of the study (e.g., government agencies, the
media, civic groups, research institutions, etc.)?
•
•
HOW CAN THE DATA BE OBTAINED?
What methods (e.g., key informant interviews, focus groups, surveys,
observations) are the most effective In gathering the kind of responses
you want?
How will you Identify the specific population to be measured (e.g., by
locality, income, gender, age, etc.)?
-
• HOW DO YOU PLAN TO COLLECT THE DATA?
-
•
What specific tests, measures and/or questionnaire items are needed
to arrive at the desired information?
Will you use random sampling and/or focus group methods?
What level of skill is required by your staff and field-workers?
NATIONAL WORKSHOP ON REPORT CARD APPROACH, BANGALORE, OCTOBER
25-27, 1999.
•
-
■
WHO WILL PAY THE BILL?
Do you have enough resources to finance a project which will take a
minimum of 12 weeks with a staff (internal or external) of about 12
people?
Have you budgeted for unexpected changes (e.g., the need to expand your
sample size to increase the reliability of results).
These questions should be discussed within your organization and cast out to a
wide net of other public interest groups, donors, your board of directors and
selected community residents. This collective brainstorming and strategizing
can provide your organization with its own set of feedback, helping you to
construct a better project and, in turn, to better serve your community. At
this point, your organization will have a conceptual approach and in many ways
will be past the most difficult point.
Your strategic plan might vary, but it should include the following points:
A. The Problem:
c. field workers
a. background
- number needed
b. importance to you and your
d. quality control supervision
community
B. Research Purpose and Objectives
e. data processing
C. Implications and Use of Findings
E. Timetable:
D. Methodology:
a. when questionnaires will be
produced
a. sample size
b. when field work will begin
who are the target
c. when raw data will be
respondents
processed
- where do they live
d. when analysis will be complete
b. questionnaire
F. Costs:
- focus group help to identify
a. pilot testing your questionnaire
issue areas
- hard Information (structured
b. staff salaries
questions)
- soft information (open-ended
c. outside consultant fees, if any
questions)
''A
06948
V' < '**'.
001
NATIONAL WORKSHOP ON REPORT CARD APPROACH, BANGALORE, OCTOBER
25-27, 1999.
A CHECK LIST ON HOW TO PLAN A SURVEY
Step 1:
Framing the purposes & Objectives of the Survey
Considerations
Is the purpose chiefly to explore, describe, or explain phenomena ?
Methods
Workshops
Brainstorming sessions
Group discussions
Step 2:
Specifying Information Needs
Considerations
Review the types of information that survey questions can measure, and use these to
classify information objectives
Is the survey a one time cross-sectional effort or part of an ongoing longitudinal study?
Methods
Focus Groups
Workshops to review drafts of information objectives and "ideal” items of information
desired
Step 3:
Identifying the Target Population
Considerations
Determine the unit of analysis
What kinds of screen or filter questions are needed to obtain information from
knowledgeable respondents?
What population attributes are important for the study?
How difficult will it be to contact the desired population
Methods
List the types of information needed from various population groups and the kinds of
analyse required to determine who thinks what and why. Check to ensure that the
intended unit of analysis corresponds with the level of information needed.
Step 4:
Selecting the Methods of Contact
Considerations
Review the merits of different types of survey approaches
Determine when survey results are needed
Balance available resources with estimated costs of the desired method of contact
Methods
Ascertain the monetary amount available for the survey project, personnel and staff
and decide whether to adjust the time frame for the study and also whether to
contract out some part of the project’s implementation.
NATIONAL WORKSHOP ON REPORT CARD APPROACH, BANGALORE, OCTOBER
25-27, 1999.
CHOOSING AN APPROACH
he approach you choose for your study should reflect the kind of
information you want to gather. That data, in turn, should be based on
the objectives you identified in your strategic plan. If you are looking
for percentages and averages, you should use quantitative research
If you prefer to focus on individual case studies, qualitative
techniques.
techniques will be required.
The Report Card methodology is rooted in quantitative research methods, but
is enhanced by qualitative findings obtained from interviews and observations.
Mixing quantitative and qualitative research methods is often thought of as
mixing oil and water. Some believe they do not blend. Others, however, see
the value in complimenting attitudes with statistics and augmenting opinions
with numbers.
In fact, many of the advantages of the Report Card’s
quantitative survey approach are derived from prior use of qualitative
methods.
Initial sessions with focus groups in Bangalore and other cities
provided valuable inputs for the final design of the survey.
How do qualitative and quantitative research methods differ and how do they
compare?
(See the comparative table on the next page)
NATIONAL WORKSHOP ON REPORT CARD APPROACH, BANGALORE, OCTOBER
25-27, 1999.
1
QUALITATIVE RESEARCH Is a method
of information gathering that conveys
feeling or insights. It is based on a
small sample, usually no greater than
30 people, and thus, is too small to
draw valid conclusions about the
opinions of an entire target
population. It mainly uses
observation and unstructured
Interviews to uncover meanings and
insights to problem and issues.
It is used to:
• Generate hypotheses
• Clarify Issues prior to undertaking
quantitative studies
• Assess citizen perceptions of a
public utility or service provider
• Examine emotional responses of
citizens to interaction with public
service agency and/or agent
QUANTITATIVE RESEARCH Is based on
statistical principles. It uses
sampling methods, questionnaires,
and computer based data processing
to answer questions of how much,
who, where and when. It tends to be
more expensive and time-consuming
than qualitative research, but
provides a certain degree of
reliability..
It is used to:
• Establish the level of citizen
satisfaction or
dissatisfaction with public service
providers
• Rank order the agencies according
to the level of public satisfaction
or dissatisfaction ratings
• Propose options and reform with
the weight of quantitative backing
The Report Card methodology combines these two methods of research with
two primary (though not exclusive) research techniques: focus groups and
questionnaires. Combining these techniques can enhance the overall validity
of your own study by 1) helping to reduce bias, 2) revealing errors in
measurement, 3) verifying and cross-checking data, and 4) Increasing response
rates by producing a better questionnaire.
Basic survey research (questionnaires) is usually accompanied by some form of
probability sampling when an entire population cannot be measured.
With
marginal resources available in both time and money, developing a small but
representative data base on a specific problem can serve the Interests of your
city or community well. This systematic statistical sampling procedure
reinforces rigor in the data collection approach. Such rigor is critical because in
NATIONAL WORKSHOP ON REPORT CARD APPROACH, BANGALORE, OCTOBER
25-27, 1999.
the world of policy, numbers are often given more importance than opinions.
The survey method can also highlight the quantitative estimates between a
problem and its possible causal factors through frequency distributions and
1
regression analysis. Once a problem’s size and dimensions are Identified (via
the findings from the survey), researchers can begin to look at which factors
are most closely related to that problem.
Some issues, however, are better addressed through open-ended, exploratory
inquiries where factors might not be easily laid out in a predetermined survey
response format. The Bangalore study of slum dwellers, for example, balanced
its large-scale survey with focus groups and individual case studies. The unique
characteristic of the slum populations required a softer approach to the Report
Card study. That is, establishing degrees of awareness, attitudes and beliefs
about the quality of public services in the Project Voice survey was as
important to the survey outcomes as determining degrees of satisfaction and
dissatisfaction.
Although the Report Card methodology Is adaptable, It must still abide by
statistical conventions. Throwing a questionnaire together quickly and asking
nearby residents or passers-by on a "first come, first questioned
basis” is not survey research, although it can be quite
functional in building good programs starts and cases <
for action in a community. This method is less rigorous ’
than a sample survey research approach. The trade-off is
energy versus rigor.
In mounting the Report Card approach itself, your organization needs to set
general parameters related to sampling units (e.g. households, bus rides,
hospital users), sample size, and the method of interviewing (by mail,
telephone, or in person). Due to the erratic nature of the very services you
may be measuring, interviewing may be the only viable data collecting method
NATIONAL WORKSHOP ON REPORT CARD APPROACH, BANGALORE, OCTOBER
25-27, 1999.
In most developing countries. It Is Important to remember that each of the
decisions has cost Implications. Sample location necessarily follows from the
intentions of one's survey, but sample size and methods often derive principally
from financial resources available and how close your organization wants to get
to the target community in conducting the project.
•
•
•
•
•
HOUSEHOLD INTERVIEWING
CONS
PROS
•
Travel
time
and
expense to locate
By being physically present, the
the respondents is high
Interviewer may convince the
• Interviewer’s presence and
person to participate.
mannerisms may bias responses
Visual materials may be used
Long questionnaires have a better • Anonymity is lost; respondents
might fear later identification
chance of completion (vs. Mailed
• Field work control and supervision
surveys)
Is difficult
Interviewer may help clarify
questions the respondent is having • Staffing capable interviewers,
especially when your study Is In
trouble understanding
distant places, is difficult.
Selection of sample can be more
precise
NATIONAL WORKSHOP ON REPORT CARD APPROACH, BANGALORE, OCTOBER
25-27, 1999.
SAMPLING
ampling is the science of selecting cases in a way that enables the
researcher to make accurate inferences about a larger population.
TWO MAJOR DECISIONS TO BE TAKEN IN SAMPLING
About the kind/ method of sample.
About the size of sample.
■
■
The above decisions are in tum governed by:
■ Information needs
• Desired level of confidence and precision
■ Available resources
IMPORTANCE OF SAMPLING/ LOGIC OF SAMPLING
Main uses of samples are in:
•
Making inferences about the population based on Information from a
•
■
sample.
Estimation
Testing of Hypotheses
Sample selection affects precision and accuracy of survey results.
MAJOR STEPS OF SAMPLING
Irrespective of the type of sampling done, this remains a constant. The 7 major
steps in sampling are:
1. Defining the population: Population means the group you want to
generalize the results of your survey to; hence, the group you would sample
from. An important distinction that needs to be made here is between the
theoretical
population
and
the
accessible
population.
Theoretical:
Population you would like to generalize to and Accessible: Population that
Is actually accessible to you.
NATIONAL WORKSHOP ON REPORT CARD APPROACH, BANGALORE, OCTOBER
25-27, 1999.
Precision in the definition of the population depends a great deal on how
precisely have we defined our research problem, which, In turn, is a result of
lack of clear transmission of the purpose or objectives of the study from the
decision-maker to the investigator.
The 3 core parts of population definition are: (1) which elements (i.e. the
units of analysis, like Individuals, households, Institutions, etc.) to Include, (2)
where, and, (3) when. The "which" question is important because from a
research point of view, each group represents a distinct population with
(
corresponding implications of any information obtained. The
J "where” and "when” represent dimensions that are designed to
define the population more precisely in terms of its extent and
time.
As such, these dimensions also define which units are to be
excluded. It is clear that the population should be defined as precisely as
possible. One useful approach is to first define the population as the ideal
one to meet the study objectives. Practical constraints then enter to define
the study population. The advantage of starting with an ideal population is
that exclusions are made explicit.
Dangers of over-defining the population: Over-defining should be avoided
unless It is completely necessary. Over-defining can limit the extent to which
findings can be generalized and operationally greatly increase the cost and
difficulty of finding population elements.
2.
Census or sample: Once the population has been defined, the
Investigator must decide whether the survey Is to be conducted among the
whole population (a census) or only a subset of it (a sample). A census may
just not be feasible,
particularly when it comes to
Developmental
Organisations. Hence, in most instances, samples are used.
NATIONAL WORKSHOP ON REPORT CARD APPROACH, BANGALORE, OCTOBER
25-27, 1999.
Two main advantages of using a sample are: speed and timeliness. Firstly, a
survey based on a sample, takes much less time to complete than one based on
a census. Secondly, in certain instances a complete count may require so much
1
time that, by the time it is completed and is made available for use, it
becomes a historical record. Another consideration in whether to use sampling
is the relative cost and effort involved.
3. Sample Design:
Operationally, sample design is the heart of sample planning. Involving both
theoretical and practical (e.g., time, cost, labor involved and organization)
considerations. Typically, questions to be answered Include:
■ Type of sample
■ Sampling unit: This Is the unit that we sample (usually people). The
sampling unit forms the basis of the actual sampling procedure The
sampling unit may consist of one or more population elements, i.e. these
units may be individual elements or aggregates of individual elements. For
example, for a Report Card survey, we generally select individual chief
wage earners or entire households as sampling units.
■ Sample frame: The physical listing of the accessible population from
which you will draw your sample is called the sampling frame, (e.g.,
telephone directory in the case of a telephonic survey). In an "on-thestreet" consumer survey, the frame may be defined as a 'listing’ of people
who might reasonably be expected to pass by the interviewer during a
specified period.
•
Refusals and Non-Response: The sample plan must include provision for how
refusals and non-response are to be handled. Of concern is
whether additional sampling units are to be chosen as
replacements and, if so, how these are to be selected.
NATIONAL WORKSHOP ON REPORT CARD APPROACH, BANGALORE, OCTOBER
25-27, 1999.
4. Sample Size: Somewhat related to sample design, but In many ways a
separate decision area for the Investigator, is the determination of the
sample size. In general, size of the sample is directly related to precision.
There are 4 general traditional approaches to this determination. The first 3
are: (1) arbitrary or Judgmental, (2) minimum cell size needed for
analysis, and (3) budget-based (particularly when a huge, cost-intensive
study is being launched). The fourth approach involves the opposite
procedure. That is, by specifying a desired precision in advance, sample
size can be arrived at.
An objection the researchers very frequently face is that "the sample size
was too small to lead to any meaningful inferences’’, but the question to be
asked is: does adding more respondents to the sample necessarily add value
to the results.
Fixed V/s Sequential Sampling: As the name implies, in fixed size sampling,
the number of Items Is decided upon tn advance In such a way as to achieve
some type of balance between sample reliability and sample cost. In general,
all observations are taken before the data are analysed.
In sequential sampling, however, the analyst goes by a decision rule that
includes not only the alternative of stopping the sampling process (and taking
appropriate action, based on the sample evidence already in hand) but also the
possibility of collecting more Information before making a terminal decision.
Observations may be taken either singly or collectively, the chief novelty being
that the data are analysed as they are assembled and sample size is not pre
determined.
In general, sequential sampling has the benefit of leading to smaller (more
manageable) sample sizes, on the average, than those associated with fixed
size samples of a given reliability. The disadvantages are that: (1) the
NATIONAL WORKSHOP ON REPORT CARD APPROACH, BANGALORE, OCTOBER
25-27, 1999.
mathematics underlying sequential sampling are more complex and timeconsuming and (2) the problem may be such that it fixed size sampling
becomes preferable to a sequential sampling.
5.
Costs of Sampling: The sample plan must take into consideration the
estimated costs of sampling. Such costs are two types: (1) Overhead Costs:
these are relatively fixed for a sampling procedure, and (2) Variable Costs:
these depend on the scope of the study. In reality, it is difficult, and perhaps
not even reasonable, to separate sampling costs from overall study costs.
Consequently, in a typical study, costs from all aspects of the study are usually
considered together. (See Annexures for a specimen of the survey cost sheet).
6.
Execution of Sampling Process: This is the last step in sample
planning. Here, in short, the sample is actually chosen. There are 2 basic
requirements for the sampling process to fulfill. These are:
The sample must be representative and adequate. A representative sample Is
a relatively small piece of the population that mirrors the various patterns and
sub-classes of the population. A sample is adequate when it is of sufficient size
to provide confidence in the stability of its characteristics.
WHY PROBABILITY SAMPLING?
A probability sampling is a method of sampling that utilizes a process that
ensures for the different units in your population, an equal probability of being
chosen. It is preferred because such a sample is most likely to be
representative.
The various types of Probability Sampling are:
NATIONAL WORKSHOP ON REPORT CARD APPROACH, BANGALORE, OCTOBER
25-27, 1999.
PS
06948
Mr
■
Simple Random: This Is the best known type of probability sampling. In
such a sample, each sample element has a known and equal probability of
selection. Here, we use a table of random numbers, a computer random
number generator, or a mechanical device to select the sample. A
mechanical device may, however, fail to mix the whole set of accessible
elements thoroughly and thus limit the randomness of
selection. Computer programs, apart from being more perfect,
<1
j
are also less expensive.
The benefit of simple random sampling is that it is easy to accomplish and
easy to explain to others. Because simple random sampling is a fair way to
select a sample, it is reasonable to generalize the results from the samples
back to the population.
However, on the negative side, it is not the most statistically efficient
method of sampling and you may, just because of the luck of the draw, not
get good representation of sub-groups in a population. To deal with these
Issues, we have to turn to other sampling methods.
•
Systematic Random Sampling: To use systematic sampling in drawing a
sample of size of say, 20, the population (say, 100 people) must be listed in
a random order. The sampling fraction would be f = 20%. In this case, the
interval size, k, is equal to 100/20 = 5. Now, select a random integer from 1
to 5. In our example, imagine that you chose 4. Now, to select the sample,
start with the 4th unit in the list and take every k’th unit (i.e., every 5th
unit). You would be sampling units 4, 9, 14, 19 and so on to 100 and you
would wind up with 20 units in your sample.
The benefits of this system are:
(1) you only have to select a single random number to start things off,
(2) it may also be more precise than simple random sampling, and,
NATIONAL WORKSHOP ON REPORT CARD APPROACH, BANGALORE, OCTOBER
25-27, 1999.
(3) in cases where the population is too large or the time available for
survey is too limited, it is more feasible to draw a systematic random
sample.
■
Stratified Random Sampling (also called Proportionate Random Sampling1
or Quota Random Sampling): This method involves dividing the population
into homogenous sub-groups and then taking a simple random sample in
each sub-group.
There are several reasons why this method is preferable to simple random
sampling. First, if one wants to be able to talk about all key sub-groups,
especially small ones, this may be the only way to effectively assure one
will be able to. Second, stratified random sampling will generally have
more statistical precision than simple random sampling. This will only be
true if the strata or groups are homogenous. If they are, we expect that the
variability-within-groups is lower than the variability for the population as a
whole. Stratified random sampling capitalizes on this fact.
In the Millenium Survey recently taken up by Public Affairs Centre, a variant
of the stratified random sampling was used. Here, while selecting the
sample, a four-stage procedure was followed. In the first stage, 6 districts
were selected from all over the state. The subsequent stages were:
II.
Block Selection
On an average, the districts have 9-10 blocks. Under the proposed study, 5
blocks will be randomly selected in each district.
III. Village Selection
In each of the selected blocks, 5 villages will be randomly selected.
IV.
Household Selection
NATIONAL WORKSHOP ON REPORT CARD APPROACH, BANGALORE, OCTOBER
25-27, 1999.
Two options were considered:
Option I
Six districts (30% of the total districts in the state) will provide the universe
for the survey. Five blocks in each district, five villages in each block and 20
households In each village will be randomly selected.
No. of
sample
districts
6
No. of sample
blocks
No. of sample
villages
No. of sample
households
30
150
3000
Option II
Recognizing the need to develop a more comprehensive database for some
(say 50%) of the sample districts, It was thought for those districts, 7 blocks
instead of 5 would be selected randomly, so that the sample structure looks
like:
Comprehensive Distt. level
study________________
State level study________
Total
No. of
sample
districts
3
No. of
sample
blocks
21
No. of
sample
villages
105
No. of
sample
households
2100
1
15
36
75
180
1500
3600
6
Option II was finally selected for the study.
>
cluster (Area) Sampling: This method was invented to overcome the
difficulties in doing a random sampling of a population spread over a large
’ When we use the same sampling fraction within strata, we are conducting proportionate stratified random
sampling. When we use dififerent sampling fractions in the strata, we call this disproportionate stratified
random sampling.
NATIONAL WORKSHOP ON REPORT CARD APPROACH, BANGALORE, OCTOBER
25-27, 1999.
area. It is used primarily for efficiency of administration and lower
Interviewing costs. However, the reliability of results in this case may not
be any better than in a simple random sampling.
The steps involved in cluster sampling are:
1. Divide population into clusters (usually along geographic boundaries)
2. Random sample the clusters
3. Measure all units within sampled clusters
■ Multi-Stage Sampling: In cluster sampling, only one level of sampling takes
place (e.g., a sampling of blocks) before the basic elements are sampled
(e.g., the households). However, if one or more successive samples within
the larger area are taken before settling on the final clusters, the resulting
design is usually referred to as a multi-stage area sample. That is, even
within sampled blocks, households are sampled and only selected
households are finally interviewed.
NON-PROBABILITY SAMPLING
Sometimes, non-probability sampling may be needed. Non-probability sampling
is different in the sense that it does not involve random selection.
The various types of non-probability sampling are:
Accidental, Haphazard or Convenience & Purposive
Accidental, Haphazard or Convenience:
This is a generic term covering a wide variety of ad hoc procedures used for
selecting respondents. It Includes the traditional man-on-the-street interviews.
Convenience sampling means that the sampling units are accessible,
convenient and easy to measure, cooperative or articulate and a relatively
large number of Interviews can be obtained quickly. There is no evidence that
such samples are representative of the populations one wants to generalize to.
NATIONAL WORKSHOP ON REPORT CARD APPROACH, BANGALORE, OCTOBER
25-27, 1999.
In relation to Report Cards, such sampling Is not used except for In those cases
where an in-depth case study is being attempted.
Purposive Sampling:
This implies "sampling with a purpose in mind". We usually would
have one or more specific pre-defined groups we are seeking. So,
when we check before interviewing someone whether he meets
the predefined criteria that we have set, we are being purposive. This method
can be useful when we need to reach a targeted group quickly. But the
downside is that we are also likely to overweigh the sub-groups in our
population that are more readily accessible. Purposive sampling was done by
PAC while conducting a study in which the target group was the property taxpayers.
■
Quota Sampling: The most commonly employed non-probability sampling
procedure. Here, the sizes of various sub-classes or strata in the population
are first estimated from some outside source, such as from Bureau of the
Census Data. For example, one may use census data to find out the
proportion of the adult population who fall into various age-by-sex-by
education classes.
In quota sampling, the Interviewer may not select the respondents
necessary to fill each quota, on a random basis. This is where it is different
from stratified random sampling.
The benefit of using quotas is that it makes it easier and cheaper for the
interviewer to select his respondents.
Quota sampling can be of two types: Proportional and Non-Proportional. In
proportional
quota
sampling,
you
want
to
represent
the
major
characteristics of the population by sampling a proportional amount of
each. The problem
here is that one has to decide the specific
characteristics on which to base the quota (such as, gender, age, race,
religion etc.). Non-proportional quota sampling is less restrictive. In this
NATIONAL WORKSHOP ON REPORT CARD APPROACH, BANGALORE, OCTOBER
25-27, 1999.
method, you specify a number of sampled units you want in each category.
Here, you are not concerned with having numbers that match the
proportions in the population. Instead, you simply want to have enough to
assure that you will be able to talk about even small groups in the
population. It is typically used to ensure that smaller groups are adequately
represented in your sample.
•
Expert / Judgment Sampling: This Involves the assembling of a sample of
persons with known or demonstrable experience and expertise in some
area.
There may be two reasons for doing expert sampling. First, because it
would be the best way to elicit the views of persons who have a specific
expertise. In this case, expert sampling is essentially a sub-case of
purposive sampling. Second reason is to provide evidence for the validity of
another sampling approach you might have chosen. You might convene an
expert panel consisting of persons with acknowledged experience and
insight into that field or topic and ask them to examine your modal
definitions and comment on their appropriateness and validity.
The advantage of doing this is that you have some acknowledged experts to
back your decisions. The disadvantage Is that even the experts can be, and
often are, wrong.
(See Annexures for more on Sampling-related Terminology)
NATIONAL WORKSHOP ON REPORT CARD APPROACH, BANGALORE, OCTOBER
25-27, 1999.
DESIGNING QUESTIONNAIRES
esigning questions that elicit accurate responses from the respondents
could turn out to be a challenging job. Clear, coherent questions with
interesting and appropriate response choices prompt accurate and
consistent responses. The major challenge is to frame questions that
are valid and reliable measures of what you want to know and to avoid things
that diminish these qualities.
A Checklist of the Basics
Which structure or format is best for a question? What kind of response choices
should be offered? What variations should be incorporated so that the
respondent wouldn’t be bored? What type of question is best for the kind of
information needed? Here are some suggestions:
•
•
•
•
•
Specify and rank order, from most to least important, the information
objectives of the survey.
Enumerate the kinds of information needed from respondents that relate to
each Information objective. Are they opinions, attitudes, beliefs, or
attributes?
Rank the Items In each topical group tn the order of their Importance to the
study.
For each Item In each group, try to answer these questions:
Why ask this? (How is it linked to the purpose of the study)
Who in the target population knows about it, and are likely to have
an opinion about it?
How will the responses to this item be coded?
What kind of statistical analysis will be performed with this
variable?
Place the most Interesting ltem(s) in the most important battery of
questions at the beginning of the questionnaire.
NATIONAL WORKSHOP ON REPORT CARD APPROACH, BANGALORE, OCTOBER
25-27, 1999.
Question Type
The two basic question types are open- ended and closed- ended. The
respondents answer open ended questions In their own words. For closed-ended
questions, the researcher offers limited response choices. The following chart
lists out the major advantages and disadvantages associated with each type.
Question Type
Applications
Allows respondents to answer in their own
words.
Useful for exploratory research
questions that need to probe people’s
preferences,
priorities,
and
positions.
Appropriate when mutually exclusive and
exhaustive response choices are difficult to
devise or when such a list increases the
complexity.__________________________
Partially closed-ended The most probable or likely choices are
presented but the list cannot be exhaustive
because there Is reason to suspect that opinion
delivery exists among a small segment of the
population.
The question type permits
respondents to offer their own answers._______
with Especially useful for determining frequency of
Close
ended
participation, intensity of feeling, or degree of
ordered choices
involvement or contact. A scale that represents
a gradation of a single concept distinguishes
this question type. This format is especially
useful for a series of attitude and belief
questions.
____________________ __
Close
ended
with Used to help establish priorities, decide on
alternative policies, or enumerate behaviours as
unordered choices
long as the choices are exhaustive and mutually
exclusive.
Open-ended
How to Avoid Bias in Questionnaire Design
A bias is said to exist whenever some feature of the survey instrument or
Interview process leads to a response that Is not a genuine measure of the
NATIONAL WORKSHOP ON REPORT CARD APPROACH, BANGALORE, OCTOBER
25-27, 1999.
respondent’s true opinion, attitude, belief, or attribute. The bias can occur In
the instructions, question wording, question order, response choices, or the
format of the instrument. Some common biases are given below:
I
Instrumentation Bias: The major sources of instrumentation bias are unclear or
vague vocabulary, poor grammar, excessively demanding questions, loaded
questions, unbalanced or overlapping response choices, and reliance on a single
question to measure complex concepts. Interviewers may also induce an bias
through voice inflection that suggests preferred responses or by
I. inconsistently
phrasing questions.
Acquiescence Response Set Bias: Sometimes, there is a tendency for people to
answer questions In a specific direction. Respondents become bored quickly
when they encounter too many questions with the same format, and they may
superficially scan for answers they think apply, to end the ordeal quickly.
Straight-Line Response Set Bias: This may occur when a long series of questions
or statements with identical answer choices appears on a page. Use of the
same "agree-disagree” scale for a long list of items Is a recipe for disaster. The
respondent may mark the first few items accurately and thereafter, finding the
process boring, speed through the rest of the items by marking the same
response for subsequent statements. Varying the arrangement, structure, and
format of questions, and selecting different types of questions, eliminates
straight line response bias.
Framing Effective Questions: The Report Card Strategy
The Report Card questionnaire follows a flow of five basic types of questions to
arrive at optimal interviewing efforts:
• LEAD IN QUESTION(S)
l
These serve as an introduction of your interview, starts the flow of responses,
and establishes rapport with the respondent.
NATIONAL WORKSHOP ON REPORT CARD APPROACH, BANGALORE, OCTOBER
25-27, 1999.
Example. Greetings! I am
from
organization. We are currently
trying to understand problems experienced with organizations which provide
important public sendees to you... Could I please talk to the heed of the
household?
I
•
QUALIFYING QUESTIONS
These are used to determine the eligibility of the respondent to provide the
desired information.
Ex. Could you please look at this card and tell me the Income range
of your household? If, for example, the respondent answers that the
monthly income is below Rs. 1,000, the Interview is terminated
because the Report Card Household studies are only Interested in
tracking sample populations at a certain income level.
•
WARM-UP QUESTIONS
These are used to focus thinking and memory.
Ex. The agencies listed on this card provide services to the public.
Which agency’s services have you used in the last 6 months? In the
Report Card survey, this type of question establishes the
respondent’s contact with a particular agency.
•
SPECIFICS
These questions extract the opinions and information sought by the study.
Ex. On an overall basis, how satisfied are you with this agency after
your experience with it? How satisfied were you with the behaviour
of the staff towards you? Did you have to pay anything extra to
people in the agency to get your work done? The Report Card survey
uses these questions to focus on 1) overall satisfaction and
dissatisfaction levels, and 2) service dimensions which contribute to
those levels.
•
DEMOGRAPHIC QUESTIONS
These describe the person and/or household who responded.
Ex. Could you please tell me your occupation? What Is your
educational background? This information is especially important
when looking at the socio-economic variables that might Influence
the quality of public services provided.
Each of the example questions listed above are closed questions. These offer
the respondent a choice of answers. They may be simple yes/no questions or
multiple choice. With the latter, be cautious about the choices you provide;
piloting your questionnaire will hopefully reveal unexpected responses. Closed
NATIONAL WORKSHOP ON REPORT CARD APPROACH, BANGALORE, OCTOBER
25-27, 1999.
-
PILOTING, CODING & ANALYSIS
Piloting
Pretesting the questionnaire in the form of a ' Pilot Survey’ is a critical quality
control strategy. Piloting facilitates the researcher to identify and correct
problems with question wording, questionnaire structure, or administration.
Remember I the time spent on planning and pretesting your questionnaire
has a direct effect on the quality of the final results.
What should you keep In mind when doing a pretest ? Three criteria are usually
indicated :
• How easily the respondent can understand the questions as worded
• Whether the respondent can understand the question consistently
. Whether the respondent answer the question accurately with the response
choices provided
—
Problems arise when interviewers do not read each question as worded,
respondents regularly ask for clarification of questions’ meanings or give
inadequate or inappropriate answers. A simple "problem-no problem rating
can be used to evaluate questions. If problems occur for given questions in
more than 15% of the pilot interviews, it can be safely assumed that the
questions are highly likely to produce distorted data or distinctively susceptible
to interviewer effects. Interviewer debriefings and the recorded or observed
difficulties should Indicate which questions need to be revised, relocated, or
deleted.
Coding
The object of coding is to give each answer a number which can than be
processed by computer. Many quantitative questions can be pre-coded and can
be completed by the field worker at the time of the interview. Open-ended
responses must be analyzed and each response given a number.
It Is also
NATIONAL WORKSHOP ON REPORT CARD APPROACH, BANGALORE, OCTOBER
25-27, 1999.
recommend that one person completes the coding task to maintain consistency
in response interpretation.
FIELD WORK
f
Regardless of how finely tuned your questionnaire becomes, it is the actual
collection of the data that largely determines the validity of your study. Strict
guidelines for data collection must be outlined and clearly articulated to each
of your field staff. Group training and/or manuals are a good way to reinforce
uniformity between field workers. One supervisor should oversee this stage of
the Report Card study. This person should enforce rigid adherence to the data
collection process through close supervision. He or she should also perform
periodic quality checks to ensure the reliability of the data submitted by each
of the field workers.
This can be done by randomly selecting completed
surveys and doing follow-up interviews with the respondents, either over the
phone or in person, to confirm their original answers.
The importance of conducting a truly random sample cannot be understated.
All of your work rests on the assumption that those interviewed are a random,
and thus, representative sample of your city’s entire population. To ensure
randomness, your household selection process should be well thought out and
grilled into the routines of each of the field workers. Once you have selected
the localities you wish to survey, establish a starting point (e.g., the post
office) and then follow a consistent pattern. For example, the interviewer will
stop at every third house.
If a complete questionnaire cannot be fully
administered, he or she will continue to the very next house and resume the
original pattern. Do not forget to include directional Instructions (e.g., turn
left at every second junction).
NATIONAL WORKSHOP ON REPORT CARD APPROACH, BANGALORE, OCTOBER
25-27, 1999.
MAKING SENSE OF NUMBERS
In many ways, the analysis process Is the easiest stage of your survey.
I
Yet, even in its relative simplicity, thorough analysis requires a
t* degree of expertise. It is best if the person assigned to this works
\ independently to maintain continuity.
Data can be analyzed using several techniques.
These Include simple
techniques of averages, data ranges, frequency and mid-potnt, as well as more
technical analytical tools. Be certain to use only the techniques that match
your objectives. Several computer programs which deal specifically with
statistical analysis, such as Statistical Package for the Social Sciences (SPSS)
and Statistical Analysis System (SAS) are widely available. Before you invest in
one, determine the complexity of the results you expect, and then decide If
the investment in a software package is necessary. With basic spreadsheet
programs like Microsoft Excel and Lotus 1-2-3, you can easily generate basic
linear regression models. The results from these simple tools of analysis often
provide adequate statistical insight into your issue or problem.
Processed data will be presented as a series of tables. These will show the
number of respondents who gave a particular answer to a particular question.
Tables will be provided for the sample as a whole and for sub-samples (e.g., by
age, occupation, or gender, etc.). It is important to generate usable data. It
is easy to ask for too many tables, but the majority of tables are often left
unused. In times of questioning or doubt, consider your original objectives.
The Interpretation process, in contrast, can be undertaken by any number of
people who possess a good understanding of the problem. In fact, the inclusion
of multiple perspectives at this stage can greatly enrich the overall impact of
your Report Card. There are some basic points you should keep in mind when
interpreting your findings:
NATIONAL WORKSHOP ON REPORT CARD APPROACH, BANGALORE, OCTOBER
25-27, 1999.
1
■
iHB
:< J
\ ; i
iiS
jjggg
i -iSfal
The interpretation stage is also significant because it is when the Report Card
becomes tangible. At this point, a report or extensive summary of the findings
should be written. Determine who you want your audience(s) to be beforehand
and prepare a Report Card suitable to its comprehension level:
In-depth
statistical analysis of the findings may be better suited for academics or
government agencies than for local citizen groups. Your report should be well
organized, readable and supported by your data.
NATIONAL WORKSHOP ON REPORT CARD APPROACH, BANGALORE, OCTOBER
25-27, 1999.
DISSEMINATION OF FINDINGS
utilization of the research effort is of paramount importance since
g
|
it helps us to understand the processes better and thereby enhance the
quality of life. Utilization occurs when the research Is of high quality,
the findings lead to practical interventions, the findings conforms with
the users expectations and whether and how much the findings challensed
current practices. Some other factors which enhance the utilization are the
client’s information needs, decision characteristics, the political climate, and
the availability of competing Information.
The research findings could be categorised as follows :
• Knowledge driven or conducted to gain knowledge
• Problem solving or providing evidence to help solve policy problems
• Interactive or combining with experience to provide solutions to problems
• Political or supporting predetermined positions or advocating
•
Tactical or supporting immediate needs
• Enlightenment or helping to make better sense of the environment
Communicating the Research Findings
A carefully planned and well-executed survey is a fruitless exercise unless the
final report clearly communicates what was done, how it was done and what
was found. The final package has to explain these factors so that the citizens
can Judge the accuracy of the research and the utility of the findings. A good
and effective way to disseminate the findings are media releases. These
releases help to minimise reportorial misinterpretations and helps in
facilitating a broad public discourse about what citizens think and what
administrators should do in response. Since the audience for the official report
and press releases are different, the design and content of these vary.
NATIONAL WORKSHOP ON REPORT CARD APPROACH, BANGALORE, OCTOBER
25-27, 1999.
The Survey Report
An effective report is well organized, clearly written, and concise. It should
I
include the following:
•
The Executive Summary: The executive summary is the leadoff for the
report. It highlights in one or two pages the contents of the report and
presents a highly condensed version of the project’s purpose, methodology,
and major findings. It should also contain a list of recommended actions
suggested by the findings.
•
Survey Objectives: This section should relate the reasons for doing the
survey, what information was needed and why, and how this information
was expected to inform specific types of deliberations, decisions, or actions
by various actors.
•
Methodology: This section should explain how the information was
gathered, when it was collected, the response rate, and what the
confidence level and margin of error mean in the context of the response
rate to the sample. The objective here is to present a thorough, accurate,
and honest description of what he or she did and how. Rememberl The
quality of the research effort is distinguished by the methods employed.
•
Major Findings: This section should summarize the results and review them
in order of their importance or interest to the audience. Tables should be
used to summarise the main findings, and the most interesting results
should be highlighted with appropriate graphic illustrations.
•
Implications of the Findings: This section should answer the "So what?”
question and discuss the deductions that are possible from the findings that
relate to the objectives of the survey. The findings should have implications
concerning what is being done right, what is not, and how particular
changes may improve, enhance, or otherwise affect the service or policy.
Whatever is gleaned from the results, the Inferences must be based on the
evidence obtained, tempered by an understanding of the limits of survey
research.
NATIONAL WORKSHOP ON REPORT CARD APPROACH, BANGALORE, OCTOBER
25-27, 1999.
Media Releases
Media releases are effective conduits to disseminate accurate information
>t
about the survey and its findings. The executive summary of the final report
should provide a good starting point to draft a release. To enable a non
technical audience to assess the accuracy of the findings, the release should
stress the following points
• Who was surveyed, when they were contacted, the method of
contact, and the size of the sample
• The response rate
• What were the major issues that were explored
• Analysis of the responses
• Identifying problem areas
• Suggest areas for improvements
NATIONAL WORKSHOP ON REPORT CARD APPROACH, BANGALORE, OCTOBER
25-27, 1999.
FROM THEORY TO ACTION: SOME CASES
l
he Report Card methodology is more than just a quantitative snapshot
of citizen satisfaction levels. With a vital plan of action, it can take on
shape and substance.
At this point, it is helpful to revisit the questions you and your advisory group
discussed from the Planning and Strategizing section. Were you able to obtain
the information you expected, and did this information come from the people
you expected it to come from? If so, what sort of operational changes did you
anticipate making then? Is your group or organization prepared and willing to
invest in new programs or to augment old ones in light of your new Report Card
findings? These questions should again be cast out to a wide net of interested
parties and advisors.
This collaborative brainstorming can examine a wide
range of possible impacts of the Report Card and, thus, help to stimulate
Innovative approaches to achieving them.
There are several valuable points to keep in mind when you are drawing up
your Report Card plan of action:
NATIONAL WORKSHOP ON REPORT CARD APPROACH, BANGALORE, OCTOBER
25-27, 1999.
Do not assume instant impact; be
ready for both positive and negative
responses:
• Be clear on your objectives and
In Bangalore, the lowest rated BDA
strategies;
multiple options are
took
over a year and a half to actively
available:
J,
respond
to the findings of the Report
- Disseminating information
- Card.- Subsequently, they have
- Mobilizing the media
■ '
requested consultation from PAC to
Education citizens about their civic
curb
corruption and Increase public
rights and responsibilities
accountability.
Catalyzing the government to respond
In New York City, the Straphangers
Developing pro-active programs and
Campaign
undertook bi-yearty Report
projects
Cards
to
measure
subway reliability.
- Creating alliances among citizen
Yet,
even
with
the reoccurring
groups, NGOs, the business
assessment mechanism, they noted an
community, willing government
overall decline in service delivery
agencies
from one survey to the next. This
suggested a disinterest In the quality
• Be Innovative and creative In your
of service by the Transit Authority.
responses
Subsequent action by the Authority,
however, positively lead to an
increase in the capital budget.
•
Know what you want to achieve
•
The introduction of the Report Card methodology Itself Is an Innovative step in
furthering the cause of improved local governance. Done properly, the Report
Card can provide a systematic measure of citizen feedback with regards to
their experiences and satisfaction levels of urban services. This quantitative
tool can be a powerful new instrument of collective public voice at all levels of
government and all stages of democracies.
Some examples of Report Card Studies initiated by RAC are illustrated below:
A Report Cordon Hospital Services for the Poor in Bangalore
" It is 5.00 a.m. The emergency ward of the government hospital is deserted.
The patient is brought in a critical condition. However, there is nobody in
sight to provide any assistance. After much labour, a doctor is ushered in by a
reluctant nurse and a ward assistant. There is no sense of urgency in the
trainee doctors’s movements. The patient’s condition deteriorates. Medicines
are prescribed, but the pharmacy is shut. The patient, a victim of apathy,
finally succumbs to her illness”.
NATIONAL WORKSHOP ON REPORT CARD APPROACH, BANGALORE, OCTOBER
25-27, 1999.
This is not a chilling adaptation from a medical whodunit. This is a real life
incident narrated by the victim’s daughter who had to undergo the harrowing
experience at a public hospital In Bangalore.
While medical services have undergone many technological advances, the
deteriorating quality of health care provided by public hospitals poses alarming
questions. The issue becomes more stark when it comes to the question of the
service to the poor; lack of awareness, illiteracy and Inability to pay open
market rates for health services further compounds the problem.
How do citizens, especially the poor, experience these services ? What are
their perceptions on the health care system ? Where are the improvements
required ? To examine these Issues, the Public Affairs Centre, MBA and the
Citizens Action Group (CAG), an NGO based at Bangalore, designed and carried
out a study to assess the nature and extent of problems encountered by the
poor in their Interactions with the health care system in Bangalore.
The Report Card on health care services was based on a sample survey of user
perceptions and ratings from four categories of hospitals : large public
hospltab, City Corporation run Maternity hospitab, Mission and Charity
hospltab and Private hospltab. The study also Involved a series of Interviews
with hospital administrators to identify systems and processes that explain
some of the perceptions articulated by the respondents. As a run-up to the
larger exercise, short case studies were generated, focussing on themes like
access, availability and quality of services, cost of services, reliability of
services including diagnostic services, timely availability of drugs, patient
information systems and extent of corruption.
The field study was designed using inputs from these case studies. It covered a
sample of 361 citizens drawn from 12, 896 economically weak households
spread over sity-five locations. The in-patient sample covered 108 users of
Government hospitals, 46 users of Corporation hospltab, 63 users of Mission
and Charity hospitals and 63 users of Private hospltab. Eighty one out-patients
were covered for the study; this includes 47 users of Government hospitals and
34 users of Mission and Charity hospitals. The survey explored areas such as :
•
Usage profile of different types of health care services
•
Quality of medical care and facilities
•
Cost of services
•
Behaviour of doctors and hospital staff
•
Dynamics of 'speed money’
NATIONAL WORKSHOP ON REPORT CARD APPROACH, BANGALORE, OCTOBER
25-27, 1999.
• Overall satisfaction with the service
The study highlighted certain Issues that call for immediate responses :
I
• The most significant issue seem to be the quality of medical care offered to
the poor in the government hospitals. To start with, the waiting time prior
to receiving medical attention and treatment seem to be quite high.
Second, injections and medicines, presumably free or subsidised, are not
available at government hospitals. However, the most critical problem
seems to be the total absence of standards or the lack of awareness among
the patients about what they can expect at hospitals. Viewed from these
perspectives, Mission and Private hospitals seem to be serving the poor in a
much better manner.
• The manner in which activities are managed In the Government hospitals
calls for immediate attention. A good illustrative example would be the
extent of cleanliness maintained. Although there was no major difference In
the frequency with which cleaning was carried out across, different types of
hospitals, the level of cleanliness at government hospitals was rated the
worst in comparative terms. Similarly, the poor are made to run from pillar
to post to find medicines (in many instances, even for obtaining life saving
drugsl) when being treated in government hospitals.
• A compulsive strategy that the poor resort to In seeking better quality of
services is to make speed money payments. This approach is applied for all
levels of medical personnel (from doctors to sweepers), and for all types of
services ( from operations to entering wards without authorisation).
Although there is a wide variation in the quantum of payments,the problem
seems to be most acute in Corporation hospitals, which are used by the poor
for maternity cases.
•
Finally, it is observed that patients who have been in-patients at
government hospitals are less likely to follow up their treatment with
subsequent visits. While part of this tendency can be expalined in terms of
the general attitude to avoid hospitals as far as possible, the poor quality of
instructions that patients receive is also an important factor.
The study generated wide interest among the media; Times of India, Bangalore
carried a series of reports and news-stories based on the findings.
Making Them Heard : A Report Card on Public Services in Ahmedabad by
SEWA members.
SEWA or the Self Employed Womens Association is an NGO committed to
improve the quality of life of women in both urban and rural areas. The SEWA
NATIONAL WORKSHOP ON REPORT CARD APPROACH, BANGALORE, OCTOBER
25-27, 1999.
movement which began in Ahmedabad in the early 70*s has today evolved into
a national forum with a membership of over a million and has a strong presence
in most urban centres.
V, Most of the SEWA members in Ahmedabad reside in low income
K
settlements or slums. The availability and quality of public
\
services in these areas leave much to be desired, leading to a
Wk
1nc,dence of diseases. Repeated efforts by SEWA to bring
> (lIlO t'ie P^ght of the residents to the notice of the officials failed
JL lyliN to produce the desired results. In this context, it was felt that
( j l\ the representation could be made stronger by conducting an
objective study which could highlight the problems more starkly.
It was here that the concept of Report Cards came in much handy. The mantle
for conducting the exercise fell upon another NGO based in Ahmedabad ■ the
Foundation for Public Interest (FPI). FPI got In touch with PAC to help implant
the Report Card Methodology to obtain systematic and validated feedback on
various dimensions of public service delivery in Ahmedabad. Members from FPI
were provided training by PAC and MBA ( who also supervised the conduct of
the survey). Around 1200 SEWA members, spread over 12 slum areas, were
selected for the survey. Salient highlights of the exercise are briefly stated
below:
•
•
Food and Civil Supplies ( 95%), Corporation Sewage (91%) and Water Supply
(88%) were the most used agencies; Corporation and Collectorate ( 13%),
Police (20%) and Health and Family Welfare (84%) were the least used.
Post & Telegraph (86%), Electricity Board (84%) and Health and Family
Welfare (84%) were rated as the most satisfactory services; the dubious
distinction of being the least satisfactory service went to Toilets (25%),
followed by Water Supply (33%) and Corporation Collectorate (36%).
•
Only six percent of the respondents had contacted an agency. Fifty one
percent expressed happiness with the behaviour of the staff, while 73 %
expressed unhappiness with the time taken to solve their problems.
•
One in three respondents who interacted with an agency had to pay speed
money. In 72% of the cases, the money was asked for by the official. The
average amount paid was Rs. 289.
But what gave the exercise more meaning was certain 'spin-offs’ that emerged
as a result of the survey:
•
A unique forum called "our city” table was set up at the SEWA office where
the members can get all information required for interacting with any
public service agency
NATIONAL WORKSHOP ON REPORT CARD APPROACH, BANGALORE, OCTOBER
25-27, 1999.
•
•
The findings of the study was conveyed to the Ahmedabad Municipal
Corporation; the Corporation has responded by starting a project called
Parivartan to improve all the services provided by it in the areas covered by
the study. The findings were also shared with the corporators of the study
areas.
Two training programmes, structured around the major findings from the
study, were conducted for the leaders among the SEWA members in each of
the 12 areas to help them monitor quality of services In their area. Further
plans are underway to conduct 'Report Card studies’ on an annual basis to
monitor progress.
NATIONAL WORKSHOP ON REPORT CARD APPROACH, BANGALORE, OCTOBER
25-27, 1999.
ANNEXUREI
SOME IMPORTANT TERMS
■
Response: This is a specific measurement value that a sampling unit
supplies.
■
Statistic: When we look across the responses that we get for our entire
sample, we use a statistic, such as a mean, a median or a mode. A statistic
represents a specific characteristic of the sample.
■
Parameter: When we measure the entire population and calculate a value
like a mean or average, we don’t refer to this as a statistic; we call it a
parameter of the population.
■
Random-Digit Dialing: In those instances where a physical listing of the
accessible population is not available, the frame is a procedure that
produces a result equivalent to a physical listing. For example, take
Random-digit Dialing, which is a variant of sample frame. Here the frame is
not a set of people but a way of selecting it.
■
Sub-sample: The group that actually completes your study as against the
group you had initially identified as your sample.
■
Over-sampling: If the sub-group is extremely small, one can use different
sampling fractions (f) within the different strata to randomly over-sample
the small group (although this will add one complication; one will have to
weight the within-group estimates using the same sampling fraction
whenever one wants overall population estimates).
•
Sampling Distribution: The sample we take for study is just one of the
infinite number of samples that we could have taken. The distribution of an
infinite number of samples of the same size as the sample in our study is
known as the sampling distribution.
■
Standard Deviation: A standard deviation is the spread of the scores around
the average in a single sample.
NATIONAL WORKSHOP ON REPORT CARD APPROACH, BANGALORE, OCTOBER
25-27, 1999.
■
Sampling Error: The greater your sample size, the closer your sample is to
the actual population itself. Hence, the sampling error is smaller. A low
sampling error means that we have relatively less variability or range in the
sampling distribution.
1
■
Sub-categories of Purposive Sampling:
1. Modal Instance Sampling: In statistics, the mode is the most frequently
occurring value in a distribution The modal instance sampling involves sampling
the most frequent or the typical case. In a lot of informal public opinion polls,
for instance, a typical voter is interviewed.
But there are a number of problems with this sampling approach. Firstly, how
do we know what the 'typical' or 'modal' case is? We could say that the
modal voter is a person who is of average age, educational level, and income in
the population. But It is not clear that using the averages of these is the fairest
(considering the skewed distribution of Income etc.). Secondly, how do you
know that a particular variable or a set of variables (age, education, income
etc.) is the only or even the most relevant of all for classifying the typical
respondent? What if religion or ethnicity is an important discriminator? Clearly,
modal instance sampling is only sensible for informal sampling contexts.
2,
Heterogeneity Sampling: We sample for heterogeneity when we Imagine
that there is a universe of all possible ideas (held by a set of people) relevant
to some topic, we want to sample this population (not of people but of ideas)
and that we want to include all opinions or views,
not necessarily
proportionately. Another term for this is 'sampling for diversity'.
This happens particularly in brainstorming processes where our primary interest
is in getting a broad spectrum of ideas, not in identifying the 'average’ or
'modal instance’ ones. In essence, what we would like to sample is not people,
but ideas.
NATIONAL WORKSHOP ON REPORT CARD APPROACH, BANGALORE, OCTOBER
25-27, 1999.
Clearly, In order to get all of the ideas, and especially the unusual ones, we
have to include a broad and diverse range of participants. Heterogeneity
sampling is, in this sense, almost the opposite of modal instance sampling.
3. Snowball/ Multiplicity Sampling: Here, begin by randomly identifying .
someone who meets the criteria for inclusion in the study. We then ask them to
recommend others known to them who also meet the criteria (these are the
primary referrals). Now referrals are obtained from these referrals, and so on,
thus leading to the term 'snowballing*.
Although this method would hardly lead to representative samples, there are
times when it may be the best method available. This Is especially useful when
you are trying to estimate various characteristics that are rare in the total
population / to reach populations that are inaccessible or hard to find (rare).
Even though some probability-based procedure may be used to select the initial
group of respondents, the overall sample is a non-probability sample. For
example, referrals wilt tend to exhibit demographic profiles that are more
similar to those of the persons referring them than to what would be expected
by chance.
NATIONAL WORKSHOP ON REPORT CARD APPROACH, BANGALORE, OCTOBER
25-27, 1999.
Position: 4700 (1 views)