6870.pdf
Media
- extracted text
-
i
■I
|:
Marvin Adelson
(pp. 433-462)
Samuel Aroni
(pp. 433-462)
Klaus Brockhoff
(pp.291-321)
J. Douglas Carroll
(pp. 402-431)
Norman C. Dalkey
(pp.236-261,327-337, 387-401)
Lawrence H. Day
(pp. 168-194)
Alex Ducanis
(pp.288-290)
Selwyn Enzer
(pp. 195-209)
Nancy H. Goldstein
(pp. 210-226)
Olaf Helmer
(pp. 124-159)
Robert Johansen
(pp.517-534, 550-562)
Chester G. Jones
(pp. 160-167)
Julius Kane
(pp. 369-382)
(pp. 3-12, 15-16, 75-83,
229-235,325-326, 385-386,
489-496.573-586, 589-614)
John Ludlow
(pp. 102-123)
Richard H. Miller
(pp.517-534)
Ian I. Mitroff
(PP- 17-36)
Norman W. Mulgrave
(pp.288-290)
Charlton R. Price
(pp. 497-516)
D. Sam Scheele
M. Scheibe
(PP- 37-71)
(pp.262-287)
J. Schofer
(pp.262-287)
James A. Schuyler
(pp. 550-562)
Thomas B. Sheridan
(pp. 535-549)
M. Skutsch
(pp. 262-287)
John W. Sutherland
Murray Turoff
, 3
1
I
I
I
(pp.xix-xx)
Irene Anne Jillson
Harold A. Linstone
I
a
■
Contributors
Jacques F. Vallee
(pp.517-534)
Myron Wish
(pp.402-431)
The Delphi Method |
Techniques and Applications
Edited by
Harold A. Linstone
JE-
I
I
Portland State University
Murray Turoff
New Jersey Institute of Technology
With a Foreword by
Olaf Helmer
University of Southern California
J
1975
(pp. 463-486)
(pp. 3-12, 15-16, 17-36,
75-83,84-101,229-235,
325-326,338-368, 385-386,
489-496,563-569,589-614)
•1
I
ja
Addison-Wesley Publishing Company
Advanced Book Program
Reading, Massachusetts
London-Amsterdam-Don Mills, Ontario-Sydney-Tokyo
I
/
< 'll.
: 5 •' ■''J.'
I
Contents
Delphi — The Tholos, Courtesy of the Greek National Tourist Office
(End papers for hardbound edition only.)
■A >
I
Harold A. Linstone and Murray Turoff
Introduction.................................
General Remarks............................
Characteristics of the Delphi . .
The Evolution of Delphi....
PHILOSOPHY
II.
■I
The Delphi method.
.
A.
Harold A. Linstone and Murray Turoff
Introduction.......................................
B.
Ian A. Mitroff and Murray Turoff
Philosophical and Methodological
Foundations of Delphi......................
Introduction.......................................
Inquiring Systems (IS)......................
Lockean IS.............................................
Leibnizian IS.......................................
Kantian IS.............................................
Hegelian, or Dialectical, IS
...
Singerian—Churchmanian IS .
Concluding Remarks . .
i
i
02^00^
* 1
1977
c.
Copyright ©1975 by Addison-Wesley Publishing Company. Inc.
Published simultaneously in Canada
All riahts reserved. No part of this publication may be reproduced, stored in a retrieval system, or
transmitted in any form or by any means, electronic, mechanical, photocopying, recording, or otherw'S •
without the prior written permission of the publisher, Addison-Wesley Publishing Company, Inc.. Advanced
Book Program, Reading, Massachusetts 01867. U.S.A.
Manufactured in the United States ot America
.
3
3 ;
.
•. 10H
-4
.13
.
" I?
•‘Delphi bibliography”: p.
Includes index.
1. Delphi method. I. Linstone, Harold A.
II. Turoff, Murray.
T174.D44 — 001.4'33 — 75-25650
ISBN 0-201-04294-0
ISBN 0-201-04293-2 pbk.
ABCDEFGHIJ-MA-798765
1
INTRODUCTION
I.
I
L
xix i
i
*.
Library of Congress Cataloging in Publication Data
Main entry under title:
xiii
xiii
piv
Biographical Data
The Editors . . .
The Contributors .
Foreword by Olaf Helmer
The illustration on the cover is taken from a silver stater (about 350 B.C.) formerly
in the collection of John Pierpont Morgan. It depicts the Greek god Apollo Pythios
seated on the sacred omphalos at Delphi. Apollo, son of Zeus and Leto, became
master of Delphi upon slaying the dragon Pythos there. He was renowned not only
for his youth and perfect beauty, but even more for his ability to foresee the future.
The omphalos, located in the Temple of Apollo, was a conically shaped stone
representing the navel of the earth and the center of the universe. Thus Delphi
assumed the status of the most revered oracular site in Greece.
E
D. Sam Scheele
Reality Construction as a Product
of Delphi Interaction .
Concepts of Reality . .
Mental Climates and Styles of Thinking . .
Group Influence on Reality Construction . .
What I Meant to Say.......................................
Summary.............................................................
Some Design Considerations............................
15
17 ■
17
. 20
. 21
. . 23
. .25
. . 29
. . 33
II
.
.
.
37
40
. 48
. 56
. 64
64
Cl
v
I
('ontcnts
HI.
A.
B.l.
B.2.
B.3,
GENERAL APPLICATIONS
73
Harold A. Linstonc and Marra} Taroff
Introduction.................................
Government Planning ....
Business and Industry
....
Comments on Other Studies
75
75
76
78
Murray Turoff
The Policy Delphi............................
Introduction.......................................
The Committee and the Delphi Process
The Mechanics of a Policy Delphi
Examples of Policy Del phis
The Problems of a Policy Delphi .
Irene Anne Jillson
The National Drug-Abuse Policy Delphi:
Progress Report and Findings to Date
•
Chester G. Jones
A Delphi Evaluation of Agreement
between Organizations
.
Introduction.......................................
Background............................................
Convergence......................................
Correlation between Questions
Bias bv Time Pei iod
. 102
. 102
C.2.
i
C.3.
104
108
. 110
. 112
. 114
. 119
'j
Laboratory Bias.............................................................
Summary........................................................................
. 165
. 167
Lawrence H. Day
Delphi Research in the Corporate Environment
Introduction
Examples of Corporate Delphi Research . . .
Delphi and Bell Canada.......................................
Delphi in the Corporate Environment ....
Future Use of Delphi in Corporations ....
. 168
. 168
. 168
. 171
. 187
. 191
Sehvyn Enzer
Plastics and Competing Materials by 1985:
A Delphi Forecasting Study
Anticipated Changes in Properties of Existing
Materials..................................................
Other Materials Suggested by the Panel as Likely
to Become Important by 1985 ....
Overall Forecasts of U.S. Plastic Production
|
•a
• 1a
I < ■’
195
196
!
202
205
Haney H. Goldstein
A Delphi on the Future of the
Steel and Ferroalloy Industries
Introduction..............................................................................
Round One..............................................................................
Round Two..............................................................................
Round Three
Comparison of Delphi and Panel Studies
Advice to Future Delphi Monitors and Designers
.210
210
.212
.216
.220
.222
. 226
g
I
I
■
•
Introduction..................................................................
The Questionnaires........................................................
Results........................................................................
Preliminary Epilogue
B.4.
C.L
. 84
. 84
. 85
. 87
. 95
. 100
John Ludlow
Delphi Inquiries and Knowledge Utilization
Introduction
Outline of the Procedures: Social. Political,
and Economic Trends
Sources of Pollution
Recommended Waste-Water I reatment and
Disposal Systems
Regional Opportunities, Problems, and
Planning Strategies........................................................
Evaluation of the Methodology.................................
Summary7 of Findings and Recommendations
vii
Contents
A.
I
. 160
. 160
. 160
. 162
. 163
. 164
. 227
EVALUATION
IV.
. 124
. 124
. 127
. 134
. 158
B.
Harold A. Linstone and Murray Turoff
Introduction..................................................................
Dispersion as a Function of Remoteness of Estimate
Distribution of Responses.......................................
Optimism-Pessimism Consistency......................
Accuracy of Forecasts.............................................
Delphi Statements..................................................
Basis for Respondents’ Intuitive Forecast . . .
Self-Rating of Experts
.
.
.
.
.
.
.
Horman C. Dalkey
Toward a Theory of Group Estimation .
.
.
.
. 229
.229
. 230
.230
. 231
. 232
.233
■ ■2331
.233
/
viii
Theory’ of Errors . . .
Probabilistic Approach
Axiomatic ?\pproach .
C.
D.
.
Al. Scheibe. M. Skutsch. and J. Schafer
Experiments in Delphi Methodology
. .
Introduction.......................................................
Description of Procedure
/\n Experiment on the Effect of Feedback .
Comparison of Scaling Techniques ....
The Effect of Personal Variables on Participant
Behavior .
Opinion Stability as a Method of Consensus
Measurement
Delphi Service Program
Closure
Norman IF. Mulgrave and Alex J. Ducanis
Propensity to Change Responses in a Delphi Round
as a Function of Dogmatism
Method..................................................................................
Results
I )is< ussioii
E.
Klaus lirockhujj
The Performance of Forecasting Groups
in Computer Dialogue and Face-to-Fage Discussion
Group Performance and Communication System
Group Performance and Type of 'Task: Fact-Finding
Questions and Forecasting..................................................
The Relationship between the Hypotheses
. . . .
Participants, Group Formation. Place, and I ime
The Organization of the Delphi Groups and the
Face-to-Face Groups.............................................................
The Questions
The Volume of the Tests..................................................
The Distribution of Expertise............................................
The Significance of the Self-Ratings
The Performance of the Delphi Groups......................
The Size of the Groups.......................................................
Hypothesis D..................................................................
Hypothesis F..................................................................
A.
B.
. 262
. 262
. 263
. 267
. 272
273
C.
277
281
282
i
D.
. 291
. 297
299
. 300
. 301
. 302
. 305
. 306
. 307
. 310
. 312
. 315
. 318
. 319
323 ,
CROSS-IMPACT ANALYSIS
V.
. 249
. 251
. 257
. 288
. 288
. 289
. 290
3 .
ix
Contents
Contents
VI.
Harold A. Linstone and Murray Turoff
Introduction.................................
.
.325
Horman C. Dalkey
An Elementary Cross-Impact Model .
Introduction.............................................
Consistent Systems of Probabilities . .
Resolution of Inconsistencies . . . .
Scenarios........................................................
Conclusion..................................................
.
.
•
.
.
.
. 327
.327
-329^
.333 |
.335 ;
. 336
.
.
.
.338
. 338
.341
357
361
. 365;
Murray Turoff
An Alternative Approach to Cross Impact Analysis .
Introduction..............................................................................
Theory.........................................................................................
Example.........................................................................................
Applications..............................................................................
Annotated Bibliography.............................................................
ffli
Julius Kane
A Primer for a New Cross-Impact Language
—KSIM (with Examples Shown from
Transportation Policy).................................
Introduction
Aircraft Competition
A Metropolitan Transit Model
Conclusions.............................................................
SPECIALIZED TECHNIQUES
.
.
■
I
«
. 369
. 369
.370
.375
. 380
.
. 383
■
A.
B.
. I
■ ? '"‘S
Harold A. Linstone and Murray Turoff
Introduction.................................
.
.385
Norman C. Dalkey
A Delphi Study of Factors Affecting the
Quality of Life.............................................
Earlier Related Studies..................................
Models of the Quality of Life
Procedure ........................................................
Results.............................................................
.
.
.
.
.
.387
.387
.388
. 390
.391
X
Additional Analyses
Discussion
C.
1).
J. Douglas Carroll and Myron Wish
Multidimensional Scaling:
Models, Methods, and Relations to Delphi
Ordinary or “Two-Way”
Multidimensional Scaling
Individual Differences in Perception and the
1NDSCAL Model
Annotated Bibliography
Marvin Adelson and Samuel Aroni
Differential Images of the k uture
....
Introduction.............................................................
Purpose........................................................................
Background
Hypotheses..................................................................
Procedure
Results
Conclusions and Plans.............................................
Appendix I: Questionnaires for Images of the
Future Study
Appendix II: Some Significant Correlations
Appendix HI: Expectations II: A Survey
about the Future
E.
VII.
A.
John W. Sutherland
Architecting the Future:
A Delphi-Based
Paradigm for Normative System-Bl ilding .
Contents
Contents
396
398
B.
■
Charlton R. Price
Conferencing via Computer: Cost Effective
Communication for the Era of Forced Choice
Introduction........................................................................
What Is Computer Conferencing?
Costs and Benefits, Financial and Otherwise ....
Origins of Computer Conferencing.................................
Some Possible Futures for Computer Conferencing . .
. 402
409
414
426
. 433
. 433
. 434
. 434
. 436
. 436
. 437
. 439
Robert Johansen, Richard H. Miller,
and Jacques Vallee
Group Communication through Electronic’
Media: Fundamental Choices and Social Effects
C.
D.
. 448
Prelude..............................................................................
The Scenario-Building Logic.......................................
An Amplification of the Scenario-Building 1 echnology
Constructing the Problem Network
Generation of Action Proposals (Prescriptive Policies)
The “Learning’' .Aspect
Summary.................................
............................
COMPUTERS AND THE FUTURE OF DELPHI
. 487
Harold A. Linslone and Murray TurofJ
Introduction.................................
. 489
.
Current Research Approaches
....
Isolating Key Elements in Mediated Group
Communication
Conclusion........................................................
441
447
. 463
. 463
. 464
. 470
. 477
. 480
. 482
. 484
•
xi ;
E.
F.
Thomas B. Sheridan
Technology for Group Dialogue
and Social Choice
Introduction
The Formal Social Choice Problem ....
Potential Contributions of Electronics . . .
The Steps in a Group Feedback Session
. .
Experiments in Progress......................................
Extending the Meeting in Space and Eime
. 497
.
>
• •49<8 4
• •5O2'ifi
. .505 £
. .509 -W
.i,«
.
. .517
■
•
,
• •520 M
.
526
. .532
,
■
.
. . p35
.
. . 535
.
. .537
.
. .539
.
. . 540
.
. .544
... 548
5
.
J
-i
Robert Johansen and James A. Schuyler
Computerized Conferencing in an
Educational System: A Short-Range Scenario
Citizen Sampling on Current Events and Futures . .
Course and Curricular Evaluation
Committee Work and Long-Range Planning . . . .
Conflict Management and Diagnosis
Interface with Computer-Based Learning
The Computer Learns to Talk
Possible Misuses of ORACLE...........................................
. 550
.552
. 554
. 555
557
.558 |
.559 |
. 566
Murray Turoff
Meeting of the Council on Cybernetic Stability:
A Scenario........................
.563 I
'! r
iwl
■J/
A:
I
■>
Contents
xii
VIII.
EIGHT BASIC PITFALLS: A CHECKLIST .
.
. 571
Harold A. Linslone
Eight Basic Pitfalls: A Checklist .
Discounting the Future......................
The Prediction Urge............................
The Simplification Urge
. . . .
Illusory Expertise.................................
Sloppy Execution.................................
Optimism-Pessimism Bias . . . .
Overselling............................................
Deception............................................
. 573
. 574
. 578
. 579
. 581
. 582
. 584
. 584
. 585
DELPHI BIBLIOGRAPHY
. 587
Harold A. Liiistone and Afuiray Turn]]
Appendix: Delphi Bibliography .
Distribution of Bibliography . .
. 589
. 590
APPENDIX:
SUBJECT INDEX
i
Biographical Data
■4''
-m
THE EDITORS
Harold A. Linslone
received his B. S. degree in 1944 from the City
lege of New York and his Ph. D. in MathematicOf
from the University of Southern California in
He has been a member of the RAND Corporatio^^Jg
Senior Scientist at Hughes Aircraft Company,
Associate Director of Corporate Development
ning at Lockheed Aircraft Corporation.
to 1973, he was also Adjunct Professor of Industj^Hg
and Systems Engineering at the University of SouUi^p
ern California. There he introduced courser^nj|
“Technological Forecasting” and “Planning Altern.a-p5
live Futures.” The latter won USC’s Dart Award
Innovation in Teaching in 1970. He has presented
numerous seminars on Technological Forecasting^^
and Long-Range Planning in the United States and^
. 615
Europe and started the Center for Technological
Interdisciplinary Forecasting at Tel-Aviv University.^
Since 1970, he has been Professor of Systems Sci^|^
at Portland State University and Director of its
interdisciplinary Ph. D. Program in this field.
Senior Editor of the international journal,
cal Forecasting and Social Change.
Murray Turoff
received his B. S. degree in 1958 from the Universit^
of California at Berkeley in Physics and Mathe-Jg
matics, and his Ph. D. in Physics from BrancfeiO
University in 1965. He has been a Systems Engiriier^
for IBM, a member of the Science and Technology
Division of the Institute for Defense Analyses and^J
Senior Operations Research Analyst for the Office of
Emergency Preparedness hi the Executive Office^ of ■
the President. In 1972-73 he introduced and taught a'^
course in “Technological Forecasting” at Amr“
University. Since 1973 he has been an Ass
Professor of Computer and Information Scici
S'
I. Introduction
HAROLD A. LINSTONE and MURRAY TUROFF
General Remarks
It is common, in a book of this kind, to begin with a detailed and explidS|
definition of the subject—the Delphi technique. However, if we were to attempt g|
this, the reader would no doubt encounter at least one contribution to thia yp
collection which would violate our definition. There is in addition a philosophh^
cal perspective that when something has attained a point at which
explicitly definable, then progress has stopped; such is the view we hold
respect to Delphi.
^3
In 1969 the number of Delphi studies that had been done could be counted^
in three digits; today, in 1974, the figure may have already reached four dig^H
The technique and its application are in a period of evolution, both htaKM
respect to how it is applied and to what it is applied. It is the objective of thM
book to expose the richness of what may l?e viewed as an evolving field^
human endeavor. The reader will encounter in these pages many diff^rcg®
perspectives on the Delphi method and an exceedingly diverse range^J^
applications.
Eor a technique that can be considered to be in its infancy, it wou|a|jMg|
presumptuous of us to present Delphi in the cloak of a neatly wrapped packagg^
silting on the shelf and ready to use. Rather, we have adopted the approach^
through our selection of contributions, of exhibiting a number of diffeie^tg
objects having the Delphi label and inviting you to sculpt from these exanwjg>
your own view and assessment of the technique. For, if anything r true”
Delphi today, it is that in its design and use Delphi is more of an art than^
i
science.
However, as editors, we would be remiss if there were not some comipoft
thread underlying the articles brought together in this volume. As long as we
restrict ourselves to a very general view, it is not difficult to present,.an .
acceptable definition of the Delphi technique which can be taken as underlyg^
the contributions to this book:
Delphi may be characterized as a method (or structuring a group communicating
process so that the process is effective in allowing a group of mdividuals, ffjHM
whole, to deal with a complex problem.
To accomplish this “structured communication” there is provided;; SQme|
I
feedback of individual contributions of information and
assessment of the group judgment or view; some opportunity for mdividimWrevise views; and some degree of anonymity for the individual responses. Ayh
reader will discover, there are many different views on what are the proper
“appropriate,” “best,” and/or “useful” procedures for accomphshin^^^
various specific aspects of Delphi. We hope that the reader will find thisj^^
The- Delphi Method: Techniques and Applications, Harold A. tinstone and Murray Turof(^)
;|r'
Ini reduction
Harold A. Linsionc and Murray Turoff
4
•
used and how it was implemented. From this the reader may be able, to . d
rich menu of procedures from which he may seleci his own repast if he should
seek to employ the Delphi technique.
When viewed as communication processes, there arc few areas of human
endeavor which arc not candidates for application of Delphi. While many
people label Delphi a forecasting procedure because of its significant use in that
area, there is a surprising variety of other application areas. Among those
already developed we find:
transpose the considerations to his own area of endeavor and to evaluate the.
applicability of Delphi to his own problems.
'
Those who seek to utilize Delphi usually recognize a need to structure
group communication process in order to obtain a useful result for their ::
objective. Underlying this is a deeper question: “Is it possible, via structured
communications, to create any sort of collective human intelligence capabil? 5^
ity?” This is an issue associated with the utility of Delphi that has not aS
received the attention it deserves and the reader will only find it addressed her^.^s|_
indirectly. It will, therefore, be a subjective evaluation1 on his part to determining
if the material in this book represents a ssmall, but initial, first step in.The; >|
long-ic rin development of collective human intelligence processes.
• Gathering current and historical data not accurately known or available
v Examining the significance of historical events
Evaluating possible budget allocations
.... ................ sj«al
Exploring urban and regional planning oplions
• Planning unis ci sily (,1111 pi is and < 1111 u iihiin d< \ el opine nt
< Pulling logclhei the slim line ol a model
C'.hai .u (ci istiiM of the Delphi
• Delineating the pros and cons .issot1 i.ileil
1.lied wiili poieniial poln \ opiions
« Developing causal relationships in complex economic or social phenomena
v Distinguishing and clarifying real and perceived human motivations
« Exposing priorities of personal values, social goals
|
The Delphi process today exists in two distinct forms. The most common is the
u|c ;
papei-and-pencil version which is commonly referred to as a “Delphi Exep^
cise.” In this situation a small monitor team designs a questionnaire which,
sent to a larger respondent group. After the questionnaire is returned thq
monitor team summarizes the results and, based upon the results, develops
new questionnaire for the respondent group. The respondent group is usually^
given at least one opportunity to reevaluate its original answers based upo®|il
examination of the group response. To a degree, this form of Delphi ,is;
combination of a polling procedure and a conference procedure which attempU^
to shift a significant portion of the effort needed for individuals to communicate. |g‘
from the larger respondent group to the smaller monitor team. We shall denote'?^
It is not, however, the explicit nature of the application which determines the
appropriateness of utilizing Delphi; rather, it is the particular circumstances
surrounding the necessarily associated group communication process: “Who is
it that should communicate about the problem, what alternative mechanisms
are available for that communication, and what can we expect to obtain with
these alternatives?” When these questions arc addressed, one can then decide if
Delphi is the desirable choice. Usually, one or more of the following properties
of the application leads to the need for employing Delphi:
« The problem does not lend itself to precise analytical techniques but can
benefit from subjective judgments on a collective basis
• The individuals needed to contribute to the examination of a broad 01
complex problem have no history of adequate communication and may
represent diverse backgrounds with respect to experience or expertise
« More individuals are needed than can effectively interact in a face-to-face
exchange
• Time and cost make frequent group meetings infeasible
• 'The efficiency of face-to-face meetings can be increased by a supplemental
group communication process
• Disagreements among individuals are so severe or politically unpalatable
that the communication process must be refereed and/or anonymity assured
• The heterogeneity of the participants must be preserved to assure validity of
the results, i.e., avoidance of domination by quantity or by strength of
personality (“bandwagon effect”)
Hence, for the application papers in this book the emphasis is not on the
,
,
,.
,
.
■ •
•
f
.1
. I X -I.-I.! ...........
I
4
this form conventional Delphi.
!
A newer form, sometimes called a “Delphi Conference,’ replaces the monitcy
team to a large degree by a computer which has been programmed to carry oqt the compilation of the group results. This latter approach has the advantage of ■>
eliminating the delay caused in summarizing each round of Delphi, thereby.^
turning the process into’a real-time communications svstem. However, it does |
require that the characteristics of the communication be well defined before J
Delphi is undertaken, whereas in a paper-and-pencil Delphi exercise
monitor team can adjust these characteristics as a function of the group
responses. This latter form shall be labeled real-time Delphi.
Usually Delphi, whether it be conventional or real-time, undergoesj|cji|r^
disih^tphases. The first phase is characterized by exploration of the subject
under discussion, wherein each individual contributes additional inforn|ad<>n^
he feels is pertinent to the issue. The second phase involves the process of JlWe refer to “intelligence” in this context as including attitudes and feelings which arc part of
the oroerss of human motivation and action.
i -n *
r' ’v JR
■! S'
S' ■■ @
®
! S' jS
I ntroduclion
Harold A. Linstonc and Murray '1’uroff
6
reaching an understanding of how the group views the issue (i.e., wheie the
members agree or disagree and what they mean by relative terms such as
importance, desirability, or feasibility). If there is significant disagreement, then
that disagreement is explored in the third phase to bring out the underlying
reasons for the differences and possibly to e\ aluate them. I he last phase, a final
evaluation, occurs when all previously gathered information has been initially
analyzed and the evaluations have been fed back for consideration.
On the surface, Delphi seems like a very simple concept that can easily be
employed. Because of this, many individuals have jumped at trying the
procedure without carefully considering the problems involved in carrying; out
There are perhaps as many individuals who have had
such an exercise.
t
disappointing experiences with a Delphi as there are users who have had
successes. Some of the common reasons for the failure of a Delphi are:
* Imposing monitor views and preconceptions of a problem upon the respon
dent group by overspecifying the structure of the Delphi and not allowing for
the contribution of other perspectives related to the problem
.Assuming that Delphi can be a surrogate for all other human communicalions in a given situation
Poor techniques of sum marizing and presenting the group response and
ensuring common interpretations of the evaluation scales utilized in the
exercise
« Ignoring and not exploring disagreements, so dial discouraged dissenters
drop out and an artificial consensus is generated
• Underestimating the demanding nature of a 1Delphi and the fact that the
respondents should be recognized as consultants and piopeily compensated
for their lime if the Delphi is not an integral part of their job function
In addition to the latter problems associated with the Delphi technique
another class of criticisms directed at Delphi is often raised in the literature.
These are the “virtual” problems that do not in themselves affect the utility of
the technique.2 'Typical of these is the question of how to choose a “good”
respondent group. 'This is. in fact, a problem for the formation of any group
activity—committees, panels, study groups, etc. One has this problem no
matter what communication mode is used; therefore, while it is a real and
!
JI
problem is the honesty of the monitor team, and......................
it is of th . same concern as the
group. In fact, there is probably mG.x,
;!
honesty of any study or <analysis
t
_
likelihood in most instances of exposure of misrepresentation
misrepresentation in
in a Delphi^- Jjd
■ore
ij
I
summary than in a typical group study report. 1Finally, misunderstandings may^
'c emu
arise from differences in language
and lugiv
logic u
if participants come from diverge.: .«j
consider these virtual issues to be somewhat I
cultural backgrounds. Since we <
made no attempt to give them special
4^
irrelevant to Delphi per se, we have
1
attention within this book. Other problems will be discussed in Chapter VIII.’; ?
It is quite clear that in any one «application it is impossible to eliminate all
There is, for example,
example, a natural conflict in
problems associated with Delphi. There
the goal of allowing a wide latitude in the contribution of information and he
goal of keeping the communication process efficient. It is the task of the Delphi:
designer to minimize these problems as much as possible and to balance the
,e of
of jthg?
thg-’
various communication “goals” within the context of the objective
particular Delphi and the nature of the participants. Arriving at a balanced
1-------design for the communication structure is still very much an art, even though
there is considerable experience on how to ask and summarize various types of
6|
I
|
-I1
'13
questions.
It can be expected that the use <of Delphi will continue to grow. .M a result of
this, one can observe that a body of knowledge is developing on how to
) structure the human c<:ommunication process for particular classes of problems.
The abuse, as well as the use, of the technique is contributing to the develop-
is design methodology.
J
ment of this
the properties of normal group communication modes and
Table 1. compares
<
Delphi conventional and real-time modes. 1 he major differences lie in such
the
areas as the ability of participants to interact with the group at their own
convenience (i.e., random as opposed to coincident), the capacity to handle
and the capability to structure the communication. With respect.^
large groups,
considerations,
there is a certain degree of similarity between a . jl
to time (
i
committee and a conventional Delphi process, since delays betweeiin committee
Also, the real-time Delphi
meetings and Delphi rounds are unavoidable.
t
> a randomly occurring conference call with< l|
conceptually somewhat analogous to
a written record automatically produced. It is interesting to observe that within
the context of the normal operation of these communication modes in the
typical organization—government, academic, or industrial—the Delphi proc^s^
significant problem, it is not a problem unique to Delphi. However, the nature
of certain applications does, in fact, dictate special consideration of this
problem and it is discussed in a number of articles. Another virtual problem
frequently arises when a particular Delphi design for a particular application is
appears to provide the individual with the greatest degree of individuality
freedom from restrictions on his expressions. The items highlighted in the table , ,J
l.iken as representative of all Dvlphis. wheicupon it is then observed that this
lend io minimize the feelings and information normally communicated in such
manners as the lone of a voice, the gesture of a hand, or the look of an eye. In .r
design docs not woi k for some othei application. The problem here- is dial ol
making loo explicit and rcMiictive a definition lol Delphi. A third viilual
‘’Sec. fur vxamplc. GonU, Welty. "A (:nii<|ue o( the Delphi Tedmiq uf," Proceedings of lhe
will be discussed in more detail in many of the articles in this book.
While the written word allows for emotional content, the Delphi process doh
many instances these arc a vital and highly informative part of a communicalion process. Our categorization of group communication processes is not meant
to imply that the choice for a particular objective is limited, necessarily, to ope
co
FABLE 1
Group Communication Techniques
Conference
Telephone Call
Committee Meeting
Formal Conference
or Seminar
Conventional
Delphi
Real-Time
Delphi
Effective Group
Size
Small
Small to medium
Small to large
Small to large
Small to large
Occurrence of
Interaction by
Individual
Coincident with
group
Coincident with
group
Coincident with
Random
Random
Length of
Interaction
Short
Medium to long
1 x>ng
Short to medium
Short
Number of
Interactions
Multiple, as
required by group
Multiple, necessary
time delays
between
Single
Normal Mode
Range
Equality to
chairman control
(flexible)
Equality to
chairman control
(flexible)
Presentation
(directed)
group
I
Multiple, necessary
time delays
between
Multiple, as
required by
individual
Equality to
monitor control
(structured)
Equality to
monitor control
or t’roup control
and no monitor
structured)
z
I
E
L
I
“•
I
I
TABLE 1 (continued)
£
Group Communication Techniques
c‘
Conference
Telephone Call
Principal Costs
Other Character
istics
Committee Meeting
Communications
—Travel
—Individuals’ time
Time-urgent
considerations
Forced delays
__ Equal flow of information to and from
all
—Can maximize psychological effects
Formal Conference
or Seminar
Conventional
—Travel
—Individuals’ time
—Fees
—Monitor time
Delphi
—Clerical
—Secretarial
Forced delays
Real-Time
Delphi
—Com municat ions
—Computer usage
Time-urgent
considerations
—Equal flow of information to and from all
—Efficient flow of
—Can minimize psychological effects
information
from few to many —Can minimize time demanded of re
spondents or conferees
I
1
'4 <
■
! ’fl
‘- ’J
10
.......
Harold .A. Linstone and Murray Turoff
Introduction
communication mode. As the readers will see from some of the contributions to
this book, there are instances where it is desirable to use a mix of these
approaches.
six topics: scientific breakthroughs; population control; automation; space ! .38
progress; war prevention; weapon systems. Individual respondents were asked
to suggest future possible developments, and then the group was to estimate the Ja
The Evolution of Delphi
year by which there would be a 50 percent chance of the development
occurring. Many of the techniques utilized in that Delphi are still commoni to 1 i
the pure forecasting Delphis being done today. That study, together .with :an4 '"I
excellent related philosophical paper providing a Lockean-type justification for
the Delphi technique,5 formed the foundation in the early and mid-sixties for aJP
number of individuals to begin experimentation with Delphi in nondefeni^|i|l
The Delphi concept may be viewed as one of the spinoffs of defense research.
I roject Delphi was the name given to an Air Force-sponsored Rand
Corporation study, starling in the early 1950’s, concerning the use of expert
opinion.3 The objective of the original study was to “obtain the most reliable
consensus of opinion of a group of experts ... by a series of intensive
questionnaires interspersed with controlled opinion feedback.”
It may be a surprise to some that the subject of this first study was the
application of expert opinion to the selection, from the point of view of a
Soviet strategic planner, of an optimal U. S. industrial target system and to the
estimation of the number of A-bombs required to reduce the munitions output
by a prescribed amount.” It is interesting to note that the alternative method of
handling this problem at that time would have involved a very' extensive and
costly data-collection process and the programming and execution of computer
models of a size almost prohibitive on the computers available in the early
fifties. Even if this alternative approach had been taken, a great many
subjective estimates on Soviet intelligence and policies would still have
dominated the results of the model. Therefore, the original justifications for this
first Delphi study arc still valid for many Delphi applications today, when
accurate information is unavailable or expensive to obtain, or evaluation
models require subjective inputs to the point where they become the dominat
ing parameters. A good example of this is in the “health care” evaluation area,
which currently has a number of Delphi practitioners.
However, because of the topic of this first notable Delphi study, it took a
later effort to bring Delphi to the attention of individuals outside the defense
community. This was the “Report on a Long-Range Forecasting Study,” by T.
J. Gordon and Olaf Helmer, published as a Rand paper in 1964.4 Its aim was
to assess the direction of long-range trends, with special emphasis on science
and technology, and their probable effects on our society and our world.”
“Long-range” was defined as the span of ten to fifty years. Fhe study was done
to explore both the methodological aspects of the technique and to obtain
substantive results. I he authors found themselves in “a near-vacuum as far as
tested techniques of long-range forecasting are concerned.” The study covered
3N. Dalkey and O. Helmer, “An Experimental Application of the Delphi Method to the Use of
Experts,” Management Science 9, No. 3 (April 1963), p. 458.
Rand Paper P-2982. Most of the study was later incorporated into Helmer’s Social Technology,
Basic Books, New York, 1966.
J
areas.
At the same time that Delphi was beginning to appear in the open literature;*
*e, <•. ■.4-'SS
further interest was generated in the defense area: aerospace corporations ant
the armed services. The rapid pace of aerospace and electronics technology
and the large expenditures devoted to research and development leading t<
new systems in these areas placed a great burden on industry and defense
4
planners. Forecasts were vital to the preparation of plans as well as th?
allocation of R&D (research and development) resources, and trend extrapolajlal-!’
tions were clearly inadequate. As a result, the Delphi technique has become a
fundamental tool for those in the area of technological forecasting and is used
today in many technologically oriented corporations. Even in the area
“classical” management science and operations research there is a growiri^T^S
recognition of the need to incorporate subjective information (e.g., risk analysis) . i
directly into evaluation models dealing with the more complex problems facing £|||
■?.. I
A
Pl
1
society: environment, health, transportation, etc. Because of this, Delphi is now* :
finding application in these fields as well.
From America, Delphi has spread in the past nine years to Western Europe,14
Eastern Europe, and the Far East. With characteristic vigor the largest Delphr 1';ii|
undertaken to date is a Japanese study. Starting in a nonprofit organizatiori; (fl
Delphi has found its way into government, industry, and finally academe. This lffl
explosive rate of growth in utilization in recent years seems, on the surface,
incompatible with the limited amount of controlled experimentation or aca<
demic investigation that has taken place. It is, however, responding to
demand for improved communications among larger and/or geographically 4||
dispersed groups which cannot be satisfied by other available techniques. As
4
illustrated by the articles in this book, aside from some of the Rand studies by 4^|
Dalkey, most “evaluations” of the technique have been secondary efforts .
associated with some application which was the primary interest. It is hoped <i|
that in coming years experimental psychologists and others in related academic Wj
areas will take a more active interest in exploring the Delphi technique.
;4
While many of the early articles on Delphi are quite significant and liberally -: ^
mentioned in references throughout this book, we have chosen to concentrators
---------------.
5O. Helmer and N. Reacher, ‘‘On the Epistemology of the Inexact Sciences,” Project Rand
Report R-353, February 1960.
•
...
.. 44W
12
■I
Introduction
on work llial has taken place in (lie past live years ami whit h represents a cross
section of diverse applications.
Although the majoiily of the Delphi efforts aic still in the pure forecasting
area, that application provides only a small part of the contents of this volume.
Chapters II and III of this book consist of articles which provide an overview
of the Delphi technique, its utility, the underlying philosophy, and broad
classes of application.
Chapter IV takes up recent studies in the evaluation of the technique.
Precision and accuracy are considered in this context. Between the reviews,
• articles, and associated references, the reader should obtain a good perspective
on the state of the art with respect to experimentation.
Chapters V and VI describe some of the specialized techniques that have
evolved for asking questions and evaluating responses. Foremost among them is
cross-impact analysis (Chapter V). This concept reflects recognition of the
complexity of the systems dealt with in most Delphi activities, systems where
“everything interacts with everything.” In essence, these sections explore the
quantitative techniques available for deeper analysis of the subjective judg
ments gathered through the Delphi mechanism.
The effect computers can have on Delphi and speculations on the future of
the technique itself are discussed in Chapter VII. The book concludes with a
summary of pitfalls which can serve the practitioner as a continuing checklist
(Chapter VIII).
We have striven to avoid making this volume a palimpsest of previously
published papers: all but four of the articles have been especially prepared for
this work. The four reprinted articles were selected from the journal Technologi
cal Forecasting and Social Change, a rich lode of material on Delphi. The extensive
bibliography in the Appendix provides a guide to those who wish to probe the
subject further. It is thus our hope that this volume will serve the reader as a
useful reference work on Delphi for a number of years.
:.K Ml
II. Philosophy
■
I
I
• «jI
'i
::
I
f
-k
II.A. Introduction
HAROLD A. LINSTONE and MURRAY TUROFF
■ wB
Any human endeavor which seeks recognitioni as a professional or scientific J? |
activity
must clearly
define the—
axioms
...........
} __________
z _________
-_____upon which it rests. The foundation of a
discipline, as the foundations of a house, serves as a guide and basis for the
placement of the building blocks of knowledge gathered through research and <.
development activities. It is the definition, exposure, and investigation of the
J'
philosophical foundation that distinguishes a scientific profession from other
endeavors.
,
In a well-established scientific endeavor, the foundation is made explicit so
that one is able to recognize when the resulting structure can no longer be
properly supported and a reexamination of the fundamentals is in order. A
classic example of this was the impact of quantum mechanics on the foundavA^fj
tions of physics. With respect to new disciplines, such as the investigation
O
Delphi methodology, the situation is one where not enough of the structure has , J
been blueprinted to discriminate which of many possible foundations supply
the “best” underpinnings.
,
.■
The early attempt by Helmer and Rescher in their classic paper uOn'th^^|
Epistemology of the Inexact Sciences” proposed one foundation, largely ofi
Lockean nature, which was very adequate for the typical technological forecasting applications for which Delphi has been popular. However, in recent years
extensions to Delphi methodology have demonstrated a need for a broadet|^|
basis. Certainly the theme of this book, which largely views Delphi as the-\||
process of structuring human communications, further enhances this position.;
The first article by Mitroff and Turoff, examines what the various classic oKrtJ
“pure mode” epistemologies of Western philosophy have to offer for insight into.
ihe Delphi process. The philosophies covered are those jepresented by Lockc^H
I
t
Leibniz, Kant, Hegel, and Singer. It largely follows the morphological structure |
of philosophical inquiry first proposed by C. West Churchman in his “Design of
Inquiring Systems.” As with any young discipline, it should not come as.;a
surprise that such a rich diversity of foundation axioms may be used to give |
form and shape to Delphi. In a sense this is an expression of the yet untappe^J
potential for future development of the technique.
■- 41?
The second article, by Scheele, illustrates how a user of Delphi may compose b
for his own view and application of Delphi a very specific philosophical
foundation. The author, being primarily concerned in many of his application?
with the perceptions of individuals as they may relate to marketing problems,
adapts elements of the Lockean, Kantian, and Singerian philosophies and
merges them with the existentialist concept of subjective or negotiated reality.
The result is a foundation for a design precisely matched to the user’s unique •
needs.
r?i
■ i 1 :■
io
Harold A. Linstonu and Murray Turoff
II. B. Philosophical and Methodological
Foundations of Delphi
1 hroughout the book one will find in the various articles explicit or implicit
support for a mode or manner of applying Delphi which rests on the philosophies brought out in these two papers. It is interesting to note that a recent
sociological perspective views Delphi as a ritual.1 Primitive man always
approached the future ritualistically, with ceremonies involving utensils, litur
gies, managers, and participants. The Buckminster Fuller World Game,
Barbara Hubbard’s SYNCON, as well as Delphi, can be considered as modein
participatory rituals. I he committee-free environment and anonymity of Del-]
phi stimulate reflection and imagination, facilitating a personal futures orien-1
taiion. I bus, the modern Delphi is indeed related to its famous Greek name-1
sake.
*— ’
IAN I. MITROFF and MURRAY TUROFF
It takes two of us to create a truth, one to
utter it and one to understand it.
Kahlil Gibran
I
Introduction
The purpose of this article is to show that underlying any scientific technique,:
theory, or hypothesis there is always some philosophical basis or theory abhut;:?
about ^
the nature of the world upon which that technique, theory, or hypothesis’
fundamentally rests or depends. We also wish to show that there is more than
one fundamental basis which can underlie any technique, or in other word4s 'W
that there is no one “best” or even “unique” phiL >phical basis which j
underlies any scientific procedure or theory. Depending upon the basis which is !
presumed, there results a radically different developmental and application
history of a technique. Thus in this sense, the particular basis upon which a
scienfific procedure depends is of fundamental practical importance and not . •
just of philosophical interest.
•
<
We human beings seem to have a basic talent for disguising through1
phraseology the fundamental similarities that exist between common metjhodologics of a different name. As a result, we often bicker and quarrel about
such superficial matters as whether this or that name is appropriate for a ; S
* rl-i /at h
tVvK>
V
1
1
I
certain technique when the rool
real veoiiA
issue ic
is twhether
the philosophical
basis
system of inquiry that underlies a proposed technique or methodology is sound
and appropriate. We are indeed the prisoners of our basic images of reality?;
Not only are we generally unaware of the different philosophical images that
underlie our various technical models, but each of us has a fundamental iu
imiage
o_ I.;)
of reality that runs so deep that often we are the last to know that we hold it.
As a result we disagree with our fellow man and we experience inner conflict | 4
J
without really knowing why. What’s worse, we ensure this
this ignorance
ignorance anil
and
conflict by hiding behind catchwords and fancy names for technique; The field
of endeavor subsumed under the name of Delphi is no less remiss in this respect
than many other disciplines. Its characteristic vocabulary more often obscures
the issues than illuminates them.
ia i i
“If
•
B!
- 81
I
I
JI
'zX. Wilson and D. W ilson, “ I he hour Faces of the Future,” New York, Grove Press. 1974.
The Delphi Method: Techniques and Applications, Harold A. Li nstone and Murray Turoff (eds.)
ISBN 0-201-04294-0; 0-201-04293-2
.
Copyright © 1975 by Addison-Wesley Publishing Company, Inc., Advanced Book Program.
.i.4,' ■Jl
All rights reserved. No part of this publication may be reproduced, stored in a retrieval system,
transmitted, in any form or by any means, electronic, mechanical photocopying, recording, or
otherwise, without the prior permission of the publisher.
;. i d
sn
i6
Ian I, Miiroff and Murray I’uroff
One of the basic purposes of our discussion is to
bring these fundamental
differences and conflicts of methodology up to the surface for conscious
examination so that, one hopes, we can be in a better
better position
position to choose explicitly
the approach we wish to adopt. In order to
to accomplish this we consider a
number of fundamental historical stances that men have taken toward the
problem of establishing the “truth content” of a system of communication
signals or acts. More precisely, the purpose of this article is to examine the
variety of ways and mechanisms in which men have chosen to locate the
criteria which would supposedly “guarantee”
°nr “true and accurate understanding” of the “content” of ia communication act or acts. We will also show
that every one of these fundamental
----- .^1 iways differs sharply from the others and
that each of them has major strengths as well as major weaknesses. The moral
of this discussion will be that there is no one "'single best way” for ensuring our
understanding of the content of a set of communication acts or for ascribing
validity to a communication. The reason is that there i
- is no one mode of
ensuring understanding or ffor prescribing the validity of a communication that
possesses all of the desired characteristics that
—l one would like any preferred
mode to possess. As we wish to illustrate, this awareness itself constitutes a kind
of strength. To show that there is no one mode that can satisfy our every
satisfy our every
iclU1_rernent’
lhat lherC ‘S n° °ne Inodc lhat is besl in al1 senses and for all
circumstances, is not to say that each of these modes docs
not appear to be
“better suited” for some special set of circumstances.
Since these various modes or characteristic models for
ensuring validity
f,°,n lhC hiSl(,ry °f Wcs,cni P’Hlosophy, another objective of
this a;Jc
article is also to show
s
what philosophy and, especially, what the philosophy
of science specifically; and
"--1 concretely has to offer the field of Delphi design. For
example, one of the things we wish to show is which among these various
philosophical modes have been utilized to date (and how) and which have been
neglected. When there has been little or no utilization of a particular philosophical basis then we may infer existing gaps in the development of the Delphi
to date.
Before we describe each of thesse philosophical inodes or
systems more fully,
we can trather easily and simply convey the general spirit of each of them by
means of the following exercise. Suppose‘ we are given a set of statements or
propositions by some individual or group which pretend to describe some
alleged “truth.” Then each of our philosophical systems (hereafter referred to
as an Inquiring System, or IS) can be simply differentiated from one another in
terms of the kind of characteristic question(s)
. , that eachi would address either to
the statement itself or to the individual (group) making the statement
or
assertion. Each question in effect embodies the major philosophical criterion
that would have to be met before that Inquiring System would accept the
propositions as valid or as true.
Philosophy:
Philosophical and Methodological Foundations
'■ Oil
The Leibnizian analyst or IS would ask something like:
'll
$J
How can one independently of any empirical or personal considerations give 'a
purely rational justification of the proposed proposition or assertion? Can one build
or demonstrate a rational model which underlies the proposition or assertion?
How was the result deduced; is it precise, certain?
The Lockean analyst or IS would ask something like:
II
'•>' I
Since for me data are always prior to the development of formal theory, how can
xm
one independently of any formal model justify the assertion by means of some^fyk . .■
objective data or the consensus of some group of expert judges that bears on the
subject matter of the assertions? What arc the supporting “statistics”? What is
r*
“probability” that one is right? Are the assertions a good “estimate” of the true^,.,,
empirical state of affairs?
(
The Kantian analyst or IS would ask something like:
Since data and theory (models) always exist side by side, does there exist JwniS
combination of data or expert judgment plus underlying theoretical justification for the ,
;
data that would justify the propositions? What alternative sets of propositions exist
and which best satisfy tny objectives and offer the strongest combination of data ^ 5
plus model?
.
1 he Hegelian (Dialectical) analyst or IS would ask something like:
. g!
or planed
Since every set of propositions is a reflection of a more general
g------ 1 *'theory
----- ;---about the nature of the world as a whole system, i.e., a world-view,
"
, does there exist
alternate sharply differing world-view that would permit the serious
sideration of a completely opposite set of propositions? Why is this opposing view,^
not true or more desirable? Further, does this conflict between the plan ant
and
counterplan allow a third plan or world-view to emerge that is a <creative synthesis
the original plan and counterplan?
'I I t
r*r^ O t 4*
ckarnh;
i f fa
XV
MAava^aeA
^aa^a^ .■
-i.- < i I
a
f
Finally, the Singerian analyst or IS would ask:
.
Have we taken a broad enough perspective of the basic problem? Have we from
the very beginning asked the right question? Have we focused on the right
objectives? To what extent are the
the questions
questions and
and models
models of
of each
each inquirer
inquirer aa
|
reflection of the unique personality of each inquirer as much as they are felt to be a ' 11
“natural” characteristic or property of the “real” world?”
Even at this point in the discussion, it should be apparent that as a body
these are very different kinds of questions and that each of them is indicatiye qf
a fundamentally different way of ascribing content to a communication,
should also be apparent, and it should really go without saying, that these do .
not exhaust the universe of potential questions. There are many more philoj
sophical positions and approaches to “validity” than we could possibly hope to
deal with in this article. These positions do represent, however, some of the
.,^1
-
.'i;
lan I. Mitroff and Murray Turoff
Philosophy:
;
■
2t
Philosophical and Methodological Foundations
■'
most significant basic approaches and, in a sense, pure-modes from which
others can be constructed.
The plan of the rest of this article is briefly as follows: first, we shall describe
each inquirer in turn and in general terms, but we hope in enough detail to
give the reader more of a feel for each system; second, along with the
description of each inquirer, we shall attempt to point out the influence or lack
of influence each philosophy of inquiry has had on the Delphi technique; and
third, we shall attempt to point out some general conclusions regarding the
nature and future of the Delphi technique as a result of this analysis.
It should be borne in mind as we proceed that the question of concern is not
how we can determine or agree on the meaning of “truth” with “perfect or
complete certainty, for put in this form, the answer is clearly that we cannot
know anything with “perfect certainty.” We cannot even know with “perfect
certainty” that “we cannot know’ anything with ‘perfect certainty.’” The real
question is what can we know and, even more to the point, how’ we can justify
what we think we can know. It is on this very issue that the difference between
the various Inquiring Systems arises and the utility or value of the Delphi
technique depends.
Our objective in the following discussion will be to draw a sufficient distinct
tion between the philosophical Inquirying System (IS) concepts so that we can •
place alternative Delphi design methodologies into this perspective.
i
sI
Lockean IS
# I
»elohi
As first pioneered by Dalkey, Helmer, and Rescher at Rand, the Delphi
technique represents a prime example of Lockean inquiry. Indeed, one i
be hard pressed to find a better contemporary example of a Lockean inquirec
than the Delphi.
The philosophical mood underlying the major part of empirical science is that..
of Locke. The sense of Lockean IS can be rather quickly
auicklv and generally grasped
erasoed .y:
in terms of the following characteristics:
(1) Truth is experiential, i.e., the truth content of a system (or communication^
is associated entirely with its empirical content. A model of a system is an Lempirical model and the truth of the model is measured in terms of our ability (a)
to reduce every complex proposition down to its simple empirical referents
(i.e., ' /■ ;
ents fi.e..
simple observations) and (b) to ensure the validity of each of the simple
Apie
referents by means of the widespread, freely obtained agreement betwi
different human observers.
(2) A corollary to (1) is that the truth of the model does not rest upon
theoretical considerations, i.e., upon the prior assumption of any theory (this js :
the equivalent of Locke’s tabula rasa). Lockean inquirers are opposed to Xhcf |
prior presumption of theory, since in their view this exactly reverses ^e'
'
<
justifiable order of things. Data are that which are prior to and justify thepry//
not the other way around. The only general propositions which are act
are those which can be justified through “direct observation” or have already
been so justified previously. In sum, the data input sector is not only prior to the formal I
model or theory sector but it is separate from it as well. The whole of the Lockean IS is built I
up from the data input sector.
.
i Ji' | j;
- ar
■
i
I
■
w -^3
8
■
Inquiring Systems (IS)
The process of inquiry, whether it be for a single individual or a group of
individuals, may be “represented” by a very general system. We start with
some assumed “external event” or “raw data set” which for the moment we
consider to be a characteristic property of the “real world,” i.e., we assume the
data set “exists” in the “external world.” (As we shall see in a moment, this
amounts to assuming a Lockean IS beginning. The point is that we can’t even
begin to describe the “world” and our “knowledge” of “it” without having to
invoke some “conceptualization,” i.e., some Inquiring System characterization,
of “it.”) Next we apply some transformation and/or filter to the “raw data” in
order to get it into the “right form” for input to some model. The model, which
may be any sort of structured process, is represented by a set of rules which may
be either in the form of an algorithm or a set of heuristic principles. The model
acts on the input to transform it from the state of “input data” to the state of
“output information.” This output information may in turn be passed through
another filter or transform to put it in the right form so that a decisionmaker
can recognize and utilize it as “information” or as a “policy recommendation.”
In terms of this general configuration, the various IS can be differentiated from
one another with respect to (1) the priority assigned to the various systems
components, i.e., which components are regarded as more important or
fundamental by one IS than by another, and (2) the degree of interdependence
assigned to the various systems components by each IS.
i
1
■MS
‘.-fl
J
I
In brief, Lockean IS are the epitome of experimental, consensual systems. On any
problem, they will build an empirical, inductive representation of it. They start
from a set of elementary empirical judgments (“raw data,” observations,
sensations) and from these build up a network of ever expanding, increasingly
more general networks of factual propositions. Where in the Leibnizian IS to be
discussed shortly the networks are theoretically, deductively derived, in;
Lockean IS they are empirically, inductively derived. The guarantor of
systems has traditionally been the function of human agreement, i.e., 1 an J
empirical generalization (or communication) is judged “objective,” “true/f/tr igS
su^ll
“factual” if there is “sufficient widespread agreement” on it by a group of
“experts.” The final information content of a Lockean IS is identified almost 1 J |
exclusively with its empirical content.
1
lan I. Mitioff and Murray Turoff
A prime methodological example of Lockean thinking can be found in the
field of statistics. Although statistics is heavily Leibnizian in the sense that it
devotes a considerable proportion of its energies to the formal treatment of data
and to the the development of formal statistical models, there is a strong if not
almost pure Lockean component as well. I he pure Lockean component mani
fests itself in the attitude that although statistical methods may “transform” the
basic raw data” and “represent” “it” differently, statistical methods them
selves are presumed not to create the “basic raw data.” In this sense, the “raw
data are presumed to be prior to and independent of the formal (theoretical)
statistical treatment of the data. I he “raw data” are granted a prior existential
status. Another way to put this is to say that there is little or no match between
the theory that the observer of the raw data has actually used (and has had to
use) in order to collect his “raw data’ in the first place and the theory
(statistics) he has used to analyze it in the second place A typical Lockean
point of view is the assertion that one doesn’t need any theory in order to
collect data first, only to analyze it subsequently.
As mentioned at the beginning of this section, the Delphi, at least as it was
originally developed, is a classic example of a Lockean inquirer. Furthermore,
lla Lotkeun basis of Delphi still remains the pnme philosophical basis of the technique to
date.
As defined earlier Delphi is a procedure for structuring a communication
process among a large group of individuals. In assessing the potential devel
opment of, say, a technical area, a large group (typically in the tens or
hundreds) are asked to “vote ’ on when they think certain events will occur.
One of the major premises underlying the whole approach is the assumption
that a large number of "expert” judgments is required in order to "treat
adequately” any issue. As a result, a face-to-face exchange among the group
members would be inefficient or impossible because of the cost and time that
would be involved in bringing all the parties together. The procedure is about
as pure and perfect a Lockean procedure as one could ever hope to find
because, first, the “raw data inputs” are the opinions or judgments of the
experts; second, the validity of the resulting judgment of the entire group is typically
measured in terms of the explicit '"degree of consensus” among the experts. What
distinguishes the Delphi from an ordinary polling procedure is the feedback of
the information gathered from the group and the opportunity of the individuals
to modify or refine their judgments based upon their reaction to the collective
views of the group. Secondary characteristics are various degrees of anonymity
imposed on the individual and collective responses to avoid undesirable
psychological effects.
1 he problems associated with Delphi illustrate the problems associated with
Lockean inquiry in general. The judgments that typically survive a Delphi
procedure may not be the “best” judgments but, rather, the compromise
position. As a result, the surviving judgments may lack the significance that
extreme or conflicting positions may possess.
Philosophy:
Philosophical and Methodological Foundations
L 23kf
1 he strength of Lockean IS lies in their ability to sweep in rich1 sources of
experiential data. In general, the sources are so rich that they literally over
whelm the current analytical capabilities of most Leibnizian (analytical) sys
tems. The weaknesses, on the other hand, are those that beset all empirical
systems. While experience is undoubtedly rich, it can also be extremely fallible
and misleading. Further, the “raw data,” “facts,” or “simple observables”
the empiricist have always on deeper scientific and philosophical analysis
proved to be exceedingly complex and hence further divisible into other entities
themselves thought to be indivisible or simple, ad infinitum. More troublesome
'I
[j
.-j
T
j
still is the almost extreme and unreflective reliance on agreement as the sole or’
major principle for producing information and even truth out of raw data. The :'S|
trouble with agreement is that its costs can become too prohibitive and
agreement itself can become too imposing. It is not that agreement has nothing |
to recommend it. It is just that agreement is merely one of the many phil-.
osophical ways for producing “truth” out of experiential data. The danger
with agreement is that it may stifle conflict and debate when they are needed
most. As a result, Lockean IS are best suited for working on well-structured .^$1
problem situations for which there exists a strong consensual position on
nature of the problem situation. If these conditions or assumptions cannot be ;
met or justified by the decisionmaker—for example, if it seems too risky to base
projections of what, say, the future will be like on the judgments of expert™^®
then no matter how strong the agreement between them is, some alternate^|S
system of inquiry may be called for.
While the consensus-oriented Delphi may be appropriate to technologicalW
forecasting it may be somewhat inappropriate for such things as technology g
assessment, objective or policy formulation, strategic planning, and resource J
allocation analyses. These latter applications of Delphi often or should invqlve ■§
the necessity to explore or generate alternatives, which is very different from the ^1]
generation of consensus.
’
U-' /'I
Leibnizian IS
The philosophical mood underlying the major part of theoretical science i£ that ||
of Leibniz. The sense of Leibnizian inquiry can be rather quickly and generally
captured in terms of the following characteristics:
(1) Truth is analytic', i.e., the truth content of a system is associated entirely.
with its formal content. A model of a system is a formal model and the truth of
the model is measured in terms of its ability to offer a theoretical explanation of* yjl
a wide range of general phenomena and in terms of nur ability as model- '
builders to state clearly the formal conditions under which the model holds.
(2) A corollary to (1) is that the truth of the model does not rest upon any ■'
external considerations, i.e., upon
the raw data of the external world. Leibnizian -‘4.Tfl
*
inquirers regard empirical data as an inherently risky base upon which to j
J
lan I. Miuolf and Murray Turoff
found universal conclusions of any kind, since from a finite data set one is never
justified in inferring any general proposition. The only general propositions
which are accepted are those that can be justified through purely rational
models and/or arguments. Through a series of similar arguments, Leibnizian IS
not only regard the formal model component as separate from the data input component but
prior to it as well. Another way to put this is to say that the whole of the Leibnizian
IS is contained in the formal sector and thus it has priority over all the other components.
In short, Leibnizian IS are the epitome of formal, symbolic systems. For any
problem, they will characteristically strive to reduce it to a formal mathemati
cal or symbolic representation. They start from a set of elementary, primitive
“formal truths” and from these build up a network of ever expanding, increas
ingly more general, formal propositional truths. The guarantor of such systems
has traditionally been the precise specification of what shall count as a proof for
a derived theorem or proposition; other guarantor notions are those of internal
consistency, completeness, comprehensiveness, etc. 'The final information con
tent of Leibnizian IS is identified almost exclusively with its symbolic content.
A prime example of Leibnizian inquiry is the field of Operations Research
(OR) in the sense that the major energies of the profession have been almost
exclusively directed toward the construction and exploration of highly
sophisticated formal models. OR is a prime example of Leibnizian inquiry not
because there is no utilization of external data whatsoever in OR models but
because in the training of Operations Researchers significantly more attention
is paid to teaching students how to build sophisticated models than in leaching
(hem equally sophisticated methods of data collection and analyses. There is
the implication that the two activities are separable, i.e., that data can be
collected independently of formal methods of analysis.
Delphi by itself is not a Leibnizian inquirer and is better viewed from the
perspective of some of the alternative Inquiring Systems. However, many of the
views and assertions made with respect to the Delphi technique involve
Leibnizian arguments. Delphi has, for example, been accused of being very
“unscientific.” When assertions of this type are examined one usually finds the
underlying proposition rests on equating what is “scientific” to w'hat is
“Leibnizian.” This is a common misconception that has also affected other
endeavors in the social, or so-called soft, sciences where it is felt that the
development of a discipline into a science must follow some preordained path
leading to the situation where all the results of the discipline can be expressed
in Leibnizian “laws.” We have today in such areas as economics, sociology, etc.,
schools of research dedicated to the construction of formal models as ends in
themselves.
In Delphi we find a similar phenomenon taking place where models are
constructed for the purpose of describing the Delphi process and for determin
ing the “truth” content of a given Delphi. (See, for example, the articles in
Chapter IV.) One model hypothesizes that the truth content of a Delphi result
(often measured as the error) increases as (he size of the Delphi group increases.
Philosophy:
I
1
25 ’)■
Philosophical and Methodological Foundations
I?
I
I'his concept is often used to guide the choice of the size of the participant >
group in a Delphi. Other formal models have been proposed to measure an |
individual’s “expertise” as a function of the quantity of information supplied
and the length of associated questions. All such models, which are independerd'm,:||S
the content of what is being communicated but look for structured relationships
in the process of the communication, are attempts to ascribe Leibniziai^ J?
properties to the Delphi process. The existence of such models in certain !
circumstances do not in themselves make the Delphi technique any more or less
“scientific.” They are certainly useful in furthering our understanding of^thtj
technique and should be encouraged. However, they are based
I
J upon assumpgeneral
applicability ||
lions, such as the superiority of theory over data and the i
with respect
to the
of formal methods of reasoning, which are quite suspect
..............
&
scope of ap
r'l’r,5,t'r»n of
nf the Delphi
Fie In hi technique
technimie and the relative experimental
exnerimental
pplication
i-t
ri
it
1
'T'L
. •i•
r
It*
1
'
bases upon which most of these models currently rest. The utility of Delphi, at '■£
least in the near future, does not appear to rest upon making Delphi appear ot
be more Leibnizian but, rather, in the recognition of what all the IS modi *
can contribute to the development of the Delphi methodology. Our current;
understanding of human thought and decision processes is probably stil too
rudimentary to expect generally valid formal models of the Delphi process at
this time.
For which kinds of problem situations are LeibnFzian analyses most
appropriate? First of all, the situations must be sufficiently “well understood”
^.^,1
h
r
r»
rv/-l I
’T'hiic T
and “simple enough” r./-»
so thri*
that ♦they
can
be »-r»
modeled.
Thus, Leibnizian TS»
IS ar#»
are best !
suited for working on clearly definable (i.e., well-structured) problems for ,
which there exists an analytic formulation as well as solution. Second, tjhe e
modeler must have strong reasons for believing in the assumptions which f
underlie Leibnizian inquiry, e.g., that the model is universally and continually
applicable. In a basic sense, the fundamental guarantor of Leibnizian inquiry
is the “understanding” of the model-builder; i.e., he must have enough faith
in his understanding of the situation to believe he has “accurately” and
“faithfully” represented it.
Note that there is no sure way to prove or justify the assumptions underlying;
Leibnizian inquiry. The same is true of all the other IS. But then this is not the |
point. The point is to show the kinds of assumptions we are required to make if ’
we w ish to employ Leibnizian inquiry so that if the decisionmaker or modeler.b
unwilling to live with these assumptions he will know that another IS may 4
a
possibly be called for.
Kantian IS
'
'■'Ol
I he preceding two sections illustrate the difficulties that arise from emphasiz
of inquiry
to v/the
ing one of the components of a ltightly
’o’ \ coupled system
t
a
,'7 I
•
•
■.
■
detriment of other components. Leibnizian inquiry
emphasizes
theory
to the
•
•
’
•
i
•
..-----------detriment of data. ’Lockean inquiry
emphasizes
data
to the detriment
off theory.
When these attitudes are translated into professional practice, what often t
Philosophy:
lan I. Mitroff and Murray Turoff
-ults is the development of highly sophisticated models with little or no
neern for the difficult problems associated with the collection of data or the
•emingly endless proliferation of data with little regard for the dictates of
rrently existing models.
The recent controversy surrounding the attempts of Forrester and Meadows1
» build a “World Model’’ is a good illustration of the strong differences
tween these two points of view. In our opinion, the work of Forrester and
' tendows represents an almost pure Leibnizian approach to the modeling of
trgc, complicated systems. The Forrester and Meadows model is, in effect,
ta independent. One can criticize the model on pure Leibnizian grounds,
g., whether the internal theory and structure of the model are sound with
spect to current economic and social theory, and some of the critics have
osen to do this. However, it would seem to us that more often than not the
•nics have chosen to offer a Lockcan critique, i.c., that some other way, say
-ing accurate statistical data, is a better way to build a sound forecast model
the world. While this is a legitimate method of criticism, to a large extent it
iily further exacerbates the differences between the two approaches and hence
..isses the real point. To us the real point is not whether the Forrester'eadows approach is the correct Leibnizian approach, or whether there is a
oncct Lockean approach, but rather, whether any Leibnizian or Lockean
iproach acting independently of the other could ever possibly be “correct.”
a rosier and Meadows seek to justify (guarantee?) their approach through the
obustness and richness of their model, and their Lockean critics attempt to
iablish the validity of their approach through the priority and “regularity” of
• statistical data to which they appeal. Perhaps if the debate proves anything,
taiscs the serious question as to whether an advanced modern society can
nlinue to rely on purely Leibnizian or Lockean efforts for its planning. In
•der to evaluate the relative merits of separate Leibnizian or Lockean injuirers, it is necessary to go to a competing philosophy which incorporates
-th, such as the Kantian inquirer.
The sense of Kantian inquiry can be rather quickly grasped through the
allowing set of general characteristics:
(1) Truth is synthetic; i.e., the truth content of a system is not located in either
■ s theoretical or its empirical components, but in both. A model of a system is a
.nthetic model in the sense that the truth of the model is measured in terms of
•model’s ability (a) to associate every theoretical term of the model with
-me empirical referent and (b) to show that (how) underlying the collection of
cry empirical observation related to the phenomenon under investigation
.‘re is an associated theoretical referent.
(2) A corollary to (1) is that neither the data input sector nor the theory
ctor have priority over one another. Theories or general propositions are built
‘Meadows, Dennis “Limits to Growth” 1972 Universe Books.
Philosophical and Methodological Foundations
up from data, and in this sense theories are dependent on data, but data cannot ■’Wffll
be collected without the prior presumption of some theory of data collection J
1
(i.e., a theory of “how to make observations,” “what to observe,” etc.), and in
this sense data are dependent on theories. Theory and data are inseparable. In other
words, Kantian IS require some coordinated image or plan of the system as a
whole before any sector of the system can be worked on or function properly. J? i
These hardly begin to exhaust all the features we identify with Kantian . ■ ! J
inquiry. A more complete description would read as follows: Kantian IS arc ' j,
the epitome of multimodel, synthetic systems. On any problem, they will build ' /J
at least two alternate representations or models of it. (If the alternate represen|
tations are complementary, we have a Kantian IS; if they are antithetical, we
|
have a Hegelian IS, as described in the next section.) The representations arc’^.'-.il
partly Leibnizian and partly Lockean; i.e., Kantian IS make explicit the strong v
interaction between scientific theory and data. They show that in order to
collect some scientific data on a problem a posteriori one always has had' to IBS
presuppose the existence of some scientific theory a priori, no matter how | ■
implicit and informal that theory may be. Kantian IS presuppose at least two J
alternate scientific theories (this is their Leibnizian component) on any prob-Tf;.!®
lem or phenomenon. From these alternate Leibnizian bases, they then build up £ Ta]
at least two alternate Lockean fact nets. The hope is that out of these alternate
fact nets, or representations of a decisionmaker’s or client’s problem, there will
be one that is “best” for representing his problem. The defect of Leibnizian and
Lockean IS is that they tend to give only one explicit view of a problem
situation. Kantian IS attempt to give many explicit views. The guarantor of
■ jh
such systems is the degree of fit or match between the underlying theory
J
(theoretical predictions) and the data collected under the presumption of that
pi
I
theory plus the “deeper insight” and “greater confidence” a decisionmaker feels
he has as a result of witnessing many different views of his problem.
The reason Kantian IS place such a heavy emphasis on alternate models is
S
that in dealing with problems like planning for the future, the real concern is
how to get as many perspectives on the nature of the problem as possible. •
Problems which involve the future cannot be formulated and solved in the
same way that one solves problems in arithmetic, i.e., via a single, wellstructured approach. There seems to be something fundamentally different
about the class to which planning problems belong. In dealing with the future,
we are not dealing with the concrete realities of human existence, but, if only in
part, with the hopes, the dreams, the plans, and the aspirations of men. Since
different men rarely share the same aspirations, it seems that the best way to j
“analyze” aspirations is to compare as many of them against one another as we _ j
can. If the future is 99 percent aspiration or plan, it would seem that the best j;
way to get a handle on the future is to draw forth explicitly as many different
aspirations or plans for the future as possible. In short, we want to examine as man1/
different alternate futures as we can.
m
I
- ‘a**
f
o
2J
lan 1. Miiroff and Murray TuroH
In the field of planning, Normative Forecasting, Planning Programming
Budgeting Systems (PPBS), and Cosi-Fffectivencss or Cost-Benefit rXnalyses are
all examples of Kantian inquiry, although these arc such low-level Kantian
inquirers as to be almost more Leibnizian in nature than Kantian. The
Kantian element that these various approaches share is the fact that they are
all concerned with alternate paths, or methods, of getting from a present state to a
future state characterized by certain objectives, needs, or goals. When these
various planning vehicles have failed, it is not just because we are dealing with
an inherently fuzzy problem—indeed that is the basic nature of the problem—
but because we have failed to produce alternatives that are true alternatives
and to show that the data, models, and objectives cannot be separated for
purposes of planning.
In recent years, there have been a number of Delphi studies which in
contrast to the original Lockean-based consensus Delphis begin “to take on”
more actively the characteristics of Kantian inquiry. 'The initial Delphis were
characterized by a strong emphasis on the use of consensus by a group of
“experts” as the means to converge on a single model or position on some issue.
In contrast, the explicit purpose of a Kantian Delphi is to elicit alternatives so
that a comprehensive overview of the issue can take place. In terms of
communication processes, while a “consensus,” or Lockean, Delphi is better
suited to setting up a communication structure among an already informed
group that possesses the same general core of knowledge, a Kantian, or
“contributory,” Delphi attempts to design a structure which allows many
"informed” individuals in different disciplines or specialties to contribute
information or judgments to a problem area which is much broader in scope
than the knowledge that any one of the individuals possesses. This type of
Delphi has been applied to the conceptualization of such problems as: (1) the
definition of a structural model for material flows in the steel industry (see
Chapter III, C, 3); (2) the examination of the current and the potential role of
the mentally retarded in society (see Chapter VI, D); (3) the forecasting of the
future characteristics of recreation and leisure (see Chapter VI, D); and (4) the
examination of the past history of the internal combustion engine1 for a clue to
significant events possibly affecting its future. While all of these Delphis had
specific forecasting objectives, none of them could be achieved if all the parties
to the Delphi were drawn from the same specialized interest group. The
problems were broader than that which could be encompassed by any single
discipline or mode of thinking. For example, the examination of the role of the
mentally retarded in our society is neither the exclusive problem nor the sole
province of any special group. Educators, psychiatrists, parents, and teachers
all have different and valid perspectives to contribute to the definition of the
“problem.” Consensus on a single definition is not the goal, at least not in the
‘proprietary Delphi in 1969 by Kenneth Craver of Monsanto Company.
Philosophy:
Philosophical and Methodological Foundations
initial stages, but rather, the eliciting of many diverse points of view and |
potential aspects of the problem. In essence, the objective is establishing how to |
fit the pieces offa jigsaw puzzle together, and even determining if it is one or 1
many puzzles.
The problem of conceptualizing goals and objectives is not an explicit part of.
the three inquiry processes we have discussed so ffar. That is, the Leibnizian and i
Lockean IS are not explicitly goal directed. For example, Leibnizian IS assume ?
that the same rational model is applicable no matter what the problem and the
objectives of the decisionmaker or who it is that has the problem. In contrast, q
the Kantian IS is e:explicitly goal oriented, i.e., it hopes by presenting a
iveral alternative models of his problem to better clarify fl
decisionmaker with several
both the problem and the nature of the objectives, which after all are
the “problem.”
Kantian inquiry is best suited to problems which are iinherently J illp' g
’ to
' > fprmuj’ ig
structured, i.e., the kinds of problems which are inherently difficult
late in pure Leibnizian or Lockean terms because the nature of the j rpblem >
does not admit of a clear consensus or a simple analytic attack. On tl e; other ;
hand, the Kantian inquiry is not especially suited for the kinds of p oblems :•
which admit of a single clear-cut formulation because here the prolifer itiop of
alternate models may not only be costly but time consuming as well. I fantian $
inquiry
may
also overwhelm those who are used to “the single
best model” :
T
/
Z
O
approach
pproach to any problem. Of course this in itself is not necessarily b^d if it ,
helps to teach those who hold this belief that there are some kinds of problems |
I
p°i I
for which there is no one best approach. Social problems inherently seem to be
of this kind and thus to call for Kantian approach. The concept of “technology
assessment” as a vehicle for determining the relationships between technology
and social consequences would also seem to imply the necessity of at least a i?
Kantian approach. Many efforts which fall under the heading of “assessments” ?
have proved to be inadequate because they were conducted on pure Leibnizian v
or Lockean bases.
, K |
Hegelian, or Dialectical, IS
WH J
I'
The idea of the Hegelian, or Dialectical, IS can be conveyed as follows:
(1) Truth is conflictual; i.e., the truth content of a system is the result of .a/j
highly complicated process which depends on the existence of a plan
diametrically opposed counterplan. The plan and the counterplan represent strongly-J
divergent and opposing conceptions of the whole system. The function of-'lh^J
plan and the counterplan is to engage each other in an unremitting debate over j
the “true” nature of the whole system, in order to draw forth a new plan thatJP
will, one hopes, reconcile (synthesize, encompass) the plan and the counterplah.
(2) A corollary to (1) is that by itself the data input sector is totally
I
!
i
-Uit) I, iviiuoif and Murray Turoff
meaningless and only becomes meaningful, i.e., “information,” by being
coupled to the plan and the counlcrplan. Further, it is postulated that there is a
particular input data set which can be shown to be consistent with both the
plan and the counterplan; i.e., by itself this data set supports neither naturally,
but there is an interpretation of the data such that it is consistent with both the
plan and the counterplan. It is also postulated that without both the plan and
die counterplan the meaning of the data is incomplete, i.e., partial. Thus,
under this system of inquiry, the plan and the counlcrplan which constitute the
ihrory sector arc prior to the input sei tor and indeed constitute opposing
conceptions of the whole system. Finally, 11 is also assumed (hat on every issue of
tmpuilance, ihrie can be found or (onslruded a plan and a (ountetplan; i.e., a dialedual
lebale can be formulated with respect to any issue. On any issue of importance there
•aill be an intense division of opinion or feeling.
Hegelian, or Dialectical, IS are the epitome of confhctual, synthetic systems.
>n any problem, they build at least two, completely antithetical, representations of
■ i. Hegelian IS start with either the prior existence (identification) of or the
creation of two strongly opposing {contray} Leibnizian models of a problem.
These opposing representations constitute the contrary underlying assumptions
regarding the theoretical nature of the problem. Both of these Leibnizian
‘•presentations are then applied to the same Lockean data set in order to
lemonstrate the crucial nature of the underlying theoretical assumptions, i.e.,
that the same data set can be used to support either theoretical model. The
point is that data are not information; information is that which results from the
nterpretation of data. It is intended that out of a dialectical confrontation between
opposing interpretations (e.g., the opposing “expert” views of a situation), the
underlying assumptions of both Leibnizian models (or opposing policy experts)
vill be brought up to the surface for conscious examination by the decision
maker who is dependent upon his experts for advice. It is also hoped that as a
esult of witnessing the dialectical confrontation between experts or models, the
lecisionmaker will be in a better position to form his own view (i.e., build his
own model or become his own expert) on the problem that is a “creative
.ynthesis” of the two opposing views.
In considering the resource allocation and decision processes which govern
our society and institutions, the role of the “expert” has become somewhat
onfused and clouded. In a historical perspective the emergence of systems
analysts, efficiency or productivity experts, and operation researchers can be
viewed as the establishment of a new group of advocates. They advocate
lecisions, policies, and actions which may optimize certain unique measures
uch as benefits, costs, efficiency, etc. However, their training does not enable
them to reflect on all the factors which the decisionmaker must account for in
he process of reaching a decision. Perhaps part of the problem we have had in
’he past is a misconception that the “expert” has the only view pertinent to the
Philosophy:
Philosophical and Methodological Foundations
3
:
KitiU
OO
decision and our error in our not attempting to balance and place in perspec*:• FJJta
.1
live the views arising from political, sociological, psychological and ethical J \ 3
considerations which may advocate alternative options. Perhaps “experts”
Lu||
be better used by the decision processes if they are viewed
viewed from
from the
the perspective
perspective /fciJ
•>
of the Hegelian inquirer as just one component of the decision analysis process,
This view of the use of expertise underlies concepts such as the Policy Delphi?
Whereas, in the Lockean IS the guarantor of the validity of a proposition
agreement, in the Hegelian it is intense conflict, i.e., the presumption that
conflid will expose the assumptions underlying an expert’s point of view that 7"! iS
are often obscured precisely because of the agreement between experts. Hegelian’<rj
IS arc best suited for studying “wickedly” ill-structured problems. These are the
|
problems that, precisely because of their ill-structured nature, will produce df:
intense debate over the “true” nature of the problem. Conversely, Hegelian IS
are extremely unsuited to well-structured, clear-cut problems because here
conflict may be a time-consuming nuisance.
Except for the Policy Delphi concept of Turoff (see Chapter III, B, 1,3), the
use of conflict as a methodology is conspicuously absent in the field of
technological forecasting or in Delphi studies in general. In the Policy Delphi
the communication process is designed to produce the best underlying pro or '
con arguments associated with various policy or resource allocation alternalives. In a non-Delphi mode of communication (e.g., face to face), one of the
most interesting applications can be found in the activity of corporate or
strategic planning. In an important case study, Mason2 literally pioneered th£
ucvciupjHcm of
ui what
wnai may be
oc termed
tcrmca the
inc Dialectical
uiaiecucai Policy
roncy inquirer.
inquirer, Thi
inc
development
situation encountered by Mason was one in which the nature of the problem
prevented traditional well-structured technical approaches to planning (i.e.5
Leibnizian and Lockean methods) from being used.
In the company situation studied by Mason, there were two strongly oppos
ing groups of top executives who had almost completely contrary views about
the fundamental nature and management of their organization. They were
faced with a crucial decision concerning the future of their company. It was
literally a life-and-death situation,
since the decision would have strong
---------------------------------------------------- reper*
_r
cussions 1throughout all of their company’s activities. The two groups each J^ffl
offered fundamentally differing plans as to how to cope with the situation.
Neither of the plans could be proved or “checked out” by performing any
technical study, since each plan rested on a host of assumptions, many of them
unstated, that could probably never be verified in their entirety even if time t<?
do this were available, which it wasn’t. Indeed, if the executives wanted to 1
around in the future to check on how well their assumptions turned out, th<
t 1A
/4
♦ »* 'I
1 ♦ t
»"v
I
« * r
II
** «
r* ♦ »•« v
1
w*
mm a.
L m
Xm
—- 1 m — — - — —
/ ?
• "y
1
-Hi
'1
.
■
■
a
-j :1
4::Wl
^4^
2Richard Mason “A Dialectical Approach to Strategic Planning,” Management Science 15, No. 8
(April 1969).
i
!■
'
Ian I. Mitroff and Murray Turoff
had to make a decision in the present. It was at this point that the company
agreed to let Mason try the Dialectical Policy inquirer to see if it could help
resolve the impasse and suggest a way out.
After careful study and extensive interviews with both sides, Mason as
sembled both groups of executives and made the following presentation to
them: hirst, he laid out side by side on opposite halves of a display board what
he took to be the underlying assumptions on which the two groups were
divided. Thus, for every assumption on the one side there was an opposing
assumption for the other side. It is important to appreciate that this had never
been done before. Prior to Mason’s contact, both groups had never fully
developed their underlying positions. 'They were divided, to be sure, but they
didn’t know precisely how and why. In this sense Mason informed both groups
about what they “believed” individually. Next, Mason took a typical set of
characteristic operating data on the present state of the company (profit, rate of
return on investment, etc.) and showed that every piece of data could be used
to support either the plan or the counterplan; i.e., there was an interpretation
of the data that was consistent with both plans. Hence, the real debate was
never really over the surface data, as the executives had previously thought, but
over the underlying assumptions. Finally, as a result of witnessing this, both
groups of executives were asked if they, not Mason, could now formulate a new
plan that encompassed their old plans. Fortunately in this case they could and
because of the intense and heated debate that took place, both groups of
executives felt that they had achieved a better examination of their proposed
course of action than normally occurred in such situations.
Of course, it should be noted that such a procedure does not guarantee an
optimal solution. But then, the DIS (Dialectical Inquiring System) is most
applicable for those situations in which the problem cannot be formulated in
pure Leibnizian terms for which a unique optimal solution can be derived. DIS
are most appropriate for precisely those situations in which there is no better
tool to rely on than the opinions of opposing experts. If the future is 99 percent
opinion and assumption, or at least in those cases where it is, then the DIS may
be the most appropriate methodology for the "prediction” and "assessment” of
the future.
It is important to appreciate that the DIS and Policy Delphis differ fundamentally from other techniques and procedures that make use of conflict. In
particular, they differ greatly from an ordinary courtroom debate- or adversary
procedure. In an ordinary courtloom de bate, both sides arc free- to introduce
whatever supporting data and opposing arguments they wish. Thus, the two arc
often confounded. In a DIS, Hegelian inquirer or Policy Delphi, the opposing
arguments are kept strictly apart from the data so that the crucial function of
the opposing arguments can be explicitly demonstrated. This introduces an
clement of artificiality that real debates do not have, but it also introduces a
strong element of structure and clarity that makes this use of conflict much
Philosophy:
Philosophical and Methodological Foundations
33
j
more controlled and systematic. In essence, the Hegelian Inquiry process
•:
dictates a conceptual communication structure which relates the conflict to the
data and the objectives. Under this conception of inquiry, conflict is no longer
antithetical to Western science’s preoccupation with objectivity; indeed, conflict
actually serves objectivity in this case. This will perhaps be puzzling to those
men ft]
who have been brought up on the idea that objectivity is that upon which then
can agree and not on what they disagree. While the Hegelian inquirer does noti ||
always lead to a new agreement (i.e., a new plan), the resulting synthesis or
new agreement, when it occurs, is likely to be stronger than that obtained .by
the other inquirers.
’
'I
' I • *■’.
s
Singerian-Churchmanian IS
:■
t
"I
I is
Singerian IS are the most complicated of all the inquirers encountered thus far
and hence the most difficult to describe fully. Nevertheless, we can still give a
brief indication of their main features as follows:
(1) Truth is pragmatic} i.e., the truth content of a system is relative to the
overall goals and objectives of the inquiry. A model of a system is teleological, or
explicitly goal-oriented, in the sense that the “truth” of the model is measured
with respect to its ability to define (articulate) certain systems objectives, to
propose (create) several alternate means for securing these objectives, and
finally, at the “end” of the inquiry, to specify new goals (discovered only as a
result of the inquiry) that remain to be accomplished by some future inquiry.
;
Singerian inquiry is thus in a very fundamental sense nonterminating though it :;
is response oriented at any particular point in time; i.e., Singerian inquirers •
never give final answers to any question although at any point they seek to give
|
a highly refined and specific response.
?
(2) As a corollary to (1), Singerian IS are the most strongly coupled of all the: i S
inquirers. No single aspect of the system has any fundamental priority over any^lj
of the other aspects. The system forms an inseparable whole. Indeed, Singerian
IS take holistic thinking so seriously that they constantly attempt to sweep in neW i
variables and additional components to broaden their base of concern. For
example, it is an explicit postulate of Singerian inquiry that the systems designer
is a fundamental part of the system, and as a result, he must be explicitly considered
in ihr systems representation, i.e., as one of the system components. The
dt \ignt i'\ psYthology and sociology are iinseparable from the system's physical representation.
Singerian inquirers arc the epitome of synthetic multimodel, interdisciplinary
systems. In effect, Singerian IS are meta-IS, i.e., they constitute a theory about
J
all the other IS (Leibnizian, Lockean, Kantian, Hegelian). Singerian IS
include all the previous IS as submodels in their
t
design. Hence, Singerian ' >3
inquiry is a theory about how to manage the application
of all the other IS. In
--------------------------..........
effect, Singerian inquiry has been illustrated throughout this chapter in our
|
9
- - ...... I
.i
Ian 1. Miiroff and Murray Turoff
descriptions of the inquirers, for example, in our previous representations of the
inquirers and in our discussions of which kinds of problems the inquirers are
best-suited to study. A different theory of inquiry would have described each of
the preceding inquirers differently.
Singerian IS contain some rather distinctive features which none of the other
IS possess. One of their most distinctive features is that they speak almost
exclusively in the language of commands, for example, "lake this model of the
system as the “true ’ model (or the true model within some error limits ± e).”
1 he point is that all of the models, laws, and facts of science are only
approximations. z\ll of the “hard facts and “firm laws" of science, no matter
how “well-confirmed" they are, are only hypotheses, i.e.. they are only “facts"
and “laws" providing we are willing to accept or make certain strong assump
tions about the nature of the reality underlying the measurement of the facts
and the operation of the laws. The thing that serves to legitimize these
assumptions is the command, in whichever form it is expressed, to take them
seriously, e.g., “Take this is as the true model underlying the phenomenon in
question so that with this model as a background we can do such-and-such
experiments.” Thus, for example, the Bohr model of the atom is not a
“factually real description of the atom," but if we regard it as such, i.e.,. if we
lake it as “true,” then we can perform certain experiments and make certain
theoretical predictions that we would be unable to do without the model. What
Singerian inquirers do is to draw these hidden commands out of every system so
that the analyst is, he hopes, in a better position to choose carefully the
commands he wishes to postulate. Although it is beyond the scope of this
chapter, it can be shown how this notion leads to an interesting reconciliation
between the scientist’s world of facts (the language of "is”) and the ethicist’s
world of values (the language of “ought”). In effect, Singerian inquiry shows how it
is possible to sweep ethics into the design of every system. If a command underlies every
system, it can be shown that behind every technical-scientific system is a set of ethical
presuppositions.
Another distinctive feature is that Singerian IS greatly expand on the
potential set of systems designers and users. In the extreme, the set is broadened
to include all of mankind, since in an age of larger and larger systems nearly
everyone is affected by, or affects, every other system. While the space is not
available here to discuss the full implications of this proposition, it can be
shown that every Singerian IS is dependent upon the future for its complete elucidation. If
the set of potential users for which a system exists is broadened to include all of
mankind, then this implies that ever}’ system must be designed to satisfy not
only the objectives of the present but also the objectives of the future. Thus, a
■Singerian theory of inquiry’ is explicitly concerned with the future and is by
definition involved with the forecasting of the future. Singerian IS attempt to
base their forecast of the future on the projections of as many diverse discip
lines, professions, and types of personalities as possible.
Philosophy:
J
Philosophical and Methodological Foundations
.:W
Singerian inquiry has been conspicuously absent from the field of Delphi 5,
design; hence, unfortunately, we cannot talk about any current applications-of ? j
Singerian IS to Delphi. There are hints of Singerian overtones in those fcW ®|l|
Dclphis ihat ask people for the contrast in their real views and the views they
‘--j
would state publicly. However, none of these has ever explored the underlying j- fl
values and psychology to the extent of warranting a Singerian label. Neverthe- r ‘S
less, we can say something about what a Singerian Delphi would look like. •'
Of all the many features that Singerian inquiry could potentially add to
Delphi design, one of the primary ones would be a general broadening of the
class of designers. That is, at some point the participants should not merely
t
participate in a Delphi but be swept into its design as well. In a Singerian
-pH
Delphi, one of the prime features of the exercise would not only be to add to
■
our “substantive knowledge” of the subject matter under investigation, but just v
as much, to add to the participants’ knowledge of themselves. How do the
participants change as the result of participating in a Delohi?
I-.cJ
Delphi? Are
Are their
their jwO
conceptions of policy formation, and of who and what constitutes
titutes an “expert,^
the same <afterwards as before? How is it possible to sweep the participants
' and
■ more consciously
■
more actively
into the design of the...........................
Delphi? What areth’ ■
values and/or psychology that led me and my fellow respondents to answer
with this view? These are only a very few of the many issues with which a
Singerian-designed Delphi would be concerned, and as a result, would thus act
■T
to build into the design of the Delphi the potential for pursuing these questions
systematically. In short, a Singerian-based Delphi is concerned with raising and
J
building explicitly into the design of the technique the self-r>-rlective question: How
do 1 learn about myself in the act of studying others and the world? Why is it that some
minds think they can best learn about the world and the contents of ojther >
minds (i.e., their communications) by formal models only? Why do others
||
and others
believe they can best learn through empirical consensual means, and
others J q!
still, through multiple synthetic or conflictual means? And finally, why do
jr
Singerians want to spend so much time studying the others? What kind of mind
is it that studies others? Perhaps, perverse; most certainly, reflective—the verjr I
■’il
spirit that moved the first pioneers of the Delphi technique to want to stud^ ■ ’"’'W
sI
.J'-l
Jaa
■
"...................
how and under which circumstances a group of reflective minds was better than
one.
Concluding Remarks
In many ways a brief commentary on the strengths and weaknesses of Singerian
inquiry provides the most fitting summary to this chapter.
The strength of Singerian inquiry is that it gives the broadest possible i
modeling of any inquirer on any problem. The weakness is the potentially
prohibitive cost involved in comprehensive modeling efforts. However, given
I
I
I
.J
Ian i. Miiroff and Murray Turoff
II. C. Reality Construction as a Product of
Delphi Interaction
the increased fear and concern with our environment, we may no longer have
the choice but to pay the price. We may no longer be able to afford the
continued “luxury ’ of building large-scale Leibnizian and Lockean technologi
cal models that are devoid of serious and explicit ethical considerations and
which fail to raise the self-reflective question. We certainly no longer seem able
to afford the faulty assumption that there is only one philosophical base upon
which a technique can rest if it is to be “scientific.” Indeed if our conception of
inquiry is “fruitful” (notice, not “true” or "false” but “productive”) then to be
“scientific” would demand that we study something (model it, collect data on
it, argue about it, etc.) from as many diverse points of view as possible. In this
sense strict Leibnizian and Lockean modes of inquiry are “unscientific” because they inhibit this effort, a conclusion which we are sure most of our
“scientific” colleagues would be surprised to find and even more reluctant to
accept. But then, believing in conflict as we do, we
we might
might have
have a
a good debate
on the matter. If one were to design a Delphi to investigate the matter, which
Delphi inquirer design do you think we (you) ought to use?
J
References
$
D. SAM SCHEELE
a.?d,‘Murray l uroff’ “Technological Forecasting and Assessment: Science and/or
Mythology?” Technological Forecasting and Social Change, 5, No. 1 (1973).3
■
The mind is but a barren soil;
a soil which is soon exhausted, and
will produce no crop, or only one,
unless it be continually
fertilized and enriched
with foreign matter.
!.
1
: li
!
Reality is a name we give our collections of tacit assumptions about what
We bring along these realities to give meaning to our interactions. Each of ui
maintains several of these realities—at least one for every significant set of
others in our lives. We have domestic realities, parental-family realities, professional realities, sexual realities, organizational realities, stylistic realities.
Since this article is about Delphi design, the important thing is not how many j
different realities we each have, but that one important product of each Delphi 'J
panel is the reality that is defined through its interaction.
Realities can be described as presumed agreemente which give meaning to
our thoughts and make reasonable our actions in each setting. Most of these:
agreements about reality are implicit, and are merely confirmed smd ' '1
elaborated by our acts and conversations. Sometimes our interactions subtly '
modify these realities. Occasionally, a group’s reality is actively renegotiated W
even constructed de novo for a new situation. Delphi inquiries might produce
any of these results. The purpose of this essay is to suggest ways of managing
Delphi interactions in order to create intentionally a reality that will pro^R ■I
the appropriate kinds of active interventions.
I believe with others that there is nothing more practical than good theory. 'q'||
Much of what is related in this article is theory, but theory in search of
application, by the reader, out of his understanding, to produce results in his
specific contexts. I have used a number of examples to illustrate how realities
are asserted, modified, reconceptualized during a Delphi interaction. Each
|l
example will require detailed consideration. This is painful if the reader is
interested merely in an overview. Therefore, I have tried to write the discursive
text so that it makes sense even if the reader skip the examples. Each example
:T
is a freeze-dried caricature of a set of rich interactions. To reconstitute, the
O
I he Delphi Method: Techniques and Applications, Harold A. Linstone and Murray Turoff (eds.)
ISBN 0-201-04294-0; 0-201-04293-2
im’q condcnscd verslon of lh<-' above directed to an engineering audience appeared in the March
1V7J issue of Spectrum, which is the magazine of the Institute of Electronic and Electrical Engineers.
I
i-'j
Those interested m attempts to construct formal mathematical representations of Inquiring Systems are directed to the
following three articles:
This chapter is, in large part, a specialization of an earlier more general article:
'■ '-I
Sir Joshua Reynolds
(-. West Churchman, The Design of Inquiring Systems. Basic Books. New York, 1971.
Ian
■
/■
The references listed below are intended to provide the reader with general reviews, further background, and some
specific examples of topics covered in the article. On the subject of Inquiring Systems the best place to seek further
explanation would be:
lan I. Mitroff, “A Communication Model of Dialectical Inquiring Systems—A Strategy for
Strategic Planning,” Management Science, 17, No. 10 (June 1971), pp. B634-B648
lan 1. Mitroff and Frederick Betz, "Dialectical Decision Theory: A Meta-Theory of Decision
Making,” Management Science 19. No. 1 (September 1972), pp. 11-24.
lan I. Mitroff, “Epistemology as a Basis for Building a (Generalized Model of General PolicySciences Models,” Management Science, special issue: on “The Philosophy of Science of
Management Science,” to appear.
I
; 1
Copyright © 1975 by Addison-Wesley Publishing Company, Inc., Advanced Book Program.
All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or
transmitted, in any form or by any means, electronic, mechanical photocopying, recording, or
otherwise, without the prior permission of the publisher.
137
?|
ib
Philosophy:
I ). Sam S< hcclc
reader must supply (he cerebral juievs ami attention, such coimeclive inteipretation being necessary to make static diagrams into what might pass for
interaction.
Another problem with examples is that they represent only a small fraction
of the myriad of potential applications for this approach to Delphi inquiry,
Clearly it would be incestuous to build a design rationale solely on generalizalions drawn from available applications. Taking an expository approach based
on cases would foster an already widespread predilection—method and
technique in search of applications. This would be the antithesis of my primary
recommendation: that the particular qualities of the circumstances that prompt
and define the inquiry be used as a basis for the Delphi design. Further, let me
suggest that the results of a Delphi be seen as the product of a carefully
designed and managed interaction and not answers to a set of abstract
questions that are obtained by following prescribed methods. Hence, a slogan
for this essay: Concepts from doing.
This paper might have been called: What to think about when considering,
designing, and managing (even interpreting) a Delphi. The reader will find
many propositions asserted that require a reflection and reinterpretation for
application to his particular undertaking. The extensive illustrations are in
tended to enable the reader to develop a feel for the importance of details of
style and tone in presenting materials to panelists. These illustrations should
not be thought of as “cases” to emulate, but as necessary to describe the more
general pedagogic points about the importance of self-conscious presentation of
information in suggestive, but open-ended, frameworks to facilitate the negotia
tion of realities. Most of the illustrations are based on Delphis we have
conducted. In several cases the substantive content of the illustration has been
changed because of the proprietary’ nature of the inquiry or the possibility that
our intent would be misinterpreted if the material were to be seen out of
context. Also included is some of our thinking which occurred when we did not
do a Delphi.
The italic text provides a setting for some of the illustrative examples. Each
illustration depicts a synthesis of the interaction between the panelists with
summarization juxtaposition, interpretation, reconjecture by the Delphi moni
tor. The role of the diagrammatic presentation of the examples is described in
the illustration below (Fig. 1). The intention is not to create order or to impose
a unique conceptualization. Neither are the diagrams supposed to be balanced,
“well-designed,” synthesizing abstractions, or even documentations. In some of
the Delphis, the major part of the panelists' comments were sent back on tape
cassettes. The emphasis is a personal verisimilitude with the process of under
taking conceptual forays. Most of the panelists' thinking processes cannot be
directly shared, so we have attempted to depict for the group some typical
points of view out of which to define a reality of relevance.
Reality ('oustruction
'Sil
11______
THESE GRAPHICS CAN BE SEEN AS A
LANGUAGE. THIS LANGUAGE HAS A
SELF-EXPLICATING GRAMMAR BASED |
ON A SUCCESSIVE REFINEMENT OF
f
SEEMINGLY NATURAL CONVENTIONS,
SYMBOLISMS, AND INFERENCES.
J Here you'll find most
dialogs begin with an
I HI Tl AL 8TATtH£HT U
--------1
■a JI
.ASKING POr'X
'■
X
J
■^1
II
f; 1
\X' ’
■
&
\
V
REFINE AND SUGGEST
HOW TO GO ABOUT IT
I
e X
'
J • a
,
re formuI at i on
cr i t ique
Fresh Insights
from Panelists
integral ion
rearrangemen t
■
-i'rf
.
Each Panelist Produces X'X
ooo
con 11nue
or
Summar ize
miiiiiiiiiiiiiiiiiiiiiiiiiiui
Fig. 1.
Role of the diagrammatic presentation of examples.
I I
r
' Ji
g
.1
The particular graphic style adopted in this paper is a personal one!' It
developed out of use. It is intended to support the process of thinking and not
simply to represent completed conceptualizations. Graphic aids can b< useful in
stimulating your own thinking and organization of ideas. Start by trying your
hand at whatever seems cogent to you and make adjustments as you go. There
are a rationale and some techniques that have evolved in the use of graphics in
thinking, but their explanation would make another book. The quality of the
I
xi'l
■ ■> •’
'I
Ji.
||
10
j /
1). Sain Scheele
drawings is intended to convey the liveliness of the concepts and their impor
tant properties effectively and quickly. This is difficult to do in writing or with
more formal diagrams. Also this drawing style encourages participation, and
the organization of the diagrams usually readily admits of modification or
extension. In addition to aiding individual thinking, common graphic construc
tions, or explicit group memories, arc useful in moderating discussions. Here
the shorthand of positional relationships and the insightfulness of successive
interpretations and alterations prove very productive. Since Delphi inquiries
arc group processes, these kinds of graphic representations have proved equally
\aluablc in Delphi applications. 1 he figures may not mean the same thing to
>ou as to me, but your explanation is accessible, and therefore should be more
useful to you than mine would be.
Some of the points may seem trivial—like, “use bright colors” or “state ideas
in emotive language”—but their impact is significant. Others—like “provide a
concrete situational context or “depict an explicit theoretical framework”—
have resulted from trying to dc-abstract Delphi inquiries. It is important to
deal with the different assumptions of panelists, monitor.1 and sponsor which,
when left alone, limit the potential fruitfulness of the Delphi interaction. Most
of the methodological insights suggested here have resulted from efforts at
designing other kinds of subjective information processes, such as diffusion of
innovations, learning by doing, technology transfer, policy development,
management of creativity, and design of service systems. Look before you leap,
but eventually leap.
Philosophy:
ambiguity often might be what you want—productive of interesting premise
An array of differently bracketed realities that include a particular objec
event-concept is often useful. For example:
i
Four young adults who are retarded enter a restaurant with older couple. Pw ’isls zfc
till3
V
asked to select likely responses for restaurant manager, waiters, and other patrons jrom a\ffw/
provided. Many panelists added their own. The panel included parents of, and professional
who work with the retarded, as well as individuals to simulate response of gen
community. The responses of the panelists could be mapped'.
MANAGER
hel pf uI Concern
sbrtled
avoUan<«
(minimize)
acceptance
benign
“AC
(mitigate)
/COUPLE \
I
y\
inertrepriaVe
concern
Uncertainty /I
' (manage)
T.*r
awet-y
court«ovjly setleJe
' WAITER
PARTY
/
Conce»lw.cnt
Cenfvrion
clues
I
a>vu«ty
nutivali^rt
JJ .
leave
< iffrtktnti*"
\
interfit >
Co«<p«in
J
OTHER PARTIES
iC£>mtrcnt»
('oiicrpls of Re ality
4 urionty
ke'pfjneu
I0O
nt
Mi
: II
tea
■,
■■
J
■
(nwntlon)
mhrttl
Monitor is a term I use for the person or group conducting the Delphi inquiry, i.e., preparing
the materials, interpreting the responses, integrating the insights, presenting the results, etc.
l or a longer exposition of, and details about, this concept see S. Lyman and M. B. Scott, A
Sociology of the A bsurd, Appleton-Century-Crofts, New York. 1970. and H. N. Lee. Percepts, Concepts,
and Theoretical Knowledge, Memphis State University Press, Memphis 1973.
•j
i- •
retarded
I he impact of one conceptualization of a situation upon others and the
influence of the various constructions of reality assumed by the Delphi panelists
generate what 1 believe are the most significant results from any Delphi
inquiry. Panelists can be made aware of these seemingly subtle differences in
the nature of the realities they presume in the course of the interactions. The
panel can then produce a common reality for the situation at hand as a result
of their participation.2 How this comes about is determined by the monitor of
the D'elphi interaction. If panelists are reluctant to make specific contributions
ot if a very wide, almost unrelatable. array of conjectures is produced, one
suspects that there are great differences in the meaning each panelist is
attributing to the way the inquiries are stated in the materials provided to
stimulate response and the way panelists expect the results to be used. This
I1
Reality Construction
at sizjrtc C
■ IH
s'
(taik-aVovt)
'*1
faJ'V ; * li
Fig. 2-A
)
i
f .1
l-.; f I
Il '
iss
:: a
I). Sain Scheele
Philosophy:
Later, panelists were asked how repeated contact with the retarded would affect the key
actors?
I
successive contacts with retarded with and without couple
wkji >
pub lie image
Utt c n
>
“Policy '
of k c r S '
personal feelings
WAITER
locking
^uidjnee
fin4
V«>i*vior-
norma 1ize
Reality Construction
Ii
i .,’5 ■k'aj
Husserl,3 Merleau-Ponty,4 Heidegger,5 and others, which led to the formulation
of a phenomenological epistemology that is now being applied by neo-symbolic
'|
interactionalists and ethnomethodologists. The concept of a negotiated reality i
can be related to Mitroff and Turoff’s discussion of the philosophical bases for
inquiry systems in the preceding article. This discussion describes a range of
:
inquiry systems (IS) using as differentiating labels the name <of' the
’ principalphilosopher whose concepts undergird each approach. The array includes the.
Leibnitzian, Lockean, Kantian, Hegelian, and Singerian IS’s. Since these
categories are well defined there, the philosophical premise for an IS based on a
view of reality as a context-specific product of interaction will be described in
relation to this framework. First, to select a label consistent with the others will
be slightly misleading, because any one name tends to obscure the contributions
of others and imply that the ideas are largely set. Dub!/ng this territory of
philosophic exploration after Merleau-Ponty seemed the least misleading. To
make a contrast with the Singerian analyst, the Merleau-Pontyean is concerned H ./I
with the particular reality created by the “bracketing” of an event or idea out
i
I
offtr tvggc«li«n s
r
RETARDED
L----------
.
OTHERS
responj
io
;CC(, ptance
or Jvoidjnce.
--------------- EXPtRiENCt
tidy to’- aj»4t"‘r'3 Contacts
DISCUSS
• become
confirm new stereotype
si
Fig. 2. What is going on in the restaurant.
On the other hand, you may want to create greater focus and consensus. If
this is so you can begin with examples for interpretation instead of general
questions. I his enables you to direct attention in subsequent rounds to contrasts
between the assumptions imbedded in the initial situations and panelists’
contributions. Differing reality constructs can produce divergence from even
seemingly unambiguous statements. Focusing attention on differences in the
reality constructs will usually yield either a more refined and widely agreedupon definition of the appropriate context or clearer and more precise distinc
tions between competing contexts—possibly leading to an estimation of the
idative probabilities of each, or a search for present options that could
influence the circumstances.
In the preceding discussion the notion of socially constructed or intentionally
negotiated realities was employed. This concept grew out of the work of
of the great din of experience, rather than explicating a pragmatic reality that J
can be used to define possible actions. Truth to the Merleau-Pontyean is
agreement that enables action by confirming or altering “what is normal” or to
be expected. By contrast, the Singerian views truth as an external articulaticln
. JI
of systems to define goals and options for action. Reality is viewed by the
Merleau-Pontyean as tthe
u“ product created out of :_*
—*1---------intentions
and1 actions instead
I g
of an external basis forr intelligent actions. To reiterate Mitroff and Turoff.
Turoff, the
?
importance is not which philosophy is “correct,” but which is appropriate io
the kinds of situations one is attempting to impact. The Merleau-Pontyeain i
2?
inquiry system seems applicable to situations either where a redefinition1 Pf ■
contextual reality can facilitate the generation of new options, or where the .
acceptance of a new reality must be negotiated to create the impetus for
technical or social change—as, for example, in defining as “progress” a reduced
or more limited consumption that would permit reallocations instead of;
“progress” as lower unit production cost to support increased demand. This i ■
philosophical point of view leads to viewing the future as a situation where both
the dominant reality and the technology are invented as well as inherited, and
where culture is transformed as well as transmitted.
Merleau-Ponty and others suggest reality be viewed in a new way: as a
currently prevailing shared assumption about a specific situation. This implies
that reality is the product of our experience and not external to it. In a
7:
|
J
j
1 IJ
KI
.'I
3Edinund Husserl, Ideas'. General Introduction into Phenomenology (trans. W. R. Boyce), Allen &
Unwin, London, 1931.
4Reniy C. Kant, Merleau-Ponty and Phenomenology, in Phenomenology (Kockelmans, ed.), Doubleday, Garden City, N. Y., 1967.
5Calvin O. Schrag, Phenomenology, Ontology and History in the Philosophy of Martin Heidegger, Revue :
Internationale de Philosophic, vol 2 (1858).
■> >
+
ir
I.J
H
D. Sam Scheele
commonsense view, reality is a collection of observable things and occurrences
which is animated by a society of individuals. .Although we are not usually
aware of these distinctions, our everyday realities can be seen as created by us
out of the meanings we give things and events.6 Since we do not exist alone, we
are continuously asserting and having validated or challenged our definition of
what s going on or “what it s all about.” Our collection of situational
definitions constitutes our reality. We select realities from our repertoire that
seem appropriate in order to know how to act, attribute meaning, and interpret
behavior. I his means that instead of continuously discovering more of an
external verity—“the reality out there”—we are, wittingly or not, continuously
adding, verifying, or revising our “shelf-stock” or versions of what is normal or
to be expected in particular circumstances. We each have a shelf-stock of
realities that have been produced by our interactions.
Earlier the basic philosophic question had been “What is the structure of
social reality?” Now phenomenological insights have transformed this question
into “What realities have been or are being socially constructed?”
What does this mean for conducting Delphis? bdrst, since the results of a
Delphi are produced by interaction, albeit highly structured, the results can be
said to constitute a reality construct foi the group.7 Because processes of
successive refinement, like the Delphi, strongly lend to induce convergence and
agreement, the monitor of a Delphi should purposely introduce ambiguities,
even disruptions. These might take the form of “angle” items8 to challenge and
redefine reality as well as “quirk” items9 to act as catalysts to explore the limits
of the reality. For example:
Mass transit could compete ivilh private vehicles by offering more than lower cost—
particularly in enhancing the use of commuting time by offering'.
Round 1:
types of services?
ClNFORMAT I ON)
pF,<*r
PURCHASING
Uf/ soiT
••
_____
(HEWS)
"
tffrt
45
Round 2: Can't money change hands—transit as market place? (angle)—
Relate services to attracting and serving youth? (quirk)
S
mJI
•*!«*»<< ikJi* a*« •»>•■**(
r*r«»i«"c* >•« th<ir •>«.
^ENTERTAINMENT)
jQPor
---- --- -- SJ J ‘••’•e — '
------------
4ir*<«4 Mntl J*
..X’iV
5^
CONTEST &
' reaFoMtcomef I "wagering*
a-t've syifemi
Round 3: Hoiv might the service be organized and supplied?
cr»4i*
_ ___ j;Ifu—“"'l ckeck
revenues
1
lew direct A
//v 5^1
e/P«>tes
COMMUNICATIONS
yonMttWM 4
*.M.*
\
ZSK-
nt
"Bi1
//
?Ue< J
1
..h:
I
ON-BOARD DEVICE$?\
Sc ’
[profits
■?
1 I
/Aoff\et+
Costs
!
______
J
V
r [ USER-CONTROLED DEVICES’
—J
T<
’O
I.t look ups
rt . > t . o. s
Reality Construction
whati jrtilaWe?
\. 'jry?! J'-V
1
•’ "««•..J or
'
rZrl'r—
Philosophy:
Fig. 3-A
6Jack Douglas, Understanding Everyday Lip- Aldine. New York. 1970.
7Hans Peter Drietzel, Recent Sociology \o. 2—Patterns of Communicative Behavior, Macmillan, New
York, 1970.
8Example: "Suppose you had invented the better mousetrap and people had beaten a path to
your door—would they buy?”
^Example: the famous rejoinder, "And how does that relate to the Jewish question?” or Stephen
Potter's functional equivalent, "...but what of the South?”
Fig. 3. Activities to enrich mass transit experience.
I I
aj
1i
Second, since the knowable reality is in competition with the other concep
tions, including the idea of reality as a negotiable construct, the unknown or ,
unexplained cannot simply be attributed to greater degrees of complexity (i.e.,
the “more data and better instruments” gambit). Thh means that further
efforts to obtain information, such as a Delphi, must go beyond attempts to .• 48
unravel what has often been assumed as merely additional complexities.
T
Instead, information should be sought that can shape reality, such as identify
ing new considerations or introducing new options. This means that the systems
I). Sam Scheele
Philosophy:
ing described are viewed as indeterminate, arbitrary, delimited, multiplistic,
en convenient fictions if this facilitates discourse, but not as complex
artesian clockworks. In conducting a Delphi then, "what if” and "why not”
ms might be introduced or highlighted if suggested by a panelist to prompt
msideration of new alternatives.
Third, the reality we construct can be expected to be different by at least as
ich in the future as our technology will be advanced or our society restrucircd. This is almost always overlooked by forecasters and other futurists,
edictions may well occur as forecast, but their occurrence will not necessarily
an the same thing then as it now seems that it would. You can note people’s
aivc understanding of this in their response to the prediction of new occuraccs with the statement, “...but I guess they [the people of the future] will be
adv for it by then.”
Tuurlh, expect reality to continue to be negotiated. This means that the kinds
realities within which occurrences will be given meaning and be understood
11 vary from those prevailing at present. To a large extent changes in reality
nape the kind of attention, consideration, and effort that will be spent on
veloping a new idea. They also determine whether the new concept seems
’ausible, desirable, and feasible. Further, both interest in, and advocacy of, a
cw concept, along with precipitous events that come to be associated with it,
n shape reality to the extent that a new concept becomes one of the ways that
*ality is defined. Our present notion of the "urban crisis” is an example of a
mcept that has become imbedded in our realities.
A Delphi inquiry, then, might explore two sides of the negotiation of reality
'ith regard to a specific occurrence: (1) how alternative realities might affect
.»e meaning of the occurrence and how likely each is and (2) how public
rceplions of the occurrence, the interests and activities of its proponents, and
he concepts and ideology that come to be associated with it will shape the
ality that is negotiated.
In many cases it also may be useful to consider the possibility of precipitating
vents. These events (seen from hindsight) have often dramatically altered the
ilective awareness of consciousness of the society. For example, contexti.'cific realities were shaped by the assassinations of the Kennedys and King
■ ■-‘i’ gun-control measures, Nader and “Unsafe at Any Speed” for auto safety
.mpaigns, and the Soviet Sputnik for the U.S. space program and educational
direction.
Let us look al how these considerations can be handled as part of a Delphi
quiry. To grasp how different prevailing realities might affect a particular
pic, one can posit several alternative "reality gestalts.” Encourage partici
pants to supply their own realities. But I believe examples are necessary to
.trify this notion initially—even if these proto-realities are overly “cari'lured.” 'Fhc example below illustrates this kind of inquiry.
round Q
Reality Construction
round
(2)
47
1
i
p ;
I
'•I
round <3
I
GRIM t
GRISLY
!
! Ji
EXPANSIVE
EXPRESSIVE
I
I
MEDIOCRE
MORI BUND
0
n
Sii
01FFUSE
DIFFERENTIATED
t
u
INTEGRATED
IMMINENCE
rounj ®
ip
SATED
SACCHARINITY
DESCRIBE
7" "
J\me»nmgs
"p"ro f e s sTonaT\^^^^'*r 1 ‘eat ‘®ns
football
II
i nnovat ions
to
cons ider
Ji
I
I
j
II
' 3'
Fig. 4.
Professional spectator sports—prevailing views.
I
Also, you might want to probe the participants’ insights regarding items that
might be significant in the construction of realities in the future. Here it is
useful to suggest items that you believe are important as possible examples.
Even include candidate-precipitating events—although their nature and timing
are by definition unanticipatable. Knowing that the meaning of events is, so to
say, up for grabs can sensitize managers to the use of public-attitude-shaping
tools. Introducing these perturbations will begin the discussion and evoke
additions and comments from the panelists. An important corollary point is:
the meaning of the future in the reality of those taking actions (of necessity in
the present) may change. It has dramatically in the past ten years or this book
wouldn’t have been considered for publication, possibly not even written. One’s
vision or concept of the future has not always shaped reality in the same way. It
is as important to know in what ways images of the future are given meaning as
to have a full complement of alternative futures (see Chapter VI, Section D).
’3; 1
' -'J
■3
>
p
■1
fl
■1
I
iti
D. Sam Scheele
The medieval glory-bcyond-thc-travail-down-here view of the future would
prompt a different kind of action from the expectation of a better tomorrow
through hard work and technology which has characterized the industrial
revolution.
row
’d (T)'
. "in
’it’ial
SUGGESTED
[clause*, of occurrences')
-I
/ \.c
lU
5
vEVENTS,
round (5)
<
i<
U
‘(
L ~1
rout'd
nedia covefogc
7
other ao^erse rucl.c-ty
t'pund (J)
oosil've events
cfmtegr*if separate scenario^
Fig. 5.
3*^
to refine '
<
Reality Construction
mentahs has at least these three characteristics:
49 n
<4^^
• an epistemology—shared assumptions about what is valid knowledge or li]}™
“truth,” how it can be discovered, and what constitutes acceptance or i tS
"proof ”
' wll
• a social theory—concepts that a society uses to explain its own workings
to itself, including the identifiers for constituting aggregates.
*
• a guidance process—methods and ground rules worked out and ac- ■
cepted by the members of a society—what Harold Lasswell describes as
'f
“who gets what, when, where, and how.” At different times this is also"
called “politics” or “management.”
z\t a macro scale the dominant habitus mentalis of society is being negotrateiMlfl
Ours is now. I would like to summarize some of the changes we expect in the.j:M|
post industrial society. To begin, I’d like to share a name we have coined for the
postiiidusi! i.d society that is descriptive of it, instead of merely indicating aft#
industrial. This new label is “the idiomergent era.” This is a neologism. Idio is a
Greek root indicating uniqueness, separateness, distinctness, as in the word
4
idiosyncrasy, and in its most peculiar sense, idiot. It has been combined with §3
the Latin root mergere, to cause to be swallowed up, to immerse, to combine or ^8
coalesce, to lose identity. The result describes a more separate and a unique S
kind of interdependent aggregation. It seems that the idiomergent era will be
characterized by both extreme variety and increased uniqueness in individual
<;
behavior and organizational purposes and at the same time be dependent on
the greater integration of the individual and organization into the structures of
a v aricty of specialized communities of interest, producing networks of cohesiveness for the society as a whole. Thus the term “idio-mergent” denotes
greater individuality and autonomy combined with a deeper involvement of the
individual in fluid social arrangements, innovative organizational structures,
and experimental personal relationships. This describes change in the vanguard
of society: a new habitus mentalis that will spawn new realities.
The habitus mentalis for the idiomergent era will no doubt affect the design of
Delphi inquiries. Exactly how depends on the response of practitioners and ■ 'i
monitors to their perceptions of what is relevant and their selection and
advocacy of ideas that come to be in good currency. I will try to share my
construct for this new mental climate. Yo" realize that I have no way of
knowing the future, nor am I merely asserting an explanation to serve as a basis
for judgment. These and many other possible descriptions of what this image of 4
the future is can be offered. My view is that I am engaged with you in
negotiating what to expect from Delphis. Use it as you will.
Before getting to differences in the epistemology, social theory, and guidance
process of the thinking style of the idiomergent era and what these augur, for
the design of Delphi inquiries, I would like to suggest another feature of this era
—an increasing concern with the future. Greater interest in the future -has
resulted from a series of changes in our dominant social reality. These changes ! ’i;
seem relevant to consider in designing systems for assembling useful inforrna- 'I '•
' ~. ■ I
Construct Scenario for Testing Models cf New Programs
round (?)
Philosophy:
A
Professional spectator sports—reality shapers.
Mental Climates and Styles of Thinking
The dominant reality of the “modern” world has been called the positivist
functionalist view. In this view reality is “out there” (we are in it, but it is not
in us), discoverable by improved access and greater attention and presumed to
be ultimately knowable—given sufficient time and diligence. Lukacs10 and
Panofsky11 call these global realities the habitus mentahs of an era. A habitus
10Georg Lukacs, History and Class Conciousness (trans. Rodney Livingstone), M.I.T. Press
Cambridge, Mass., 1971.
Envin Panofsky, Meaning in the Visual Arts, Doubiedac, Garden City, N.Y., 1955.
aI
4
I
Ii
■
■;4
I
1). Sam Scheele
(ion. First, change became accepted as normal. Reality became dynamic, but
this alone could not make the future more important
important now;
now; things
things would
would just
just
work out differently. Second, technological development and social and physi
cal mobility have increased the number of options available to decisionmakers.
1 his made their reality wider, but not necessarily extended deeper into the
future. Third, the number of self-evident paths, inevitable selections, and
unquestioned preferences was reduced by an illusion of choice fostered by great
increases in the availability of education—both schooling and media-supported
awareness. This ballooning of information has expanded reality to include the
possible, the imagined, the dreamed, the fantasied, and all other “residents of
the future in good standing.”
1 he itch for a fix on the future afflicts both people and organizations.
Individuals want to know more about the future so they can choose places to
live, things to experience, roles to adopt that will be highly desirable when the
future occurs. Businesses analyze future opportunity areas and establish or
ganizations to prepare to exploit them. Institutions assess their roles in the
future and begin reconceptualizations. Government investigates the future and
identifies many unacceptable contingencies that require more controls. In most
cases the future reality is portrayed as an unfolding panorama of large-scale
changes in general circumstances and situations whose meanings are anchored
in the present. I he inference to be made for the sponsor of an inquiry into the
future was, “If I know more about what’s going to happen, I can get ready for
it,” Il was like spying on or getting the drop on the future. These futures were
not the world of will, action, and events; but broad backdrops of scattered
possibilities from which a series of succeeding “presents” would coalesce on the
moving pointer of time. The message was that to “realistically” take effective
actions the future must be considered. T his future dimension of reality, now so
important, has been chiefly portrayed as a passive set of probable possibilities.
The growing recognition that the “meaning” of things and events is not
determined, but in fact refers to social agreements that have changed in the
past, and which can be renegotiated, even for each circumstance, marks the
beginning of the idiomergent era. The reality of the future as a broad
techno-cultural script is too ambiguous to support particular undertakings.
Attempts have and will be made to bracket specialized future realities that
contain only highly relevant possibilities—but in most instances the number of
considerations in even a “bracketed” future will not support finite analysis to
select a single course of action. But the process can be reversed. Decisionmakers
can attempt to negotiate a specific reality for the future which, if they are
successful, can come to be widely enough shared that it is realized through
successive choices. 1 he future will be invented. Thus in the idiomergent era you
can expect attention to focus on the selection of methods to create and portray
new realities that have the potential to be actualized through tacit agreement
between the intentions of those affee ted.
In order not to lend credence to the prevailing presumption that Delphi
inquiries are concerned only with the fuiuit, let
return to look at wavs that
Philosophy:
Reality Construction
51
Del ph is can be used to contribute to the store <of knowledge, enlarge the ,1
society’s understanding of itself, and improve the style of governance ini an
}
idiomergent culture. For each of these three characteristics of an era,_jhe
1|
currently prevailing concepts will be contrasted with their idiomergent cotJn- j
terparts. This will be done rather telegraphically to provide a background for
’J
...................
an example of an item in a T
Delphi
inquiry. Elucidating these points more
clearly would require another book.
.
yiVa
Differences in Epistemology concern not only what is sought and accepted as
knowledge, but how information is categorized and organized to su ppprt
actions. As a man thinks, so is he.
1
Industrial
Idiomergent
■
Reality is external and knowable.
Reality is constructed and negotiable. .. Jg
Hypotheses are general and are offered
for validation.
Hypotheses are context-specific and sei
to redirect the ongoing discovery process.
Categories of description are based on
observable differences in measurable
dimensions.
Categories for description include intend
tions and selections and are based on an
esthetics of knowledge.
Anticipate future conditions by extrapo
lation of past performances and be
haviors (exploratory images).
Invent future images out of creative in
tegrations of expectations : normative
gestalts).
II
“ J
I I
■ 1
To illustrate these epistemological differences, the best example seems to be
the development of categorizations of consumers for use in market planning for
new products. In the industrial era, consumers were defined demographically,
geographically, and by aggregates of features that indicated a consumer’s
relative propensity to buy either specific types of items or to buy in a particular
way. One idiomergent approach would be to develop first a model for the
hUj /
/M >
<
a
> ( cc S x => )<!
J
5 Sts
k
D
O
oo
J*}
1
process of product introduction, next those categories of consumers likely to be . 1
involved, and then where and how to locate them and their probable numbers. ■ j!
Differences in Social Theory are alternative views of how the society is organized,
its tenants perpetuated, and its actions explained. Society is a name for us to
call “what is.”
Industrial
Idiomergent
■'•Ml
|i'.
Behavior follows laws that can be derived
from situational observation.
Behavior occurs in activity sets that preI
sume and assert meaning.
i
; pll
Actions are explained in terms of expres
sion of internal motivations and conflicts.
Actions are taken in response to assessment of interactional consequences.
which define possible actions.
Actions and the need for their explication 4^
define roles and organizations.
■/ •'.J®
Society is categorized by structural and
functional properties to permit mea-
Categorizations of society mark limits for
special realities in order to facilitate com-
pt.|S-|oo Organizations and roles are structures
06370
I
viit-enw>n!
nd
ian
t i/%rt ** rsrl
-• V-i •» •*» rx
t
' .t|
,J
|), Sam Scheele
1 or the Delphi inquirer these differences in social theory suggest greater
emphasis on finding out about the appropriateness of societal norms, roles, and
institutions, limning how they came into being and, probably most importantly,
suggest ways they might be reshaped. The example describes an application to
one particular field.
"I
Philosophy:
53 : 1
Reality Construction
Round 3: Identify potential difficulties and problems that might be expected to be
encountered.
*
■
; S
I
Round 4: What might you propose to avoid, amelonate, or resolve these difficulties?
’
■
Round 1: Please elaborate on the diagram below—additional relationships between the
ielarded and blind person living together to provide mutual support.
J
provide mob ility and visual ass is tance
/
BLIND FEMALE
CONFINED TO
WHEELCHAIR
||||||||illlllllllllE
\
38 YEAR-OLD
16 YEAR-OLD
>
TRAINABLE RETARDED
MALE WITH NO
PHYSICAL IMPAIRMENTS
mutual assistance in daily activities
‘
n te rpre t stimuli and lead in decisionmaking'
(1) Aggregation of deviants could tend
to further segregate them from
the community.
](2) Need times to be apart and means to
to tell each other that avoids
interpretation as rejection.
(3) Complete dominance of retarded male
by blind female -- OR reverse.
(A) No mention of sexual interests(?)
(4)
j(5) Possibility of marriage, death,or
life-cycle changes ending the
relationship with ill effects.
IJlll HI Eaphasi te
®f reUtiMisUr|
UBWvWity SwareMU Vy U«*l
rrnTiI II
|lUll|j
2
.ru1...
•! *i**t
UyL
nnilln
JL/rni Swarf
J] 11111
of oHerl
ev«?y two W«ckl
- Vjf c4 o«
4
VLfm
W*»t W»rP«.t .rth rotifer.*- 11111 Ik
of
’lound 2: 7he relationships described by the panel have been aggregated below’, suggest
ehere indicated which significant others might contribute to make this relationship more
workable, and what they might do. .4s a neighbor living in the same apartment building,
vhat could you do to contribute to this relationship?
Z3
<
|
'
moUx-itj
tVij yHi'-t.lity
IU11 l| £ rotewary' oi" r rclahoM "pt* w
*«»« iC»yi
winy
»
IK'Vibh'U
5 it,
poM
:W.I'h« +» •ovt to.,
«xcv.»5«t or«^«
Wjlj || r«jyl»r rotoho"* — W«1 otwl/y pair Inp kvt » », 4’i ♦
11111 11II1111111i1111111111111111liIIt111111111111111Ik 11II1111111111Iff
Fig. 6-A
:il - IJ
! SiI
..
•
■
I.
Fig. 6. Symbiotic life-support relationships for the retarded.
1
|
'■
I d:'S
Differences in Guidance Processes used in a society reflect implicit, usually
unwitting, agreements about the mutual concerns of groups (from pairs to
organizations to segments of the society as a whole). These concerns and their
mutuality change. Sometimes they change in response to awareness of threats
and opportunities. At other times, improvements in techniques such as com
’I
munications, media, information systems, or analytic schemes change the way
fundamental relationships are perceived. Economically and politically the
negotiation of power continues. The society’s guidance process is not simpK
decreed—as by a constitution or framework of laws. It includes these, but it u
built up rather than laid on. A society’s guidance process embodies in conven ...■kJ
tions all of the expectations of individuals concerning how things ought to b<
“fairly” and “knowledgeably” decided in order to produce the greater goo<
and what to do if it’s not happening as it ought. Societal guidance is all w$
ourselves are not contending about and in which we assume someone else is Th
watching out for our interests.
/i
i
Fig. 6-B
)
I
rt
• I
I
i■ J
j I
Philosophy:
I). Sam Scheele
Industrial
Idiornergcnt
Selection of conceptual models based on
analytic evaluations including ideological
considerations.
Selection of approaches based on evalua
tion of prototypes and related implica
tions viewed pragmatically.
Competitive determination of potential
market for goods and services exchanged
in a predominately private marketplace.
Entrepreneurial innovation to create
markets for goods and services exchanged
in a publicly moderated marketplace.
interests do they have which could be appealed to?
1
EMPLOYEES
Wit r«4wctio»u 5
lower cost in time
and expense of the
trip to work
(
I
X_ ,
management
cost
SHAREHOLDERS
U-. OF
*f X,
.
/non - TAtARy'
BENtFITS
'irre-.l faru.Tw*
»VIUP
Zstrmce,r»r-y | iMMUl
fchittt AaX .Thfr hr*'/3
h“'«ina
\yr«X*lty
to H
j
&
j| (X'fjU
irr thoice'
tn.iceX. X ■H.- m X.
I
PRIMARY
ENTERPRISES
1-
- --IW
anO
■ i':*
r
i
emergence of the idiomergent era will contribute directly to the need for more
Delphi and Delphi-like processes for collective consideration of topics. How-
i?:
Capita I
0vtl»y«
Re^ul^tory
P.I.ciO
E n v iron •
EjCil.t
_
_ nt
> X.d
hl«
protection
__ 1 i i
iHCrtfcASt
I
~ u;
PROFITS
r-h
MMOR
n
Connd er, where
pottiMe.re lo
simuructonj cating office*,
M«nvf»ctori'ngt
r,V1,e
3M auemhly
reiocxt i.m I
4eer*tr#|I liei /ecilftort
L nw houemg
SUPPUE.RS
e\ci, evaluation of the usefulness of Delphi inquiries will be made more on
their suitability as a process for discussion and choice than on the concerns
A«p •THAR
R***.
iXTlXfcSTT
StftVICt
ftlrtlNUItl
SJ
jjT
Round 4: Suggest administrative mechanism to manage program to achieve objectives.
tion of a topic that has no established institutional advocates.
I
reduce “overhead" costs of quality urban lifestyle?
EMPHASIS ON
RELOCATIONS
lojn
Y
e/gry I ?
3o minutes
Pools
,
n rpiwployer f^orMore^ or A
_____ >
/ WQIKt-R TAAMt- |
F0R
EMPLOYEES
f ■1 ~"'i X
XmvKi - »<ki
Zeffic*. Jilo
/^AUe’.VIy
J
i
^/deeentrabijtion
jof the workplace
Primary Sector
Employers
or rat i©"
■■
I —J
I
Requirements for New
and Expanding Enti t iesBgtHJ^ «
I
I
HP
-yy/lNCENTtVES
gairerr-
relocation
of
2
ho using
C
^v$»n
a [>....<>
..0^4^
|
It
Ntw SlRVlCi^C
Property 1 Dvsiaeti Taxes
to reflect the
Employee TraMH>rtat io* lotpact
50Vt RNMtNrl
tmeioYin
Service Businesses
fl
7
:|
&
Local
Govemmen ts
MCAT»ee SAViMAS
>h txjmm or &ipes*tMCtt M OVTU*X»
F«M> MMS MUiWCP
r WquiRt au. Ntw
M4VCIMA TO CtfJTAiauT*
U KANfMATATuH
SVITCM* C»>1t XAVAITO
U1AI. *VA<T OF MVtiNA
ANO MOT <M*nT <HT Of
t tyncm exr<Mtion
■—
Fig. 7-C
=
•f ■' i
State^^
•VC J j- Fig. 7-A
Round 2: Select most promising approaches}
I
||
- I
'■I I
Round 1: What actions are possible to reduce peak impact demand for transportation to
CAR
I
I
y WSfe’
iNruitMCi
LKAlMMAL
CHA ICO
99
•riAATOlU
Fig. 7-B
about the accuracy, insightfulness, or agreeableness of the results. The next
example was chosen to illustrate how a Delphi can be used to focus considera
I
/i
prodvci|V,t-y
Management incorporating participation
of those directly affected in the selection
of means to respond to short-range op
portunities and vulnerabilities identified
by multidimensional plans produced by
continuing review of an unfolding con
text.
I
GOVERNMENT
INTERESTS
FAX INCENTIVES
ehvifenaenUl J
. wrajc charges Jg
Mnereas*
1 he changes in style of societial guidance that I expect to characterize the
Staggered hours
:■ J
■•ii
■ •'$1
Collaborative development of proposals
for action for particular situations with
participative review of specific implica
tions.
Management predicated on adherence to
comprehensive plans developed to reach
selected goals and objectives defined in
limited dimensions to be periodically re
vised.
I
55
Round 3: What interests and organizations would be involved in implementation and what
f
Representational selection and approval
of proposed actions within general juris
dictions based on broad policies.
Reality Construction
sal
j
a
I
T
1)0
I). Sam Scheele
Philosophy:
Round 5: Consider implementation'. What kind of demonstration program or other approach
might be used to rally and mobilize support^
SfcUECT IO MAJOR EMfuOYtRS FOR
KCUCXATION PeMON STHATION FROfcEAM
Ef>OM
in
t »>pplitr<
important
.
A
Z'v’
"^77^
e*Plc>e»I I
RELOCATED FACILITIES
Fac.I■ r i*}, t«
Refill to
: o ?t r J
Fig. 7.
Est»wl>«k
-ro asjut
IK RELOCATIONf
3
TZ
\v
0°
”1
tJTa »li S-AINT
Of
New ctNieaj
Relocation incentives in urban transportation policymaking.
i
I
Group Influence on Reality Construction
We all have highly idiosyncratic experiences, ideas, and fantasies. While they
actually are our own, their meaning is created in the crucible of their
interaction with what is going on in our various contexts. For example, a dream
shared with people you have recently met has a different impact on the reality
being built than the same dream would have if described to an informal caucus
during a psychiatry conference. In the first circumstance, your reality is being
created from such elements as your image as a person, the meaning of a
relationship with you, the place of personal interpretation in your respective
motivational structures, and the bearing of dreams on events to come. In the
second circumstance, the dream becomes material for negotiation of profes
sional prowess, theoretical differences, clinical inferences, concepts of con
sciousness—even a trigger for discussions of what this discussion group or the
profession is all about. You can imagine that sharing facts, feelings, and
proposed actions would also impact differently on the reality being constructed
in these contrasting contexts. Try imagining the contrasting reality that could
be created in each of the two contexts just described (telling potential friends
about yourself us. a discussion about you as a topic in a professional group) for
each of the following: the recent death of a relative; your interest in traveling
to India; an idea for creating a “kitty” for vacationing by selling the work of
local artists to your fellow employees.
57
Delphi inquiries are conducted with groups where the individual participants
might be expected to view the group differently, particularly since they
often anonymous. For most, “being on a Delphi panel” has no particular ’
meaning independent of the topic. Frequently the presentation of the inquiry
items .nxl other information is indefinite or ambiguous about the nature of the';
panel as a group. This directs the panel’s attention and energy, in part, to
task of defining the reality of their relationship instead of creating richer
realities for the domain of the proported topic. Further, the quality of th’elZ'*^
reality within which the individual inquiry items are elaborated varies with the Z
ways the panelists view the meaning of the group and its findings, insights’,
recommendations.
This means that it is not enough that the panelists share a firm idea of the
group's identity, but the perception of that identity tends to shape the nature of
the individual messages, the quality of the interaction, and the character of
what is produced. Table 1 (below) presents a crude taxonomy of the features qf
groups and some notion of how a group’s conception of itself affects 1
products of Delphi interactions.
The monitor (or modulator) of a Delphi inquiry strongly shapes which
conceptualization of the group is assumed by each participant. This usually
implicit
group
the mode of
that
each finds
'S
r— concept
——r‘ of
— the
— &•
—r dictates
—
.... .interaction
............. —
-------appropriate and determines the reality each uses as a frame of reference in
attempting to contribute. Several different assumptions can often be made
depending on how
1
individuals interpret the implications for reality of the , '
messages in the communications and materials provided them to begin the
interaction. Until each panelist is comfortable with his notion of the group’s ;
nature and believes it to be confirmed by responses, it is difficult to produce
meaningful contributions. In the paragraphs below, I suggest some ways in ; . ■■M
d'J
which a Delphi monitor can shape the group’s conceptualization of itself and
adopt a mode of interaction that can produce results which will be congruent
with their anticipated use.
Transactions will likely become the dominant mode of interaction if there ir: 1 ’v|
&<r<ecT ihcemTuES
ikiwep RecocAT'ONS
^/r«< oilA' ■ bv i links
r-vc
Reality Construction
■
■11
■< 1
• abstract categorization of the panelists by expertise with inclusion
obviously icicvam
uuviuusxy
relevant spceiaiucs.
specialties.
• formal statement of items for consideration, possibilities for estimation, .
or hypotheses for assessment.
• reiteration of responses categorized by original items with few additions j; i|
.•fl
i
or deletions.
■
• a product expected of the inquiry as a pooled judgment that will have a
“validity” believed to be greater than that of any individual.
i \j
!1
In adopting this mode panelists assume that the participants have no other
commonalities and expect none. For most this means they will adopt the most i
familiar model of interaction—probably “answering a questionnaire as
r
f
f-’
I
jB
Philosophy:
I). Sam Scheele
Table 1
o
Group
Examples
Collections
of
Individuals
Occupants, attendees,
shoppers, passengers,
gathering, respondents
Mode of
Interaction
I ransactions
Experiences
Unique but stylized, comfort
able, shaped by random inputs
with a strong interest in
symmetry and completeness;
occasionally evoke powerful
new gestalt.
Purposive
Group
Colleagues, families,
members, associates,
collaborators, teams
Episodes
Tightly structured, matter-offact, repetitive, presumed heavy
with meaning, resist redefini
tion when alterations (usually
catastrophic) occur
Unions, professions,
swingers, activists,
supporters, delegates
Event
i
I
Students, aged,
work force, minorities,
dieters, communities
Affairs
Amorphosis, working against
stereotype, exploring for
unity, proclamalory, infre
quently build basis for
becoming affiliation
Agents
for
Society
Citizens, representa
tives, concerned people,
reformers, preservationalists, ecologists
Occurrences
Global, dogmatic, historical,
mythological, polemical,
sometimes deteriorating into
diffuse “glossolalic” diatribe
I
I
T
‘accurately’ as possible.” Exchanges will tend to be formal. They will center o
proof or “refutation” of ideas identified by the monitor. There will usually-bie^l'J*
a statistical integration of the group’s assessments. Anonymity will likely be^/P '|l
used to reinforce the panelists’ self-concept as impartial observers in
1!
"analytic” reality. Panelists who are uncomfortable with this reality will devote r’
great attention to selected issues, often digressions from the principal considera
tions.
Variations in the transactional mode can be effected by (1) beginning the
Delphi
with open-ended,
agendaless
items in search of meaning (to CApiWIC
explore U1C
the’ ; '
I.......................
I
“
O
agendas and gestalts of the future held by the panelists); (2) selecting the
extraneous and derivative responses of the panelists for feedback and later
exploring the interpretation of the discourse with the panelists (to identify
latent options, issues, and considerations); (3) extracting the explanatory pre
mises from the panelists’ responses after two rounds, then focus on estimation, '
followed by attempts to elaborate a system of relationships (to build theories
and models from premises derived from content rather than technique); and
(4) starting with a preliminary design of a thing or program, proceed to iterate 4 ;i3i
critique and redesign to produce a better design by successive interaction of
multiple viewpoints, or to derive by interpretation of the “designer-panelists’” ' ’•
responses which considerations seemed important or even critical to the design. L? |
...........
Ritualistic, symbiotic, one
dimensional, subtle shift effected
to build power or influence,
attempt to be prescriptive, some
times unsuccessful and produces
schism
Defined
Group
O
t>
Perfunctory, patterned, pro
visional, ambiguous—in
exceptional cases completely
new and perturbing: possibly
creating a new pattern.
Friends, parly,
clique, players,
diners, meetings,
trips, class
( j roups
O
Nature of
Realities Produced
Casual
Groups
o Affiliative
i
I
I
I
Reality Construction
I
' 'I
Obviously, there are many other manipulations that could be developed, but i
these are sufficient to illustrate the type of variations that can be created for
particular applications. A Delphi should not be undertaken to validate: concepts which you already have developed and refined; panelists want to make '
significant contributions and these will seldom build meaningfully on highly j;
elaborated initial concepts.12 Delphi inquiries are obviously applicable to much ;
more than obtaining pooled judgements about particular options.
Experiences become the prevailing mode of interaction when:
if
>
• the panelists are familiar with each other and identify with the subject V
or sponsor of the Delphi inquiry.
• the involvement is for a fixed duration, with known consequences that ] ' 4
follow a familiar pattern.
• original items serve primarily as jumping-off places for further inquiry.O the generic form of expected product of the inquiry is clearly indicated.
[
Inherent in the assumption of this mode of interaction is a commitment
the anticipated process and results. The principal interest of the panelists is the
particular qualities or insights each seeks to contribute. Even when anonymity jZ t|
I
of panelists is maintained, their messages tend to be informal and approvalseeking (other-directed conjectures) instead of raising formal distinctions and
'i
8
- ----
I
^Presenting a less-than-complete concept takes guts. To each panelist there will be “obvious*’
omissions. You will be severely criticized. But, I believe a stance of calculated naivete produces the
best results, if you live through it.
[
I
1
>
Ji
1). Sam Scheele
striving for accuracy and defensibility. Panelists in later rounds will attempt to
make contributions that will affirm their brilliance or uniqueness. The result
can be omission of important but obvious points and dangling insights (the
monitor can point out these). When the panel is largely drawn from a single
discipline or field of application, the interaction quickly moves out to the
ethereal zone instead of enriching the context for action. Modifications by the
experiential mode include: (1) try to organize the panel into “teams” to
represent particular interest groups or points of view (to preview negotiations or
develop highly detailed but still preliminary agreements for consideration); (2)
present initially a clear and detailed description of the expected product of the
inquiry; or a unique and special situation that calls out for great imagination
and close collaboration to
be described
first so
« that the prevailing rules for
.. ----------------- ...u
judging the suitableness of contributions are partially suspended (to spark
creativity and develop a relatively complete conceptualization of a viable: new
approach).
Episodes are the characteristic mode of interaction for groups that:
• are made of individuals who have a significant and continuing relationship, possibly more than one.
• deal with familiar topics or operate in well-known ways.
• are more influenced by consensual implications than external factors in
making choices.
• are more concerned with the perceived quality of the interaction than
the product—in fact the idea of a product, as such, may not even be
considered.
I his mode of interaction has the highest emotional content and the most
potential for prompting action based on the insights produced during the
Delphi inquiry, which is by definition supplemental to the group’s normal
interactions. In these cases you should consider face-to-face group processes.
Other techniques (such as program-planning method—PPM, nominal group
ing, multiattribute utility assessment) may be as productive, require less time,
and have more spillover benefits than a Delphi. It is also imaginable that a
group of strangers could sufficiently internalize instructions and materials that
would be dramatic enough to induce an episodic style of interaction for an
inquiry. Such an inquiry would have to be virtually a simulation. The
substantive insights produced by a small purposive group can be expected to be
highly specific to their context. The insights produced by such groups regarding
the personal and emotional dimensions of the topics are probably not generally
indicative. In fact, it may be necessary to create a simulated purposive group in
older to probe the interpersonal and emotional dimensions of circumstances
that are themselves the subject of conjectures or competing proposals.
Variations on this mode of Delphi inquiry’ are limited because alteration in
the premises usually induces the group to adopt a new reality resulting in a less
Philosophy:
Reality Construction
.
|
highly bonded mode of interaction. The most interesting results are produced
in instances when the group’s strongly shared reality is collectively modified irir<
response to new insights and even to incorporation of new members. However^ .
this collective modification of reality is not always completed successfully and it
can be expected several realities will prevail. This can result in dissolution of
the group, a breaking down of the group process, or the production of highly , ;
emotional and idiosyncratic contributions.
■■Wi
Events, often with more form than content, distinguish the interaction of O
groups which:
• are made up of individuals who see themselves as having a commoi•nality
worth recognizing, but not becoming related or involved on
oth“’l
matters.
-- ■ i
• center their interests on the specific issues of their commonality and,
continuance (survival) and enhancement.
• are guided in their interactions by “the way things are doney:g
hierarchical considerations and formal divisions of responsibilities
!
••••itiesiP
within the group.
• make choices to allow greatest individual divei . while maintainingr ,
the essential solidarities that define the group.
. ,s.
fhe inherent one-dimensionality of the interactions of these groups exhibits .
itself most clearly in the importance of agenda-setting. Whether highly infor^t
mal—“wait, should we be discussing this”—or employing a very formallyd
maintained docket, these groups find it essential to screen topics for relevance)’
to the organizing commonality. Issue focus has become more necessary) asvfl
society becomes more diverse, in order to keep an affiliative group frpn^g^
fractionating into several puposive-type groups, each of which would maintain^;
multidimensional commonalities. To survive and grow, however, such affiliative!';
groups usually have to deal with threats and opportunities that are on (thef
perimeter (if not outside) the group’s reason for being. This means that most
;
the messages exchanged can be assigned to one of two distinct classes: proforrrui |||
statements necessary to group maintenance and expository statements intended>zffl
to modify the group’s prevailing notion of their common interest. There will be;'
very few exchanges about things that affect them as individuals or alter the
group’s relationship to the society.
b• ®
Delphi panels who think of themselves as affiliative groups tend to proceed ihi i.j
two-step responses to the items in the initial inquiry. First, they m.>.<e narrow |
and direct responses to each item. Second, they have additional consideration^.
j
I he whole panel is usually asked whether these should be dealt with. If
panel or some substantial subgroup agrees that the additional items should bc>
added, then they are taken up. Consequently, much of the reality that 13
created has to do with the meaning of considering certain topics and not the" .4
meanings derived from actual consideration of those topics. A Delphi should‘O
I
B
11I
I
iJ, Sam Scheele
never be attempted with a “single-purpose” group until they have agreed that
there is something that needs their attention which would not lake care of itself.
Affairs are the interactions of groups characterized by:
• being asked to consider items that apparently are of interest.
0 having an abstract categorical definition of their
membership.
continuing attempts to give direction and create meaning for the
interaction impelled by recognizing the insiufficiency
"
of the group’s label
to serve as an adequate definition for their involvement.
• a tendency for speakers to become "spokesmen,”to indulge in formal
ism, and to imply collective interests for the group.
lhe beginnings of these groups are abstract categorizations which have
meaning for the inquirer or the explainer, but seldom for the group itself. So
when asked almost anything, these groups scramble in several directions in
search of cohesive interests. I he direction of a question or inquiry presumes a
meaningful response. A variation of this occurs when the inquiry is directed to
explore differences within the group. In this instance the inquiry presumes
distinct factions. I he group will often develop these “factions” in formulating
responses, but the factions are transitory. A corollary dynamic in the interac
tions of these groups is the prefacing of comments with remarks which either
attempt to identify the speaker with the whole group or a major faction, or will
uniquely distinguish him and his contentions. These interactions tend to be
characterized by a series of declarations—"Run it up the flagpole and see who
salutes.” Occasionally in short interactions, and eventually in almost all, a
senes of declarations will coalesce a faction (occasionally including the whole
group), which will concentrate thereafter on the construction of a position and
purpose. Once constituted, such factions interact as large affiliative groups,
usually in a way that produces schisms and dissolution.
Because researchers, administrators, and most of the rest of us have been
almost conditioned to think of society as divided into distinct kinds of "labeled”
groups, the response of abstractly defined groups (e.g., doctors, women, radicals,
ducaled) is frequently sought in inquiries which end in producing results that
do not satisfy the inquirer. One way to remedy this is not to expect a coherent
position to emerge, but to look for a pattern of diversity in the group’s assertions
and the ways in which these points are qualified. I hese groups cannot examine
issues that apparently concern them, because, to paraphrase Gertrude Stein’s
description of Oakland, there isn’t any ‘they’ there.” Instead, such groups can
be used to design product or service “lines” or to formulate arrays of options
that seem complete—covering most of the considerations. Also, these groups
can be used to define and map the factions implied by a number of related
specific proposals for action. Panels constituted in this way are used all too
frequently. As an alternative, consider making up panels that have two or three
distinctive subgroups and manage feedback selectively.
Philosophy:
Reality Construction
Occurrences are what goes on with groups where:
63V' |
i■ 3
• the membership is made up of individuals who believe they are acting I
for the interests of the society in general as they see it.
• topics serve as occasions for participants to expound then general ’
ideologies.
• most messages refer to particular cosmologies about the: way
supposed to be or are <appended to statements that describe world-views. ;
• the envisioned outcome
a set
is m
a collection vf
of more
------- of
— —
—- of
—• interactions m
widely agreed-on precepts that will guide the group, its constituehtV i
and, if possible, all others.
| ’ yS
I
All panels on occasion become forums to prescribe for the ills of society.
These groups virtually insist on this mode. In most instances, this is coun
terproductive. This is because in most circumstances it is necessary to produce a -;
fine-grained, narrowly bracketed reality to support specific actions. The reason
jil
for arranging this kind of a panel is to obtain a broad consensus, but not on.
T
everything. Since nothing is ever done in general, but only in a particular 1 O
instance, introduction of situation-specific facts and considerations can help totl-tS
focus the panel on agreeing about society’s interest in the instance at hand, ’
fl
.J
instead of on society’s interest generally in things of this kind.
Another reason to arrange a broadly based general panel is to develop a?
(I
contextual mapping that would describe the overlapping large-scale realities f
which underlie different parts of a society’s response to any complex issue. Here? 'J
it is useful to inquire about “for instances” that are likely to be seen by differential
elements of society as analogous to the situation being inquired into. “Well, it’s'B® I
just like thus-and-so... ” and similar phrases are powerful connectives to;
ready-made meanings for a proposed idea or approach. These loose, even'1
unrational, associations tend to bring along existing emotional loadings whichf 3
panelists in this mode tend to deny. In a similar way, apparent metaphors (e.g., ? :!§
like David and Goliath) and apt personifications (e.g., rape of the ecology) 5
should also be explored. This can be done by asking about which ones suggest
themselves or provide several examples to be selected from, modified, or added b
to. Identifying analogies and metaphors is a more fruitful way to define subtle
differences in global constructs than asking individuals to examine their coreftM
values and beliefs and describe them directly. While there are problems with
interpretation of these complex symbols, the interpretations can be iterated for’ ■ ’.I
refinement, and also for validation. Since the “picture” is changing, and
“observation” using a Delphi inquiry is not a snapshot, there is a gross limit on f J?'
the fineness of the picture of a whole society that can be produced. But, evenM^B
faint shadows on pale gray are helpful in many situations.
Do not do a Delphi with representatives or spokesmen for society when you
want information for particular practitioners; use experts “about society” who
are respected by the practitioners. In fact, you may want a split panel
•
|
D. Sam Scheele
Philosophy:
65
Reality Construction
i
■ . I
containing both experts and practitioners to create a joint reality shared bv
both that can be potentially extended to society.
Making a Delphi Context Specific
»Vhat I Meant to Say
The conceptual productiveness of the interplay of intentions and circumstances is lost in many Delphi processes because of the lack of concreteness in
the context. The mental life of most individuals involves attribution of ai^
intimately known context, even when this context is the world of abstract ideaL5‘SJ
Most of the foregoing is tentative and much of it highly abstract. It is even
embryonic. 1 share it for whatever use you can make of it and apologize for its
density. I here are obviously many hypotheses here for research on the process
of Delphi interaction that go well beyond the issue of “accuracy” or “relia
bility.” My only suggestion is that you try to supply or create examples from
your own experience that seem to explicate these generalities. 1 attempt to
summarize the main points below. In the next—and final—part of this article,
1 will offer a few' pragmatic suggestions for the designer—monitor of Delphi
inquiries. Obviously, not all of these strategies and techniques should be
at tempted simultaneously.
However, the subjects of most inquiries, including Delphis, are stated in terms
of general propositions; those things that apply only to an instance are defined
as being of no significance for action (those things yet to be determined,
decided, or done). So the difficulty lies in the extraction of generalizable
propositions from particular instances.
I 'B-wl
I can offer two approaches; there are undoubtedly more. The first approach
; -j
s
Summary
(1) Each Delphi interaction produces a shared reality which is initially
formulated by the panelists from their expectations and the style of presenta
tion used in the initial materials; this particular reality is elaborated and
modified by the succeeding interactions.
(2) During the process of interaction the panelists’ responses can be expected
to deal with personal esteem, group self-concept, and relevant world-view's as
well ftS to convey substantive ideas, forecasts, and estimates.
(3) Patterns of styles of interaction can be fostered, retarded, even trans--Hiued in order to produce results that will have greater insight, more useful
ness, higher impact.
(4) 'The size and shape of the reality within which things come to be viewed
s more important than the specific substantive descriptions produced by the
panelists.
(5) The believability and significance to the user of the results of a Delphi
nquiry depend as much on the user’s perception of the clarity, compellingness,
and fit of the reality implied and possibly defined by the results as on the
perceived quality of the information.
1
»
is to develop a concrete example with detailed features that are typical of a
more general case. Another source of specific contexts is to create hypothetical
constructs which make a distinct break with most present circumstances and • ’ \
are almost fanciful, e.g., on another planet. Because the hypothetical construct j
will differ in important respects from the panel’s conventional circumstances, it ■ j
often prOVOke new thinking, retard
rlicriiccmnc
or,nzw.
•.that
'
discussions of routine experiences
are rooted in presently accepted and unquestioned conventions, and create
unclaimed and common experience
base that can facilitate collaborative
(dr.
. ...... _
competitive or <compliant) interaction. Emergencies and other kinds of special
circumstances can also be used as hypothetical constructs within which possi-W • .1
bi lilies and insights can be developed for later transfer to the more general
everyday circumstances.
Once a case is selected, proceed with the Delphi, focusing on the detailed
case initially. Then shift the inquiry by asking the panel to create genera
lizations—possibly proposing some yourself and iterating the ones they pro
posed. You can next elaborate these general propositions by asking,“What
other circumstances do you know or can you imagine might prevail, and within
each would these propositions be valid?” The second approach is to develop^
several special cases for initial inquiry
using subpanels. Later, the results of
...................................................................................
each subpanel can 1be shared
'
' with the whole. Then, ask
’ rfor general' proposijJ
tions. What is forthcoming can then be refined and elaborated with examples; J
-------Bi
.
1
i
,'
Some Design Considerations
I
: '.•.U'hi-''
■ L
I shall now briefly focus on some design considerations to help you develop a
better Delphi and orchestrate the interaction. These suggestions do not constiutv a checklist. I hope thev will trigger some ideas and preparations that
dicrwisc would not ha\e occurred to you. Each couid separately be the subject
4 a much longer discussion. Supplying these discussions is left to you, the
leader, when you pursue particular applications.
The Domain of Time as a Context
Another shortcoming of “looking at things in general” is that time is assumed
as an undifferentiated flow—that one minute or hour is as good as any other. If
asked, we all would acknowledge perceptual differences—time dragging or
racing. But, unless our attention is specifically directed to it, we tend to
■
s
•
I
I). Sain Scheele
overlook structural distinctions in time. The structural features that limit the
allocation of time are of increasing importance as time becomes a scarce
resource. When inquiring “What will people do?” “How can resources be
used?” “Which group is being designed for?” the time domains become critical
concepts. Time domains can be categorized or partitioned in many different
ways, as can any continuum. Conventions in the dominant culture have created
workdays (and, increasingly, work evenings and nights), regular times off,
lunches, weekends, holiday weekends (recently added to by Congressional
action), annual vacation times, “the holidays,” special events, and so forth.
'1’here are also times in the so-called life cycle—youth, teenage, young married,
l ime domains can be treated as the subject of policy and design, for example,
staggered work schedules and new work patterns (4 ten-hour days, 6 twelvehour days with a week off, etc.) You can also invent new “times” for things, like
the enculturation of daily exercise periods. Figure 8, below, deals with several
of these considerations and relates them to the selection of market opportuni
ties.
Demand for recreational services is limited more by the time consumers have available for
their pursuit than by (he expenses involved. Time is experienced in periods that have each
lume Io have then own meaning and r\pnlution\. '/'<> urate new markets pa lenealionu!
srirnf\, the meanings of pailuulai pemuls <>i dumaui\ tan be allcied and new pent>d\ tan
be cieated and given meaning.
67
Reality Construction
Philosophy:
\ Ia
Roilnd 2: Identify time domains where additional involvements for working adults without
children would be most welcome and suggest recreational pursuits.
;i
''
i i JSs
.------- $ee«i waited - rot rewtRliiing
LUNCH TIME I BREAKS
^S9
' J------ 1 crtojit ikJ Iui« to Av«U
H\e<tir>< otkeri - enffr-tki.rf
:
■
■■
i
interort A^W irvoluoxtot
1
invthlinf
pulzle* J.
\
till
£)■ •
7 tor 3 Jk^t "Of
I V»<aFio«« miHoA
I OM
I
or
\ otnRt* j
flSriuMtt
4r
I
I
$»'•. Lzg
l.f
co.t'H
SS) : A
SV J'i
4<v«WF
Legi timizatiorr
of Interaction
wore
l«tcr«it>»g atkerj
aU »k’iilzJ
Deci sion\
Making
Processes
L'J
I
ItamU
Udo
SPECIAL TREKS
1> 1 temporal
I
, . ;ft,I
4
-A -Mpli
Round 3: Select market opportunities you believe are the most attractive and suggest specific
(y/)r\ of piotluil.x and services and actions to create expectations and aggregate markets,
l
j
-few
_______________
Round 1: Suggest new time domains you think could be developed and indicate how existing
domains could be revised or differentiated to create new markets for recreational services.
INTERACTIVE PASTIMES
f WORK-BREAK DIVERSION
J
Too.. >o vr-tt-wfrk W't*
I
to
ftrtii l|
<*•*
4
i'i
______________________ ______________ _■
ff (VENTURES FOR ART
'
1
b»rb •»«•»♦!
»»<l fhi<kaI it'Jll • »ilb IU
ot\»r4
| use DR BAM PUBLIC tPACt M FltLP^
J)
je I.. t r.ni<.[- »ural moJiw*.
itjoor A o/idoorr
IHtJ
AMTRAoFocoay
HisTeni
W-TsS? ef
s? Arr
A“*
fe
__ , '>a|
i
ioter f evtat*o» - coiticrao*
4«vicr
"fc'U
P'fK ATTKACTn/E
Move BAU. To TARGET
OR
OFRSNC-NTS AwA'f
IHlTIAItS]
C-*w' To PLAY WITH
■'1
lp«« lary hollow pl)ftK
revue
r«fi»k
Ptv UiFRiAFTAL
H>noH»T RATIO*
’b*!i
.
—K fr
,‘w.4»
fr«
- it»y
h*rj 1
ANNUAL VACATIONS
EVERYDAYS
I.Ik
fdnving
s.
^7»io»utir>^
LU^CRtS
BREAKS
*4Htl.l 9»
>»*tr /
Xl**f4j/
x
arrange contests;
'TOO FULL
WEEKENDS
ff^krz
Ir? notice io out standing
mot a I
i|Lfe fer-for rhjnces
relay!
T^?
fwtur*
Fig. 8-A
ISPECIAL OCCASIONS
Create associations' vviK service
____ cer+rrs in
facil.hci,at
I I dnve-i»<s,and w.tk FiMF-trthjit depot}
Fig. 8
movt ill?
pltiic
tarjrk
Spring - loiktU
•LAUMCHER’
■ D.
<mor
I
pr«.«<S |
■irtinir^
^hiHin£ oppone»ls ‘Vall*
ball w/ckctroaics y‘.v icoonj
Trek
|IRO***f< Fr«t«Upe eVir"r"'-|lll||
rvivl t, ■ ci ime ns i o nf I
-ARHSTIC - IHVRUTIVE
'ahmual
CVINTS
Xctrttr arti job
(- or It-trc1 cnvicej
I io try out. nrw
t i4e<< L *kl,g *
play serially
L-rv otLori
for one week
wwt.i F i >• f
■
or »r,
USE AJ ICEBRIAACR
enviro""«"ts
/
..-h
#n»r
playiro alone ■ at sametime interactive
wirb others
play gawc with you"J effectives
>o<*r«Uf rvlfk aoj rffl-’ F1*/
serve »• bas>» for
Camra'Si" to interest othtr}
fj
/ FUNOS
'
For
^CHARillES
"""ECIALHED AVC11QNS[
EVAbUSU^R^OEDS * Kt-PORTS
Time domains and market opportunities.
I
HI I
JJi
. iI
I
•u
I). Sam Scheele
I
Philosophy:
Reality Construction
-"i
' I. w
69 i
■
>'eUing a Product Out of Results
Every Delphi inquiry can be expected to produce results of some kind.
Jsually, if they are thought about at all, results are seen as things to be
'■captured” from their existence “out there somewhere” by the Delphi process,
.nstead, try to visualize, or think about, the results of a Delphi as being created
y the interaction. Just as the important properties of a building are not simply
(hose of the sum of its “stones,” the results of a Delphi are not just the
.idividual items produced in the interaction but the reality comprised by the
hole. I believe that a Delphi will be more productive if the monitor sees his
role as producing results and not as “surveying” things that are already there.
Here are some more design tips:
Creating panels is usually the first task, unless they are a given—as with an
organization that wants to study its own future in a participating manner. Even
.ten you may want to augment the group. Three kinds of panelists are
ngredients for creating a successful mix: stakeholders, those who are or will be
directly affected; experts, those who have an applicable specialty or relevant
xperience; and facilitators, those who have skills in clarifying, organizing,
ynthesizing, stimulating... plus, when it seems appropriate, individuals who
can supply alternative global views of the culture and society. The proportion
f a panel from each category should be tailored for each application. Note\
There are almost never enough women on panels.
There are no general rules of thumb for creating panels. For example, where
ptions and interests are clear but acceptance of direction and action is
fractionated, stakeholders might predominate. If it is clear who has to act, but
nut clear how, a heavy salting of experts may be best. Where issues, re.lionships, and values are unclear, a preponderance of facilitators may prove
'Host useful. Pay attention to the minority as well as the majority views
expressed by each type of panelist. Also, watch that panelists contribute as you
xpect—experts, in particular, drift into acting as stakeholders in aggressive
■uoments and facilitators in passive ones.
After abstractly thinking about the panel, it comes down to names. Most of
te time you will not know everyone you want. Sometimes you can start with a
'•mall group of potential panelists and begin by discussing possible names and
. marching as a group for interesting and appropriate candidates. This is a belter
rategy than searching lists of relatively unknown names by categories.
Frequently, staff members of professional societies and other “people brokers”
. an be utilized to suggest panelists. Often the most fruitful part of a Delphi
rocess is assembling a panel, not simply for the particular effort, but because it
enhances your contacts. At least, try to develop two alternative tactics for
Jentifying panelists and consider their trade-offs before embarking on the
Delphi.
Stimulating response of any kind from quality panelists is not easy. Getting
quality inputs is even harder. Few people like “questionnaires.” And, from
their point of view, engaging in abstract speculation with people you do not
•know or whom you want to impress about a subject that is not central to your,
■/■fl
interest (and where your performance is unrelated to your survival) is hardly
compelling. Motivation requires more than a good cause, a pep talk, and
thanks. If prospective panelists tend to be uninterested, find a “worthy” or
prestigious sponsor, or make participation of significant publicity value. Then a
token payment or “honorarium” will stimulate interested responses. Payments
alone, even of fairly sizable sums, will not assure quality participation. It is
difficult to pay panelists based on performance since it is difficult to agree on
what is ex]pected; panelists generally believe that “what I do is what’s enough.” * a
Attractive and potentially stimulating peers> are probably the most powerful j
incentive for participation. Consider gifts and “in kind” rewards for partici
pants, because often a sponsor can provide these at modest cost—particularly
travel. Still another way to organize participants is to employ a two-step
approach: suggest that key participants use their staff, students, or others they
can induce to participate to formulate the specific responses. The key partici
pant then collects, reviews, and submits them for each round. Here publicity
and recognition of importance by other individuals are powerful incentives.
Once initial participation is secured, the next important consideration is the
quality of the materials, their tone, style, and presentation. Use lots of color. !
Give the materials style. What you send out reflects the significance of the I
/nl
1
1I
I
• A'HI
I
Ao
Ba
ll
Pl
i
inquiry and the value that is placed on it. Use emotive language and vernacu
lar expressions to engage panelists and convey importance of results—not
another abstract study. Detailed situational descriptions, mini-scenarios, bits of
conversation—all help. Other media besides print can be used effectively to
make response easier and more scintillating—even speedier. These include tape
cassettes and even local “interviewers” in cases where the Delphi sponsor or
manager has a large potential manpower pool that could be employed at low
cost. Don’t forget to pay attention to creating an audience for the results and
communicating this to panelists. Examples of good and bad responses are also
useful, particularly for discouraging stereotypical remarks and obvious but
unhelpful “insights.”
Orchestrating interaction requires attention to details in the panelists’ responses
and a feedback of an overall movement, countervailing forces, or whatever
macro-observations are appropriate to describe what seems to be going on
between and with the individual items. Highlight divergence and consensus,
even when both are true for different sets of responses to a quirk item. Be sure
to cheer at least one item from each panelist—everyone needs to be
appreciated. Introduce ambiguous factors if discourse focuses on individual
items when you would rather explore relationships between items. You can also
“stipulate” certain constraints when particular items have evoked massive, but
trivial discussions of immaterial distinctions. Where appropriate, supply tenta-
-I
■Si
iiat
I
a
-‘J
I
•9
..J
70
D. Sain Scheele
live theoretical constructs and cosmological frameworks. Indicate the way
responses are being categorized for review and whether your interest centers on
enriching the decision environment or identifying and refining options for
consideration. /Mso, you can employ cross-impact analysis (see Chapter V),
comparisons with analogous situations, and other techniques to provide a basis
for evaluating responses in a different light. Again, at the risk of overemphasis
in this essay, as interaction progresses, share with the panelists the reality
consiruct(s) you believe they are referring to and shaping by their responses.
Inlerprelation and summation of responses is never complete, because the panelists
do not send in their “heads” but only their responses. A complete record of the
interactions is unwieldy and diffuse and difficult to use. It is important to begin
‘’interpreting” responses during the interactions, even at the start. This makes
interpretation subject to review by the panelists and can include their refine
ments, which 1 have found most insightful. Summation can be made from
several points of view—using the one best for each major consideration in (he
inquiry. It is often useful to point out nil findings, omissions, and ignored items.
'Fry to capture and describe the reality that was negotiated by the panelists and
the monitor, because this provides a perspective for understanding and indi
cates the application foreseen for the results.
Communication of the results may not be felt by some to be a legitimate concern
for researchers. Findings, it is supposed, have their own importance. However,
the process of acquiring “understanding” carries with it an impetus for action
that is not conveyed by presenting the conclusions alone. Most Delphi users, I
believe, intend to influence the formulation of policy or the making of decisions.
Doing this requires attention to the communication process. Well-organized,
iucid, attractively presented reports help—but they are not enough. To aug
ment the report you should attempt to: (1) create involvement—one way being
to include the intended users or members of their staffs as panelists in the
Delphi, or else to engage users in parallel deliberations that focus on the same
issues, infusing from time to time interim considerations developed by the other
Delphi panel; (2) generate interest—this usually means fanning respectable
controversy and building a climate of expectation for, and awareness of, the
Delphi inquiry among those who are important to the users, particularly
colleagues and constituents; (3) present the results interactively—this can be
done by organizing a simulation exercise where the panelists, the staff of the
Delphi monitor, or some of both take roles that are significant in the user’s
operating environment and then enact a scries of situations that can de
monstrate which Delphi results are applicable, and when and how they might
be used. It may be useful to define a communication process as part of the
initial conceptualization of the Delphi design so that the implications for its
communication are anticipated in the way the interaction of the panel is
orchestrated.
I
I
Philosophy:
Reality Construction
7i Inta
I’ cannot imagine that all the recommendations in this Chapter will prove
unassailable or always useful. I hope they will spark some iinterest and provoke
!■ , ■
a few insights. 'Phis will require a lot more of you than of me. I have benefited
in putting this down, but you will have to work to make it apply to your
situation in order to profit. To quote Henry James: “Many things have to be ‘■‘WWrS
said obscurely before they can be said clearly.”
|
li
•a
&
■'w
-!,1f
I
„ kg?
#1
Is
'i
III. A. Introduction
HAROLD A. LINSTONE and MURRAY TUROFF
11
* lilt
-■ it
In this chapter we sample the rich menu of applications. The purposes of thcrJ
Delphis are as varied as the users. Seven authors focus on specific planning$
tasks in the areas of government and business. Additional studies are alsat|
sketched in this introduction (“Comments on Other Studies”).
Government Planning
The four articles covering this field address national, regional, and organiza
tional planning problems. Turoff deals with the basic concept of the Policy
Delphi and reviews several efforts of this type. Ludlow’s concern is resource
management in the Great Lakes region and a major aim is to establish
improved communications between technical experts and interested citizens.
Jillson focuses on the
f use of Delphi as an integral instrument in national drug
abuse policy formulation, operating from three distinct perspectives: “toppriorities in System |
down”, “bottom-up”, and “issue oriented”. Jones <explores
’
Concept Options for the U.S. Air Force Laboratories, with emp’ isis on
comparing views of four in-house organizations competing for funds.
'
All four move beyond the use of Delphi as a forecasting
f
tool and stress its ;
value as a communications system for policy questions. A policy question is ■ ■
--------------- - aspects, qsuch
as gas
Oaigoal
formation,
for which
'
defined-J Ihere
as one-I
involving ’vital
formation,
for which
overall
experts,
only
advocates
and
referees.
Its
resolution
must
!'
there are no
take into consideration the conflicting goals and values espoused by various •.
interest groups as well as the facts and staff analyses. It should be clearly
understood that Delphi does not substitute for the staff studies, the committee
W
Hi
If.
*
...
deliberations, or the decision-making. Rather, it organizes and clarifies views in J
an anonymous way, thereby facilitating and complementing the committee’s
i '^n
w°Whereas
rk- Turoffs panelists constitute a homogeneous group, Ludlow seeks to-m.h'Mi
IIPI
establish a communication process between the potential users of new know-:g|||^
ledge and a team of interdisciplinary researchers. He raises a point which is of. VO®
concern for Delphi studies generally. The probability used is of the personal
subjective type; it can be interpreted as a “degree of confidence”. Scientists and '"-H
engineers are brought up on a different kind of probability—frequency
The Delphi Method: Techniques and Applications, Harold A. Linstone and Murray Turoff (eds.) ■■
ISBN 0-201-04294-0; 0-201-04293-2
• :
Copyright © 1975 by Addison-Wesley Publishing Company, Inc., Advanced Book Program.
'
AH rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or
j
transmitted, in any form or by any means, electronic, mechanical photocopying, recording, or .
otherwise, without the prior permission of the publisher.
75
®
^<1
Harold A. Linstone and Murray Turoff
currence, i.e., the limit of the ratio of the number of successes to the total
iinber of trials as the latter approaches infinity. Thus the frequency type of
lability assumes repeatability of the experiment (e.g., tossing a coin). But
■e subjective probability has meaning even if an event can occur only once. A
•xing match is a one-time event; the odds usually associated with such a
tch indicate the degree of confidence” in the outcome on the part of
formed bettors. Both definitions are mathematically valid and have been used
develop distinct probability theories. Businessmen intuitively use the “degree
oilfideiicc” concept and therefore have no built-in resistance when faced
ill it in Delphi <|ncslionnaiics.
-aidlow also presents an evaluation of Delphi by the three participating
ups——technicians, behaviorists and decision-makers. Not surprisingly the
iter prove to be the strongest proponents of the technique. They are, after all,
one group which must regularly seek a consensus and usually has to make
isions on complex issues without adequate information.
liss Jillson s article is a progress report on a study designed to develop drug
ase policy options, explore the applicability of the Policy Delphi to questions
octal policy generally, and determine the practicability of using it on both
i “as-needed” and “on-going” basis (i.e., indefinite duration). The particiJs include researchers, administrators, and policymakers—both in the field
1 in impacted areas (e.g., police chiefs). A unique feature is the use of the
ree perspectives. In the “top-down” approach, the objectives for the next five
rs are emphasized; the “bottom-up” approach deals with factors which
■’tiol transition between various states or levels of drug use and employs a
itrix format; the “issue-oriented” approach crystallizes statements of policy
es in “should/should not” form.
fones Delphi reflects the typical consensus or Lockean oriented approach in
design to gain consensus among representatives or organizations subject to
.‘lent pressuies in their competition for limited financial resources. He uses
senior managerial and technical personnel (both military and civilian)
-resenting most departments in the four organizations. Different organizaal view-points are apparent although significant self-interest biases are not
icctable. This effort contrasts very nicely with the Kantian nature of Ludlow,
Hegelian approach of Turoff, and the mixed Kantian-Hegelian work of
on.
iness and Industry
a corporate environment Delphi fills several roles. Bell-Canada’s Lawrence
lists three: educational device for senior management, environmental
nd background material for technological planners in research laboratories,
. trading material for use with planner-counterparts in other organizations.
the corporation is large and diversified it may have the analytical staff to
General Applications:
Introduction
'■ 77• rMilll
'ISMI
IJ.
run the study and the expertise to form the panels “in house”. TRW used 140
of the “the most imaginative and creative members of TRW’s technical staff of
more than 7,000 graduate scientists and engineers”. It should be noted that in
the hierarchical environment of a business, the rate of response or participation •
tends to be higher than average. A university professor may feel no compunction about ignoring questionnaires or giving perfunctory or dilatory
dilatory answers; an
an
.
employee has stronger incentives to cooperate in a company exercise. Goldstein’s5 '
experience in this regard is echoed in numerous other cases.
'
I’he abilily io tonduct a Delphi without bringing the respondents together
physically is anotherad\-.image in the large corporation with units spread over
a wide geographical area (e.g., multinational corporations). Overseas personnel
can be drawn into a Delphi with relative ease and at minimal cost.
Day covers ffour Delphis performed by Bell-Canada’s Business Planning
Group: Educational Technology, Medical Technology, Business Information
Processing Technology, and Home Communications Services.
In cases where the corporation does not have the expertise in either the
subject of the forecast or the Delphi procedure, it may turn to a professional
consulting organization. Enzer’s article describes a Delphi on the subject of
plastics undertaken for a client by the Institute for the Future. The field of
materials for the future is a particularly difficult one for the forecaster. First,
the number of possibilities is overwhelming (e.g., tailormade plastics). Second,
■W
in a hierarchy (e.g., relevance tree) which has metasystems at the top, followed
by systems, subsystems and components, materials are close to the bottom. This
means that a given material can branch upward in many ways, i.e., it can find
use in a multitude of old and new systems by 1985. Third, each substitution of
a new material for an old one may prove sensitive to slight variations in the
relative prices, with major deviations resulting in the 1985 forecasts. These
factors strongly suggest the desirability of using at least two entirely different
methods of forecasting to provide a reasonable degree of confidence.
Nancy Goldstein gives us a hint of this in her article on a Delphi covering
steel and ferroalloy materials. She compares the Delphi results with those of a
conventional panel study conducted simultaneously. There is considerable
agreement in the forecasts. Ideally one would seek a comparison between
methods w'hich are more radically different than a conventional panel and a
Delphi. However, this particular comparison makes one startling methodologi
cal point: the conventional panel study brooks no uncertainty and no areas of
disagreement in its forecasts; it presents a definitive view and a set of con
clusions. The Delphi, on the other hand, reflects the uncertainties and high
lights the differences among its participants. It is more concerned with explor
ing minds than setting down precise recommendations (cf. Scheele, Chapter II,
C)The Delphis by Day, Enzer, and Goldstein are largely Lockean in nature.
However, the use of panels of differing backgrounds by Day has Kantian
aspects, and Goldstein suggests Hegelian overtones.
' s-iMS
’mi
1
a
'pl1
•J
I3
Harold .\. Linstone and Murray Turoff
Another use of Delphi has recently evolved in business in connection with
isk analysis.1 Il concerns the uiuvrtaintics associaied with new projects or
iivc.himcnis. Normally decisions musl be made in the absence of adc(|iiatc
nformation. The potential market for a new product is uncertain and the
levclopment costs may exceed the engineers' estimates. Marketing personnel in
in operating unit of the corporation frequently exhibit a glowing optimism
Tiich neglects to credit the competition with high intelligence or quick
eaction capability. Engineers tend to assume that the cost of a complex
noduct is a linear function (i.e., sum) of the cost of the components which
unprise it. They neglect the interactions which result in nonlinear behavior:
■he total cost is much greater than the sum of the parts. Thus the cost is grossly
underestimated. One recent study of a large number of defense-oriented
levclopment projects indicated approximately a 50 percent chance of 50
2
rcent cost overrun.
Delphi may be used to advantage to provide input to the risk analysis. The
nosl critical part of such analysis is the subjective probability distribution
assumed for the uncertainties. Delphi can serve to probe the views of personnel
. onnected with the project as well as outsiders (e.g.. corporate offices or other
mils), senior executives as well as junior engineers and scientists. The anonym
ity is particularly valuable in a highly structured environment where in
dividuals may feel constrained in expressing their own views.
Comments on Other Studies
Many applications of Delphi are carried on as integral parts of planning
projects or as staff work of a proprietary nature. Therefore, a considerable
uuount of very good Delphi work has not been published in the open literature
ji in a form adequately explaining the process used. Before proceeding to the
separate application papers it is worthwhile to note a number of Delphi studies
■f a unique nature which are not yet otherwise documented explicitly for those
interested in the technique itself.
The Delphi method has been applied extensively in the medical area. I he
nitial work by Bender et aP was largely of a straightforward forecasting
variety. However, a number of Dr. Bender’s Delphis did deal with estimating
1 See D. B. Hertz, “Risk .Analysis in Capital Investment,'' Harvard Business Review, Jan-Feb. 1964,
;> 95, and D. H. Woods, “Improving Estimates That Involve Uncertainty,'' Harvard Business Review,
Jul-/\ug. 1966, p. 91.
2S. H. Browne, “Cost Growth in Military Development and Production Programs,'' unpublished,
Dec. 1971.
3 Bender, Strack, Ebright & Von Haunalter, Delphic Study Examines Developments in Medicine,
Eutures, June 1969. George Teeling-Srnith, Medicine in the 1990's, Office of Health Economics,
England, October 1969.
General Applications:
/
Introduction
79
the necessity and desirability of potential medical research accomplishments.
And it did not take very long before the application was broadened to include
unique objectives other than future projections.
Dr. John W. Williamson, of John Hopkins University’s School of Hygiene
and Public Health, has utilized Delphi extensively for estimating historical
data. Typical questions deal with determining the incidence of a given disease 4
and the estimated rate of success in utilizing various treatment methods. This
is, of course, an area where current reporting practices do not give reliable data
owing to differing standards across the country and the effect of multiple
complications, e.g., death due to pneumonia while ill with cancer. Usually Dr.
Williamson would ask respondents for their best estimate of a number, then a
low and high value which would shock them. Also requested would be an
estimate of their confidence in the estimate and a statement whether their
estimate was based upon a particular source, such as an article they had read.
In a number of these exercises questions were asked which dealt with the results
of unpublished new clinical studies. In this manner one could observe how well
the Delphi panelists actually did on part of the exercise and utilize this insight
to gain an impression of their capability for providing answers to the rest of the
exercise.
An excellent and, perhaps, classic example of this is a study Williamson did
at the Philips Electric Corporation Plant in Eindhoven, Holland, in 1970.
Approximately 50 doctors who are involved with the company’s medical
program for the 36,000 employees participated in the Delphi. The first part of
the Delphi asked the physicians to estimate the percent oi male employees
absent from work due to sickness during differing intervals of time. The
population was further divided by young, old, blue collar, white collar. This
required sixteen estimates from each doctor. When the real data was collected
from the computer files three months later it was found that 12 of the 16
estimates were within 10 percent error and the other 4 within 30 percent error.
Of course, this represented only a small portion of the exercise as the real .
objective was to determine what effect various potential changes to the health
care program would have on the absenteeism rate. However, one could see that
the physicians involved had a good feel for the situation as it existed. Also it ;
was possible to examine how well various subgroups did, e.g., general practi- ,
tioners vs. specialists. Dr. Williamson has conducted four major studies of the
above type (involving validity checks) with a total of approximately 200
respondents over the past four years.
Professor Alan Sheldon of the Harvard Medical School, together with
Professor Curtis McLaughlin of the University of North Carolina Business
School, did a Delphi in 1970 on the Future of Medical Care. A unique feature
of this Delphi was the process of combining the events evaluated by the
respondents into scenarios in the form of typical newspaper article The
respondents were then asked to propose additions or modifications to the
i
1
I
I
1
4
*
I
Harold A. Linsionc and Murrav Turoff
scenarios and give their reaction to the scenario as a whole. This concept of
utilizing the vote on individual items to group events into scenarios classed by
such things as likelihood and/or desirability has become a standard technique.
Also with respect to scenarios it has become fairly common to provide the
respondents on a forecasting Delphi with a scenario or alternative scenarios
providing a reference point on considerations outside the scope of the Delphi
out having impact on the subject of the inquiry. 1'or example, in forecasting the
future of a given industry the respondents might be given a “pessimistic.”
“optimistic,” and “most likely' scenario on general economic conditions and
asked that their estimates for any question be based on each alternative in turn.
While there have been a number of Dclphis on the general future of medical
care, a recent Delphi by Dr. Peter Goldschmidt of the Department of Hygiene
and Public Health, Johns Hopkins University, dealt with the future of health
cate for a specific geographical entity. Ocean (:it\~.~Maryland. The problem in
health planning in this case is the tremendous influx of vacation people in the
summer months. In order to examine the future growth of the Ocean City area
and its resulting medical needs, it was fell the Delphi panel should include
individuals who resided in the area and simultaneously worked in endeavors
related to the mainstream of the local economy—recreation. 'Therefore, the
Delphi involved long-time residents, hoteliers, bar owners, real estate dealers,
and civic officials as well as the usual “experts" such as the regional planning
people from local government and industry. This widening, or broadening, of
the concept of “experts” to that of “informed" is becoming quite customary in
the application of Delphi. In this particular Delphi, Dr. Goldschmidt was able
to check the “intuition” of his respondents by comparing their estimates on
vacation populations in Ocean (lily currently with estimates he could analyti
cally infer from the processing load history of the sewage plant that serves the
aiea. As in Williamson’s case the results were quite good.
A superb example of the Delphi technique was carried out by Richard
Longhurst as a master’s thesis at (lornell University.* The Delphi attempted to
assess the impact of improved nutrition, family income, and prenatal care on
pregnancy outcome in terms of birth weight and the resulting l.Q. and
mtelleclual development of young children. The resulting output were of a
form useful for incorporation into cost-benefit analyses of government programs
io improve the nutrition of pregnant mothers and young children. 'This is, of
course, an excellent example where data exist to indicate that malnutrition in
the mother or young child has some degree of impact on the long-term
intellectual capabilities of a child. However, this evidence is not of direct
quantitative utility to the type of analyses an economist would like to perform
in evaluating a government program.
The respondent group was divided into two subpanels. One in the area of
4 Richard Longhurst, “An Economic Evaluation of Human Resources Program with Respect to
Pregnancy Outcome and Intellectual Development," M.S. thesis. Cornell University. Ithaca, N.Y.
December 1971.
General .Applications:
Introduction
intellectual development comprised experts in child psycholo^B
ment; the other, in the area of pregnancy outcomes, was compose^
pregnancy, nutrition, and medical care. The group was given a s^B
of low-income mothers in a depressed urban area as the population^
concerned with. This was a real group on which a good deal
socioeconomic status were available. In the first round the respondent
asked to sort out the relative importance of environmental components^^^B
might be manipulated by the introduction of a government program.
second round presented a set of feasible intervention programs which related
the factors brought out in the first round. They were then asked to estimate for^
each program the resulting incidence of low birth weight and the average l.Q.
score of 5-year-old children resulting from the pregnancies under the program.
'They provide the baseline data on these parameters for the target group as it
currently existed. Certain programs were estimated to reduce the incidence of
low birth weight from 15% to 10% and to raise the five-year l.Q. scores from 85
to 100 points. 'The shift in l.Q. can then be used to shift the average education
and earning power of the children when they are adults. This then is translat
able into dollar benefits that can be used to compare the merits of alternative
programs in this area. The Delphi itself involved respondents for three rounds
and questionnaires were tightly designed to take about fifteen minutes to fill
out.
I
i •
The area of trying to translate scientific knowledge into an informed judg
ment on evaluating and analyzing decision options is clearly a potential one for
the Delphi method.
.Another effort in health care planning is the work of Professor David
Gustafson at the University of Wisconsin. This work has been tied into the
Governor's Health Planning and Policy Task Force effort. One of the Delphis
Professor Gustafson conducted dealt with delineation of the current barriers to
the performance of research and development in the health services area—the
rather interesting topic of trying to clarify what the real “problem” is. The
respondents were asked to delineate barriers of three types: (1) solution
development barriers; (2) problem selection barriers; (3) evaluation barriers.
For each barrier the group developed comments, implications, and possible
reactions or corrective measures. A vote was taken on the significance of each
barrier. This was an excellent example of utilizing Delphi to try and isolate the
significant part of the problem. Very often, in planning areas, preconceptions
by one individual lead to tremendous efforts on the wrong problem. The
specification of a particular problem usually predetermines its method of
investigation and at times its conclusions.
The use of Delphi for regional planning has probably become popular
because of the feeling that there is a necessity to establish better communica
tions among many individuals with diverse backgrounds.
In this area a significant number of Delphis have been conducted by various
Canadian government agencies such as Health and Welfare, Department of
1
.1
a
I
I
I
1
1
0
if
11.
’hi . \
I .i i r l< >ii<- .on I M in i ,i\
I
II
(icnci al .Applications:
Introduction
83
Public Works, Department of the Environment, and the Postal Senice. to
name a few. Most of these arc being done by internal staff and very often they
tend to be short, focused on a very specific issue, and require a diverse
backuroud of respondents. A good example is one done in 1974 by Madhu
Agowal on “'rhe Future of (atiz.cn Participation in Planning Federal Health
Policy.” The Delphi sought to explore and delineate specific options for citizen
participation and to determine the consequences of such programs.
'There has been very active use of the Delphi in the educational establish
ment and a survey of that work may be found in an article by Judd? Curiously
almost all educational Delphis have been confined to administrative matters
operations research : Each participant could propose significant contributions
to the literature in the areas of theory, applications, and economic impact. The
group then voted on each item for importance and impact. Also gathered were
brief comments on the significance of an item and suggestions for important
areas of future research.
Michael Marien has used the Delphi process to elicit from a panel of 14
futurists the most significant books (a “hot list”) on the future. Considering the
volume of material being produced in most professional areas, these particular
A similar concept has been applied in some corporate or organizational uses
of Delphi, where the study examines historical performance or factors that have
affected the market place for a particular item. The objective is usually to focusi
on 50 to 100 key items out of hundreds of candidates so that a concise summary[
that anarchy sometimes results. There is also an entrenched bureaucracy which
feeds on well-structured procedures and questionnaires of all kinds.
of the historical perspective can be prepared for management. This is largely'
in this environment.
However, to find a clue to what may prove to be the most serious difficulty,
we must turn to the conclusion of a (non-Delphi) survey of school administra
tors conducted recently by R. Elboim-Drorb in Israel on the subject of educa
tion in the year 2000:
'The lack of creative imagination as revealed in this study, the limited number of
new alternatives and innovating ideas expressed by the subjects, and especially the
students (of school administration), are a serious sign.
In such an atmosphere Delphi can be as barren as most of the paperwork
which traditionally suffocates educational bureaucracies. When the educational
field begins to see Delphi in the deeper context discussed in Chapter 11. when it
the process of getting a group to filter out the signal of real information from
the multitude of communications or noise that may exist on a particular
complex topic. This concept is very similar to what P ofessor James Bright
tefeis to as the monitoring function in technological forecasting, and an
excellent example of a Delphi on “Events Leading to the Limitation or
Elimination of the Internal Combustion Engine” forms the basis for an exercise
in his recent book.7 The example, based upon a Delphi conducted in 1969 by a
chemical company, was, to the best of our knowledge, the first which dealt
exclusively with evaluating the past.
z\ much deeper systemic study of the past is envisioned in the “retrospective
futurology" approach which applies dynamic programming to historic societies
such as the city-state of Athens. The “hyper-sojiphisticated polling of experts”
mentioned by Wilkinson in conjunction with this> concept8 strongly suggests the
Delphi method.
starts to consider Delphi as an educational tool as well as a planning tool, then
it may be able to escape this trap.
If there is a single message in the philosophy of Singer (see p. 33). it is that
the past and the present arc often as hard to interpret, or conjecture about, as
the future. It is therefore not surprising to see the Delphi method applied to
these areas as well as to the future. A recent Delphi devoted to examining the
past is an exercise by Professor Russell Fenske of the University of Wisconsin.
This involved about twenty-five leading researchers in the field of operations
research who utilized the Delphi to review the “state of the art of industrial
5 R. C. Judd, “ 1 he Use of Delphi in Higher Education,” Technological Forecasting and Social Change,
4,173 (1973).
6 R. Elboim-Dror, ‘‘Educators’ Image of the Future," paper presented at the Third World
Future Research Conference, Bucharest, September 1972.
I
applications of Delphi are likely to become quite popular in the future.
and hardly considered as a teaching tool. It is not surprising that educationa
lists are enthusiastic about the method. I here is a high degree of participative
planning in higher education. .Authoritarianism is eschewed to such an extent
Delphi is used for several aspects of administrative planning: general goals,
curricula, campus design, and development of teacher ratings and cost-benefit
criteria. Judd describes many of the problems encountered in the use of Delphi
i
/\usdnnTexasl8^972 Bne^IntToductlon to Technology Forecasting: Concepts and Exercises, Pemaquid Press,
J W dkinson, R Bellman, and R. Garaudy, ' The Dynamic Programming of Human Systems,”
YorkS’l97a3 p^’
StUdy °f Democratic dilutions, MSS Information Corp., New
I
B. 1. The Policy Delphi
General Applications:
Policy Delphi
85
The Committee and the Delphi Process
Traditionally the approach in most organizations to the examination and
exploration of policy issues has been the committee process. Certainlf it is well
documented by a number of writers on the functioning of government organi
zations. that the committee system is a structure that evolved initially to
promote the advocacy process associated with policy analyses.2,3 The committee
approach brings people together across organizational lines in order that all
views at similar organizational levels in the whole organization may be repre
sented, and a meaningful view arrived at after the differing interests have been
adequately expressed and advocated. It is the contention here, however, that
from a pragmatic viewpoint, the committee approach in government and most
other organizations no longer functions as effectively in the realm of policy
formulation as it once may have.
j Many organizations today have become bigger, serve more functions, and
MURRAY TUROFF'
A Seer upon perceiving a flood should be the first to climb a tree.
Kahlil Gibran
introduction
The Policy Delphi was first introduced in 1969 and reported on in 1970.’ It
represented a significant departure from the understanding and application of
lite Delphi technique as practiced to that point in time. Delphi as it originally
was introduced and practiced tended to deal with technical topics and seek a
consensus among homogeneous groups of experts. The Policy Delphi, on the
other hand, seeks to generate the strongest possible opposing views on the
potential resolutions of a major policy issue. In the author’s view, a policy issutT4
is one for which there are no experts, only informed advocates and referees. An
expert or analyst may contribute a quantifiable or analytical estimation of
some effect resulting from a particular resolution of a policy issue, but it is
unlikely that a clear-cut (to all concerned) resolution of a policy issue will result
from such an analysis; in that case, the issue would cease to be one of policy. In
(he face of the policy issue, systems analysis, opeialious i (search, and other
iclalcd (list i| dines can do no more than supph a 1.1< m.il basis for advo< a< v.
I he expert becomes .in advoc ate lor elfci tivencss oi clln icn< v and must
compete with the advocates for concerned inlcresi groups within the society or
organization involved with the issue.
The Policy Delphi also rests on the premise that the decisionmaker is noT'
interested in having a group generate his decision; but rather, have an [
informed group present all the options and supporting evidence for his con
sideration. 1 he Policy Delphi is therefore a tool for the analysis of policy issues
and not a mechanism for making a decision. Generating a consensus is not the
prime objective, and the structure of the communication process as well as the
choice of the respondent group may be such as to make consensus on a
particular resolution very unlikely. In fact, in some cases the sponsor may even
request a design which inhibits consensus formulation.
•Murray I uroff,
I he Design of a Policy Delphi.
span a much wider range of complex interacting functions. Committees that
I1 truly represent all interests on an issue are often quite large and unwieldy. By
the time one has reached the point of twenty or more people constrained to
reach a view in a limited amount of time, a complete and free exchange of
views among all concerned is often too time consuming or impossible within the
scope of the allocated effort for the job.
With increasing size of organizations, the ratio of the number of people at
the top echelons to those in the lower echelons has decreased over the years,
p.u ti< ul.it ly in govci nnivnt. This implies that those at the top must spend more
lime dcvoied io day-to-day management (unctions and less time for committee
pan n i pal ion on the longer-range issues associated with policy. As a result, the
responsibility for committee participation falls more and more on lower-level
people. Individuals at the lower levels are less likely to be advocates of anything
until they have had ample time to clear it with their supervisors. This often
forces the committee into added weeks of delay whenever any new point is
made and usually results in the early or premature termination of new
considerations that might result from the advocacy process.
If an organization is top heavy a similar problem also develops. Power
becomes too diffuse and no one feels he has the authority or jurisdiction to act
as an advocate on the broader issues that usually arise at the policy level. There
are so many narrowly defined functional responsibilities that everyone is taking
care not to tread on their neighbors’ territory.
The complexity of issues today usually calls for a great deal of additional staff
to supplement the committee process. More often than not, this time or support
is not allocated to or available for committee participants. In an atmosphere of
budget cuts, belt tightening, and competition for limited funds, it may appear
/echnolo^tcal ['urfcusting and Social Change 2, No.
T he Delphi Method: Techniques and Applications, Harold A. Linstone and Murray Turoff (eds )
ISBN 0-201-04294-0; 0-201-04293-2
Copyright © 1975 by Addison-Wesley Publishing Company. Inc.. Advanced Book Program.
All rights reserved No pan of this publication mav be repr<xiuced. stored in a retrieval system, or
transmitted, in anv form or by anv means, electron.c. r.'.rer.an.ca. photocopsing. recording, or
oihervv.s.v. without the prior permission of the publisher
i
'Charles F. Schultze, “The Politics and Economics of Public Spending,” Brookings Institution,
Washington, DC., 1968.
3Numerous references to Lindblom's writings on committee processes appear in the work cited in
reference 2.
a
Ti
>
General .Applications:
Murray Turoff
.d\antageous not to advocate, not to be noticed, and especially not to be held
countable for views, promises, or positions which require effort to document
■•/Ubstantiate. In addition, in most organizations todav. we have individuals
-ho ate not familiar with many of the new decision aids coming out of
derations research and systems analyses but who have an intuitive feel for the
oniplexities of the particular business or function the organization is involved
Ae also have a good many individuals who have been trained in many of
it modern management techniques and who arc sometimes a little too
mfident that these approaches can be applied to every problem. 'Fhe lack of
lective communication between these two groups has brought about the
effectiveness of many committee exercises.
xt is the above factors, or any combinations of these factors, which have
otivated attempts to seek substitutes for the committee process. Contrary to
v. above, the earlier writings on Delphi have usuallv presented a separate but
anonical set of problems associated with committees that tend to reflect
wchological characteristics of committee processes:
J
87
1I
\ iew of the role of the Policy Delphi—a mechanism for reviving the advocacy
process in organizations through improving the effectiveness of lateral policy
formulating committees. In this way, Policy Delphis operate as precursors to
the committee activity.4
-—M
'fhe Policy Delphi, therefore, is not in any way a substitute for studies,
analyses, staff work, or the committee. It is merely an organized method for
correlating views and information pertaining to a specific policy area and for
allowing the respondents representing such views and information the opporL1}0!1)!!0
lo and assess differing viewpoints. Because the respondents are
anonymous, fears of potential repercussions and embarrassment are removed
and no single individual need commit himself publicly to a particular view
until after the alternatives have been put on the table. F; < n in those cases
where the Policy Delphi uses only the committee or sponsoring body as the
respondent group, it has the advantage of eliminating the principal bottleneck
in the committee procedure by providing a clear delineation of specific differ
ing views, thereby providing an opportunity for the committee members to
prepare their respective cases adequately.
A Policy Delphi should be able to serve anv one or any combination of the
following objectives:
■
t he domineering personality, or outspoken individual that takes over the
committee process
- i he unwillingness of individuals to take a position on an issue before all the
facts arc in or before it is known which way the majority is headed
1 he difficulty of publicly contradicting individuals in higher positions
Fhe unwillingness to abandon a position once it is publicly taken
Hie fear of bringing up an uncertain idea that might turn out to be idiotic
and result in a loss of face
Given a small committee of around ten individuals with sufficient time to
msider and explore the issues, and some assurance that the privacy of their
•spective remarks will be respected outside of the committee room, it is
lOubtful that any of the above issues would greatly inhibit the process,
lowcver, as the size of the committee increases, the time available decreases,
nd the organizational considerations listed above present themselves, the
/sychological problems will also come into play.
Delphi, however, is not a replacement for the committee process. Fhe
reposition presented here is that the Policy Delphi can be utilized to revise the
ilectiveness of the committee approach.
A Policy Delphi can be given to anywhere from ten to fiftv people as a
u .nrsor to a committee activity. Its goal in this function is once agairTnoTso
meh to obtain a consensus as to expose all the differing positions advocated
id the principal pro and con arguments for those positions. In many policy
■reas. a larger number of respondents, in the area of twenty or more, is
ommcnsurate with the number of differing interests that must often be
■nsidered in the increasingly complex issues facing organizations.
Once the Delphi has been accomplished, a small workable committee can
.lilize the results to formulate the i<-quired policy. 'Fins then is the author’s
Policy Delphi
..............
; It
i 1
if
I
• To ensure that all possible options have been put on the table for considera-!
lion
• Io estimate the impact and consequences of any particular option
• To examine and estimate the acceptability of any particular option
i
t
!
!
'Fhe ability of the Delphi technique to improve current practices for han
dling the first objective seems quite clear. Whether or not it can meet or fulfill
any portion of the other objectives probably depends on whether the design
team can distinguish the motivation of the respondents in making a particular
judgment on an option. More specifically, when a difference in judgment does
occur on an option, is it based upon uncertainty and/or lack of information
w ith respect to consequences, or is it based upon differences arm-ng the
self-interests as represented by the respondent group? If the Delphi can be
designed to make this distinction it should be able to serve these latter
objectives of examining and distinguishing consequences and acceptabilities.
Because in some cases people are not fully aware of the motivating factors
behind their views, the exposing of these factors could require fairly sophisti
cated approaches, such as multidimensional scaling.
'7
:I
i 7
I
1
I
-4
!•
The Mechanics of a Policy Delphi
A Policy Delphi is a very demanding exercise, both for the design team and for
the respondents. There are six phases that can be identified in the communica4Jerry B. Schneider, “The Policy Delphi: A Regional Planning Application,” Technological
Forecasting and Social Change 3, No. 4 (1972).
I
Murray Turoff
General Applications:
Policy Delphi
89
lion process that is taking place. These are:
(1) Formulation of the issues. What is the iissue that really should be under
consideration? How should it be stated?
(2) Exposing the options. Given the issue, what arc the policy options
available?
(3) Determining initial positions on the issues. W hich are the
ones everyone
already agrees upon and which are the unimportant ones to be discarded?
Which are the ones exhibiting disagreement among the respondents?
(•I) Exploring and obtaining the reasons ffor disagreements. What underlying
assumptions, views, or facts arc being used by. the
— individuals
....... to support their
respective positions?
(5) Evaluating the underlying reasons. How does the group view the separate
arguments used to defend various positions and how do they compare to one
another on a relative basis?
(6) Reevaluating the options. Reevaluation is based upon the views of the
underlying “evidence” and the assessment of its relevance to each position
taken.
In principle the above process would require five rounds in a papcr-andpencil Delphi procedure. However, in practice most Delphic
)is on policy try to
maintain a three- or four-round limit byy utilizing
.........................
the following procedures: (1)
the monitor team devoting a considerable amount of time to carefully preforinulating the obvious issues; (2) seeding the list with an initial range of options
but allowing for the respondents to add to the lists; (3) asking for positions on
an item and underlying assumptions in the first round.
W ith the above simplifications it is possible to limit the process to three
ounds. However, new material raised by the respondents will not get the same
complete treatment as the initial topics put forth by the monitor team. Still,
'•ciy successful Delphis have been carried out \yjthin a three-round format.
Ilirnately, however, the best vehicle for a Policy Delphi is a computerized
version of the process in which the round structure disappears and each of these
phases for a given issue is carried through in a continuous process.5
Il is also necessary on a Policy Delphi that informed people representative of
the many sides of the issues under examination are chosen as participants.
1 hese individuals will not be willing to spend time educating the design team,
)y way of the Delphi, on the subject matter of concern. The respondents must
gain the feeling that the monitors of the exercise understand the subject well
•nough to recognize the implications of their abbreviated comments. 'Pherefore,
he initial design must ensure that all the “obvious” questions and subissues
have been included and that the respondent is being asked to supply the more
.ubtle aspects of the problem.
In some instances, the respondent group may overconcentrate its efforts on
]^.^urray rur°ff, “Delphi Conferencing,” Ttchnological Fortcasling and Social Change, 3, No. 2
I
some issues to the detriment of the consideration of others. This may occur
because the respondent group finally obtained was not as diversified as the total
scope of the exercise required it should be. With proper knowledge of the
subject material, the design team can stimulate consideration of the neglected
issues by interjecting comments in the summaries for consideration by the
group. It is a matter of the integrity of the design team to use this privilege
| sparingly to stimulate consideration of all sides of an issue and not to sway the
i respondent group toward one particular resolution of an issue. If, however, the
respondent team is as diversified as required by the material, there should be
ho need to engage in this practice.
’
A Policy Delphi deals largely with statements, arguments, comments, and
discussion. To establish some means of evaluating the ideas expressed by the
1 respondent group, rating scales must be established for such items as the
relative importance, desirability, confidence, and feasibility of various policies
and issues. Furthermore, these scales must be carefully defined so that there is
some reasonable degree of assurance that the individual respondents make
compatible distinctions between concepts such as “very important” and
“important.” This is further complicated by the fact that many of the respon
dents may not have to think through their answers in order to remain
consistent in answering different parts of the questionnaire.
1 he Delphi technique is not just another polling scheme, and the practices
that are standard in polling should not be transferred to Delphi practice
without close scrutiny of their applicability. Consider, for example, a poll of
different groups in an organization asking for their budget projections over the
next five years. This is a comparatively straightforward request which does not
ask any one group to place itself in context or to worry about consistency with
other groups in the organization. A Delphi on the same subject would ask each
group to make projections for every group’s budget and, in addition, to project
separately a feasible total budget for the organization as a whole.
T’he normal budget process in an organization is essentially a poll. A few
research laboratories have in recent years attempted a budget review process
via the Delphi mode, but unfortunately these are never reported in the
literature because of the proprietary nature of the subject material. In prin
ciple, it would appear that the Delphi offers more opportunity for people to
support budget items outside of their current management function and often
to obtain a better appreciation of the budget trade-offs that have to be made.
LThere are many different voting scales that have been utilized on policy type
Delphis; however, there are four scales, or voting dimensions, that seem to
represent the minimum information that must be obtained if an adequate
evaluation is to take place. On the resolutions to a policy issue it is usually
necessary to assess both desirability and feasibility. One will usually find a
significant number of items which are rated desirable and unfeasible or
undesirable and feasible. I hese types of items will usually induce a good deal of
discussion among the respondents and may lead to the generation of new
if
I
r
|
J
■‘I
I
■a
I
II
■
|
::'R
■i
i ■
A
i.
i;
ptions.
Murray Turoff
1 he underlying assumptions or supporting arguments arc usually
General Applications:
Policy Delphi
Possibly Feasible
some indication this is implementable
some R&D still required
further consideration or preparation to be given to politi
cal or public reaction
Possible Unfeasible
some indication this is unworkable
significant unanswered questions
Definitely Unfeasible
all indications are negative
unworkable
cannot be implemented
valuated with respect to importance and validity or confidence. In this case a
>crson may think an invalid item is important (because others believe it to be
ue) or that a true item is rather unimportant. I.t is usually unwise to attempt
) ask for a vote on more than two dimensions of any item., 1 lowever, if one has
<iablished a significant subset of items utilizing these scales then further
uestions can be introduced focusing on the significant subset. For example,
here is the possibility of taking desirable options and asking the probability for
ach, given certain actions are taken.
Typical examples of these scales follow. Note that no neutral answer, is
llowcd other than No Judgment (which is always allowed on any question). A
cutral position offers very little information in policy debates and it is usually
.•sirable to force the respondent to think the issue out to a point where he can
akc a'nonneutral stance. In other words, the lack of a neutral point promotes a
\’ery Important
Important
1 Jesirable
1 'mlcsirable
'cry Undesirable
will have a positive effect and little or no negative effect
beneficial
justifiable as a by-product or in conjunction with other
items
will have a negative effect
harmful
may be justified only as a by-product of a very desirable
item, not justified as a by-product of a desirable item
will have a major negative effect
extremely harmful
not justifiable
Slightly Important
Unimportant
no hindrance to implementation
no R&D required
no political roadblocks
acceptable to the public
a most relevant point
first-order priority
has direct bearing on major issues
must be resolved, dealt with, or treated
is relevant to the issue
second-order priority
significant impact but not until other items are treated
does not have to be fully resolved
insignificantly relevant
third-order priority
has little importance
not a determining factor to major issue
i
■
■
1
! |
I
1
I
no priority
no relevance
no measurable effect
should be dropped as an item to consider
Confidence (In Validity of Argument or Premise)
Certain
Feasibility (Practicality)
Definitely Feasible
1'
Importance (Priority or Relevance)
desirability (Effectiveness or Benefits)
Will have a positive effect and little or no negative effect
extremely beneficial
justifiable on its own merit
si
i -i
.cbale which is in line with developing pros and cons as one primary objectivc.
'his design choice has sometimes upset those who feci consensus is the only
alid Delphi objective.
very Desirable
91
Reliable
low risk of being wrong
decision based upon this will not be wrong because of this
“fact”
most inferences drawn from this will be true
some risk of being wrong
willing to make a decision based on this but recognizing
some chance of error
some incorrect inferences can be drawn
'i
i
Murray Turoff
Risky
substantial risk of being wrong
not willing to make a decision based on this alone
many incorrect inferences can be drawn
Unreliable
great risk of being wrong
of no use as a decision basis
General Applications:
•
•
93
__
_
The number of professionals acting as the design-monitor team must be at
leasLJXvo so that one can check the other. Ideally, one sshould
.....................
be knowledgeable in the problem at hand (but not precommitted) and the other should
.
have editorial talents.
month or more is needed to develop the first-round questionnaire. In
addition to the questionnaire, a factual summary of background material is
usually supplied, and in some cases single or multiple sets of scenarios
specifying certain items the respondents are to assume as given are provided
for the purpose of evaluating the issues. Typically these scenarios deal with
future economic conditions such as the rate of inflation. Sometimes it is more >
he first and foremost problem in conducting
conducting a Policv Delphi occurs with
lhe tnittal steps tn the process. If the respondents feel strongly about the issues
and this should be the case, they will generate a large amount of written
material. If they are |provided
’ ’ a certain number of items to deal with on the
first round then each of them will make approximately the same number of
appropriate to introduce a set of alternative assumptions making up
scenarios and let the respondents form a group scenario by voting on the
wtillcn comments ot additions m icspoiisc. I Ik sr iimsl be absli.H (cd < .ircftillv
• iiid diiplit a lions among the irspondrnts rlimin.if. <|. < )n (he avci.igr, (he
juiiicn material in die (|iicsiloiin.m<• |ol lhe sciond lound will be five to Icq
times that of the first round.
\ alidiiv <>f each.
• I ..u h questionnaire should be pretested on coworkers who have not been
involvt <1 iu the design. There is a very high probability that this will identify
items that arc stated in a confusing manner.
• l ake care to avoid compound statements to be voted upon. The question “if
After the votes are taken on tthe
‘
second round, the material should be
rearranged by the average vote on the third round. In other words, t.
. referring’to
the preceding scales, the options should be reordered bv Desirabilit
-- . . —-- ----- ...ty and the
supporting arguments reordered for each option
(
’
’by ’Importance. When the
votes are in, the resulting summary for the third round should clearly poi
point out
which items exhibited polarized distributions, which
.
ones exhibited a flat
distribution across the whole range, skewed distributions,
distributions. or on which items
.1 and 11 arc true’’ should be broken into two separate items. The exception is
statements of the form “if A then B,” which are quite useful in some
situations.
• The respondents, if new to Delphi, will respond with compound and some
times lengthy comments. Therefore it is a good idea to show them some
examples of the form you would like comments to take, in terms of being
only a very small sample of the respondents were able to make a judgment. I'or
short, specific, and singular in nature.
• If there is a trade-off between the ease of summarizing the results and the
ease of the respondents in providing the answers and understanding the
obese items, additional comments should be solicited. If possible, the revote
should be put off until a fourth round when everyone can see the additional
remarks. In a three-round exercise a revote is taken on the third round.
In many cases it may be desirable to keep track of certain subgroups making
•
up the respondent group as a whole. 1 his provides a mechanism to check
whether polarized views reflect the affiliations or the backgrounds of the
results, the choice should always favor the respondent.
The respondents should be allowed to suggest changes in the wording of
items which should then be introduced as new items. Experience has shown
that the vote on a policy item is very sensitive to wording. Because of this
property, the material on Policy Delphi can mushroom in size and represents
considerably more effort than the traditional forecasting Delphi oriented to
icspondenis. Depending on the application, this information can be fed back to
the group. Schneider in his article on Policy Delphis proposed a very concise
'Measure of Polarization" among the subgroups, l ake all two-by-lwo com
largely quantitative response after the first round.
• When asking for revotes on an item, the individual rt"'' >ndent should be
shown his original vote. 'The respondent should also be provided two copies
binations of subgroups and add the absolute value difference of the average
isn
vote on a given item. I his sum of first differences is now an index which
provides an appropriate ranking of the degree to which differences exist for
of the questionnaire so that he may retain one for later reference or to do t
rough work. He can type his answers on the other copy if he is concerned
each item relative to the group of items as a whole. I he same measure may be
applied to each individual who voted on the item when a subgroup breakdown
is not appropriate. Note that in opposition to average and standard deviation
this measure is a strong function of the number who voted when applied on an
individual basis.
Some additional guidelines on carrying out the Policv Delphi process are as
follows:
l he Policy Delphi
with security. On Policy Delphis security can be a problem with respect to
convincing the respondents that it will be maintained. The design team
should sc£^ip_a,pi;ot:edure where they themselves cannot identify the returns! '
•
i!
i-
with the individuals involved.
The respondents must be convinced that they are participating in an exercise
which involves a peer group. Therefore it is usually necessary on the letter of
I
II
.J
I
I
1
fl
4
Murray Turoff
invitation to indicate the types of backgrounds reflected in the participant \
gfiptip. In some cases, a list of the respondents involved can be provided if
there is no other effective way to convince the group of the significance of the ■
exercise.
As can be seen, there are many things to be considered in running a Policy
Delphi, or any other Delphi for that matter. 1 he Delphi concept seems so
simple that many people have thought it an easy thing to do. Consequently II ‘
there have probably been more poorly done Delphis than ones that have been |
well done.
One additional aspect of the Policy Delphi which usually argues for four or
mure rounds arises in the situation where the respondents feel very strongly
about their respective views. In such a case they sometimes have an attitude
where they cannot imagine that there arc rational and intelligent people who
hold a contrary view. Even with a vote on the first round on a given issue, the
reaction of this type of respondent to the vote presented on the second round is
i hat the individuals holding the opposite view to his just don't understand the
problem completely. A few simple comments will clear up their ignorance. It is
only until the third round comes back that this type of respondent feels the
shock resulting from a realization that the other side also feels it has some valid
points to be made. Therefore, it is only at the third round that this type of
respondent begins to put a great deal of careful effort into the points he is
making and to consider more carefully what the other side is saying. The
material generated out of this type of process could have a significant impact
on the group views if carried back in a fourth round.
I he selection of respondents is one of the most difficult tasks. However, this
problem applies to any committee or study effort. The sponsor is likely to have
a certain candidates in mind. The design team should try to structure the
problem in order to get a comprehensive coverage of the topic. It is also a good
idea to mix in a couple of lateral thinkers and devil’s-advocate types, just on a
matter of general principle—i.e. those individuals who always manage to come
up with the unexpected. \
It is possible on a Policy Delphi to observe two very different phenomena
taking place. One is when the exercise starts with disagreement on a topic and
ends with agreement. This can be very useful to those sponsoring the study if it
does occur, but, as has been said, is not a necessary result. Anol her process is to
start with agreement on a topic and end with disagreement. In a sense this can
be viewed as an educational pjoeess taking place among the respondents who
suddenly realize, as a result of the process, that the issue is not as clear-cut or
simple as they may have thought. Unfortunately, to this point in time there has
not been sufficient exploration of the use of the Delphi technique as an
educational process. Schneidei' also discusses this point. As he pointed out,
Delphi could be used bv a planning agenev to interface more effectively with
representatives of the community and serve an educational function for both
group
General Applications:
The Policy Delphi
95
Another unexplored use of the Policy Delphi is the investigation of the
performance of past policy actions. Too many organizations do not have an
appropriate mechanism for taking stock of what they have accomplished.
Understanding of what has occurred is often lacking and can lead to future
mistakes in policy formulation.
• -J
I
1
Examples of Policy Delphis
One of the first Delphis that bordered on being policy oriented was an exercise
undertaken in 1968 by the National Industrial Conference Board. It was titled
“An Experimental Public Affairs Forecast.” It involved 70 people representing
the following areas of expertise:
Economy, Business, and Labor
Science, Technology, and Change
Government, Law, and Politics
Resources
Education and Training
Communications
Culture, Family, and Behavior
International Security
I
17 J,
'
6‘
f
9
5
8
12
4
The vast majority had titles ofchief executive or director. All were considered
by the Conference Board to be distinguished in their field.
The overall objective of the study was to obtain a rank ordered list of
National Priorities or /Xreas of Major Concern to the Nation, areas which could
create major public problems in the seventies and eighties and should receive
attention by U. S. leadership. The top ten in that list in order of priority were:
(1) division in U. S. society; (2) international affairs; (3) education; (4) urban
areas; (5) law and order; (6) science, technology, management of change; (7)
economy; (8) resources; (9) values; (10) population.
The Delphi was completed before the presidential campaign and one may
note a degree of correspondence between the priorities set by this exercise and
the Republican campaign themes. While the Delphi dealt with policy con
siderations, it was largely oriented to putting the pieces of the problem together
by collecting information and views from a diverse set of respondents. Therefore
it largely reflected a Kantian-type exercise. The bulk of the material produced
was a collection of commentaries on the problem areas with some estimate of
when particular problems would arise. Each item was handled in terms of the
following categories of information:
• description of the item
• di sc riplion of public reaction to the item
• beginning date of maximum impact on U. S.
• intensity of impact on U. S.
• opportunity for leadership to change the expected
1
I -
1
1
I
'■'h
I
I
■1 i
!•A'1
al
Murray Turoff
1 he Delphi appeared to be quite adequate in meeting the needs of its sponsors;
however, the exercise has never been described in the literature so one can only
infer this from the final report, which unfortunately did not receive public
distribution beyond those immediately involved and some individuals working
in the Delphi area at that time. One major fault of the study was the decision
made by the staff people not to abstract the comments of the panelists but to
icldin the full text. In part this decision was probably influenced by the
distinguished nature of the respondent group. The result was a very large
volume of material which is a little painful to wade through to gather the
particular nuggets of wisdom that were produced. Qne_gpalof a Delphi
Delphi desjign
desigi
should, therefore, always be to obtain a filtering of the essential from the
superfluous.
The next Delphi in the policy area was one conducted by Emory Curtis as a
consultant to San Mateo County in California. This one involved around 80
immunity people representative of the many different constituent grot^Z
making up the public body. A great deal of effort went into obtaining a
broad-based distribution of respondents. They were provided a large number of
policy options dealing with the structure and functions of the county govern
ment, and asked to vote oh these for relative agreement on a seven-point
gicement scale. Additional items were added as a result of the first round.
However, the one shortcoming of this exercise was the lack of exploration of the
factors underlying disagreement when it did occur. The exercise produced some
new options and exhibited consensus where it occurred but provided no
mechanism for effectively resolving disagreement. However, it represented one
of the first attempts to) use the Delphi in policy areas related to community
government.
In 1970 a Delphi was conducted by the Office of Emergency Preparadness
and the Rand Corporation on the subject of Civil Defense Policy. This Delphi
introduced a number of unique features. It exhibited the structure of a
Hegelian inquiring system as opposed to the earlier Lockean- and Kantiantype Delphis. First, it recognized in the design that strong disagreement already
existed on a number of the issued involved. For a number of items therespondents were asked to choose sides by circling “could/could not,” “should/
should not,” these being choices in the wording of the items. They were also
asked to develop the strongest arguments on the various sides of a given issue.
1 he sponsor was not interested in having the group make a decision for him,
but in having the group develop, compare, and evaluate the best possible
arguments on each side of an issue.
—
The details of this exercise are well documented in the literature.1 As typical
of these types of Delphis, the respondents generated about eight times the
amount of material they were initially given on the first round, which con
tained some seventy items for evaluation. Basically policy options were/)
General Applications:
The Policy Delphi
97 f
- devaluated on scales
:
of desirability and feasibility, while supporting points were
- ----- - w
evaluated on importance or validity,
this Delphi was really
i
validity. This
the first to
,
incorporate a structured debating-type format, which appears to be the useful
approach for the exploration of policy issues.
In 1970 Professor J. B. Schneider at the University of Washington adopted
thee same approach to the exploration of transportation planning as it applied ' •' j/'0
to highway
1
development in the Seattle area. His report of the exercise4 is an
excellent
techniques to urban nlannin?
planning nrnhlems,
problems.
........... 1 example of applying these techniciues
He also contributed some very useful observations on the methodology for
handling disagreement in that particular context.
Following the same line
1
of development, Joel Goodman of the College of .■
Marine Studies, University of- —
Delaware,
conducted a policy-type Delphi on the
Coastal Zone Land Use Planning Issue. This involved a large number of people •
representing government business, public groups, and specialists. This exercise
f
was done in 1971 and 1972. It converged the following types of items into
different sections of the questionnaires: respondent characteristics; respondent ■fl
attitudes; arguments pro and con; general policy and budget items; specific
policy issues; specific programs; strategic issues.
Some sample questions from that exercise follow:
I
i
1
• As an individual, list in order of priority your 5 principal concerns with respect to the
way in which the coastal zone is developing.
(a) Health hazard
(f) Not enough housing
(b) Unsightly buildings
(g) Not enough boating facilities
(c) Dirty water (visual appearance)
(h) Not enough camp ground
(d) Too much land going to waste
(i) Beaches too narrow
(e) 'Foo crowded
(j) Too many fisherman
(k) Other
■■I
• Why arc you as an individual concerned with pollution in the coastal zone and
its effects upon the marine environment? Check up to three responses and signify
relative importance by numbering principal reason as “1.”
1
"1
j al
(a) biological danger
(b) potential loss of recreational opportunity, i.e., swimming, boating, etc.
(c) potential loss of aesthetic values, i.e., vistas, landscape, etc.
(d) potential loss of income or revenues
a
(e) community involvement
(f) other (specify)
I-
a
93
Murray Turoff
General Applications:
The Policy Delphi
Indicate by check mark who should assume responsibility for establishing:
Quality limits Use patterns
for coastal
for coastal
zone environ zone shore
ment
lands
(1)
(2)
Use patterns
for coastal
zone submerged
lands
(3)
Use patterns
for coastal
waters
(4)
a. State
b. County
c. Local Com
munity
99
A Delphi study conducted by the Federal Department of Public Works in
Canada illustrates the incorporation of policy options into an essentially
non-Policy Delphi. The department’s major role is providing accommodation
for federal civil servants, and the Delphi was undertaken as part of a model for
forecasting government employment with the purpose of determining future
accommodation needs. But the department’s mandate extends beyond simply
providing buildings to house federal employees. It is concerned with the total
work environment of the civil service.
Consequently, the Public Works Delphi also explored the existing procedure
for space allocation, which at present is based on the average salary of all
^employees using that space, and asked respondents to comment on f ... *
that process.
In the first round, after reviewing the present process, the respondents
j were
asked to list what they felt were the strengths and weaknesses in the
process,
and asked to suggest possible options for change. In the second round -these
d. Other
(Specify)
options were voted upon according to Desirability and Feasibility, keeping in
mind that a particular option could be desirable and unfeasible at the same
time, or vice versa. Some examples of suggestions for change according to
Desirability were:
If standards for the quality of the marine environment are to be maintained, then the
authority and responsibility for regulation should be vested in:
• formula approaches, if used, must reflect the quality of space as well as thfc
quantity
• relate space to function not salary
• more emphasis on multipurpose facilities
• DPW should lead the way in educating agencies in new building concepts
•SELECT ONE
A state agency within the
executive branch
b. A county agency
Individual municipalities
cl. Criteria established by
slate; regulation by
municipalities and counties
e. A new organization respon
sible to
with elected/appointed of
ficials (Eill in blank and
select one or the other means
of acquiring the officials.)
\ large number of the current Delphis ha\e started to incorporate policy
ues even when that was not the primary concern. Such issues have the
sychological advantage of making the exercise of more interest to the respon
ds. The policy orientation has been introduced in some different ways,
‘stead of asking individuals to extrapolate data into the future in terms of
.cii best estimate of what they think will occur, a policy approach would be to
what would be a desirable and possible extrapolation as well as an
■'desirable and possible extrapolation. Based upon those estimates one can ask
i»at are the factors that could make the curve go one way or the other.
I he Delphi also looked at possible parameters for i
measuring building
performance that would go beyond the usual cost/benefit measures, such as the
ratio of rentable square feet to total square feet. Specific suggestions or concepts
for consideration fell into the following categories:
• psychological and motivational impact on employees
• transportation to building
• aesthetic value of building
• community and public service
• energy and environment
The respondents were asked to vote on f
the Desirability and Feasibility of
specific suggestions and to suggest ways in whichi some of these concepts could
be measured..
Public Works then used the Delphi exercise not only to fulfill its immediate
objective of forecasting federal government employment but also to explore
policy options relating to its mandate of fulfilling broader social, economic, and
environmental objectives.
Another excellent example of a T
Delphi mixing policy issues with future
forecasts was one done by the Canadian Department of Health and Welfare
i on
ii
'W
I
1
I‘I
ilt
II
-
jp
if
Murray Turoff
ie Future of Genetic Counseling Services in Canada. The exercise involved
respondents ranging hunt research genetic-jsts to public health
otke-rs. l lu- design was well balanced between ''tec hnical issues" of what was
csscble at what point in time and "policy issues" o( who could, would or
nould do what. In tins latter area, the issue of a genetic registry and its
Hernial abuses as well as uses were explored. The Delphi used the same sort of
ales that were mentioned earlier. However, it tended to redefine a scale such
importance’ for each question it was used on. This had the merit of
inimizing what the respondent had to remember, since each question was
Igely self-contained. It also minimized the chance of confusion by placing the
.ale within the context of the particular question. Furthermore, it allowed
ore variety in the sequencing of questions. Most other designs, by grouping
leslions of a given type under one explanation, can produce a feeling of
aotony as the respondent goes through the exercise.
he Problems of a Policy Delphi
<-• have already mentioned the danger that a Policy Delphi can be misinipreted as a decisionmaking tool as opposed to a decision-analysis tool
cryone at heart is a decisionmaker, or wishes to be, and it is all too easy on
part of the designer to appeal to this unrequited desire. It should be a
alter of intellectual honesty for designers to make clear just what the objective
die exercise is. If we have a problem in organizations today, especially
l'.urnmental ones, it is that the responsibility for a given decision is not clearly
used on one individual. z\ decision should be made by one individual, and
i ole of the Policy Delphi and other tools is to provide the best possible
:urmation and ensure that all the options are on the table. To do this the
*lphi must explore dissension. Both Dalkey and Helmer in the early writings
Delphi expressed the need to establish clearly the existent basis for observed
tension. However, this implies a good deal more work for the design team
J lias often been neglected in the majority of the early exercises. When^a"
mg minority view exists and is not explored, the dissenters will often drop
u. leading to an “artificial” consensus on the final product.
Dnce a Policy Delphi has been started, there is no way. to guarantee a
•cific outcome if it is to be an honest exercise. This is something the sponsor
ist be well aware of. Occasionally a sponsor, particularly in a policy exercise,
i desire that the group not reach a consensus on any particular option,
tile it is consistent with the objective of a Policy Delphi to choose a
spondent group such that a consensus is unlikely to occur, it can never be
iianteed that it will not be a result. However, there is a fine line between
Iphi as an analysis tool and Delphi as an educational or persuasion device. It
possible to consider using a Delphi to educate at least a part of a respondent
General Applications:
■4
The Policy Delphi
101
group on options they may not be aware of. Unfortunately, very little work has
been done on (he use of Delphi in an educational mtxlc even though most
designers would agree that educational processes take place in most exercises.
A Polic y Delphi is a forum for ideas. In opening up the options for review,
items may arise which can be disconcerting to members of the group. If a
sensitive area is under review and an attempt has been made to have diverse
representation in the group, then premature leakage of the results can occur. In
such a case, individuals outside the exercise may misinterpret what is taking
place. Phis problem of lifting items out of context occurs all the time in the
committee process. A workable approach to this problem in the Delphi process
is to incorporate members of the press into the respondent group when dealing
with major public policy items.
As with any policy process, there are imany ways to abuse the use of the
Policy Delphi: the manner in which comments are edited, the neglect of items,
the organization of the results. However, such a process is a rather dangerous
game and not likely to go uninoticed by some segment of the respondents. There
are very few greater wraths than that
--------of1 a respondent who discovers himself to
be engaged in a Ibiased
'
exercise. Furthermore, Delphi has reached the point7
where there is no longer any/ excuse on a professional basis for making many of
the mistakes found in earlier exercises. The person seeking to undertake a 1
Delphi today should be reasonably familiar with what has taken place in the
field.
iI
I
'.i
i:
R-SHOO
06870
I
fes
-O
.'
3, 2. Delphi Inquiries and Knowledge Utilization
General Applications:
JOHN LUDLOW
researchers to the communities which are to benefit from the research. One
approach toward this objective was to include on the panels—on the same basis
as the researchers—people who were believed to be influential in the political
processes through which regional planning is accomplished. Their knowledge of
the issues and the region was beneficial to the deliberations, but more impor
tantly, their participation was judged to be an effective way of communicating
information to regional planners and decisionmakers.
'Two of the three panels were made up of researchers who were designated as
technicians and behaviorists. The third group was made up of concerned
citizens who were designated as decisionmakers. In addition to forecasting, the
method was used in several other roles involving the quantification of subjective
judgments. The exercises were designed to be progressive and cumulative, with
an emphasis on an orderly development of informed judgments.
The Delphi inquiries were one of several Michigan Sea Grant projects
related to the general task of transmitting new knowledge to people and
organizations in a way that results in effective use. Respondents in these
exercises—a group with exceptional qualifications—served as the primary
resource in evaluating the methodology.
The technical panel was composed of thirty-three individuals whose expertise
was primarily in the physical sciences and who were divided about equaliy
between Sea Grant researchers and faculty, graduate students, and others in
the School of Engineering. A second panel included Sea Grant researchers who
were not selected for the technical panel. Generally their academic
backgrounds and interests were oriented more to the behavioral sciences, and
for this reason they were labeled behaviorists. They represented a wide range of
ages, academic disciplines, and university schools and laboratories. Participants
for the third panel were randomly selected from groups of Grand Traverse Bay
area residents believed to be influential in the following fields: civics, business,
planning, politics, natural resources, government, education.
The names associated with the panels, although somewhat arbitrary, arc
reasonably consistent with the roles each group would be expected to play in
planning the management of regional water resources. The technical panel
operated independently of the other two panels and its output was fed into the
deliberations of two broader-based panels, which operated independently in the
earlier rounds and as a combined panel in the final round. The nature of their
participation is summarized in Table 1.
In order to provide continuity, a person’s judgments on the previous round
were used whenever he or she could not respond on a particular round.
Several significant modifications and refinements in the basic Delphi
methodology were tested in the Michigan Delphi inquiries. These changes were
motivated by the perceived threat of a manipulated consensus, the desire for
constrained or conditional judgments, and recognition of desirable aspects of
interpersonal methods not obtainable using the Delphi technique exclusively.
Ehe concept of informed judgments as contrasted with expert opinion provided
>
Introduction
The development.of methods to obtain, refine, and communicate the informed
judgments of knowledgeable people is one of the most crucial problems in
planning and decisionmaking. The task is particularly challenging in the
Michigan Sea Grant Program, which emphasizes a svstems approach by a
multidisciplinary group of researchers. Some of these researchers arc experts in
xtremely specialized areas, representing a wide range of technical, economic.
■. ial, legal, and political disciplines.
I’rom its inception the general goal of the Michigan Sea Gram Program 1 has
>ccn to provide the common management effort necessary to develop and bring
uj bear university expertise on short-and long-term resources management
problems in the Great Lakes. I he major approach of the program has been the
ievclopment of basic information and predictive models for resolution of
: esource problems, followed by applications and/or demonstrations of such
mformation and models to appropriate agencies and groups. Over 120 research
.nd faculty personnel from practically every major school or college in the
university arc presently active in the program. Research and planning groups
cpresenting federal, state, and local government agencies, industry, and conerned citizen groups arc also part of the problem-solving team.
I he Grand Traverse Bay watershed region was selected as the focus of pilot
dot is to develop research and planning methodologies that will be applicable
: dealing with problems and opportunities of all the G real Lakes, and in
• trlicular Lake Michigan. In the area of finding mechanisms to improve the
•ordination of the Sea Grant effort al the University of Michigan it was
<•< ided to investigate the polcnli.il foi utilizing the Delphi technique.
1 he Michigan Sea Grant Delphi inquiries* were designed Io obtain and
.•fine an interdisciplinary group of researchers' judgments about issues and
evclopments that should be considered when planning for intelligent
management of the water resources of the Great Lakes.
.an important objective of the exercises was to convey the judgments of the
' I he term “Sea Grant Program” was derived from the National Sea Grant College and Program
•ct, whose intent was to involve the nation’s academic community in the practical problems and
pportunities of the marine environment, including the Great Lakes.
• 1 he tvrrn “Delphi inquiry” was propounded by Turoff and refers to the complete Delphi
•roccss. He observed that any particular Delphi design can be characterized in. terms of the
'ujuiring systems” specified in Churchman’s writings. See reference 1 at the end of this article.
• he Delphi Method: Techniques and Applications. Harold A. Linstone and Murray Turoff (eds.)
BN 0-201-04294-0; 0-201-04293-2
opyright
1975 by Addison-Wesley Publishing Company, Inc., Advanced Book Program.
• I rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or
■ .msmitted, in
<•! t ■.
mr.!..-, '
■ .•
al ph'>• >•pvinu. reconline. or
Delphi Inquiries and Knowledge Utilization
103
I
I ?l
J
I
I
I
I
j
i
i
-r~"" *
John Ludlow
General Applications:
Table 1
Participation in Michigan’s Sea Grant Delphi Probes
Activity
Technical
Panelists
Behaviorists
Decision
makers
Contact established with
Delphi administrator
33
16
22
Unavailable after the
start of the Delphi probe
6
0
3
Written comments and
evaluations made on at
least one round
28
11
21
Written comments and
evaluations made on 3 or
more rounds
14
6
9
Written comments and
evaluations on final
round
20
9
11
Written evaluation of
methodology or evalua
tion interview
29
12
16
I
I
Important Developments and Requisite Technology
I.
i
Outline of the Procedures: Social, Political, and Economic Trends
I he portion of the Delphi inquiry concerned with social, political, and
economic trends was designed to provide respondents on the broader-based
panels with some basic reference points in making subsequent judgments
/garding future social and technical developments.
I he information package for round one presented the trends for eight
measures which have commonly been used to indicate the social and economic
development of a region. Curves were plotted from 1950 to 1970 taking
advantage of the 1970 census and the standardized enumeration procedures of
ic Bureau of the Census. Panel members were asked to extend the curves
hiough 1990 and to indicate the numerical values for 1980 and 1990 [2].
In the second round, curves representing the medians and interquartile
mges were prov.ded for the panelists, as well as pertinent comments submitted
y respondents on the previous round. Panelists were asked to reconsider their
estimates, and if any of the new estimates were outside the designated consensus
mge for the previous round they were asked to support their position briefly.
105
On this round the graphs of three additional statistical measures were intro
duced for consideration. A cumulative summary of the group response was
provioed in the information package for round three to serve as background
information for other panel deliberations.
)anels; it also provided an <opportunity to exploit an inherent characteristic of
i he method—to inform during the process
p
of soliciting judgments.
—
7.
Delphi Inquiries and Knowledge Utilization
*
i
rhe Delphi method has had its greatest application and acceptance as a means
of compiling a list of future technical events or developments and collecting <
subjective judgments regarding them. In the Michigan inquiries social, politi
cal, and economic developments were also solicited and evaluated so that
panelists would be encouraged to consider all environments in making judg
ments regarding water quality, waste-water treatment systems, and research
priorities.
The initial evaluation matrix for the technical panel did not present a list of
potential developments, something which is usually done in order to facilitate
participation and generate additional items. It was believed '.hat this unstruc
tured approach would result in a wider range of suggestions; however, the
information feedback of the second round did include—in addition tj> items
suggested by respondents—thirteen events that were taken from Delphi exer
cises conducted at Rand and the Institute for the Future. These events covered
areas considered by the researcher to be of interest to the panel and were also
good examples of how developments should be specified to avoid ambiguity,
particularly with respect to occurrence or nonoccurrence.
L I
The evaluation matrix for the third round provided the respondent with his
estimates for the second round
and1 a summary of the group’s response.
-------- ---Comments submitted by respondents were: also provided, as were the median
estimates for technical and economic feasibility if they differed significantly.
The evaluation matrix for the third round was designed so that a panel
member could easily determine if his reassessed estimates for a specific deve-;
lopment were outside the group’s consensus range—arbitrarily identified as the
groups median 2o percent and 75 percent estimates. If a respondent’s latest
estimate was outside the consensus range for the previous round he was asked to
support this "extreme” position briefly.
——*
The evaluation matrix for the fourth round presented a more comprehensive
summary of the previous round than had been provided up to this point m the
exercises. Statistical summaries were presented not only for all the respondents
but also for those who rated their competence relatively high and for those in
the latter group who indicated a familiarity with the Grand Traverse Bay area.
In addition, the persons arguing for an earlier or later probability date than
that indicated as the consensus were identified by a number which correlated to
a list of biographical sketches.
On the final round of the technical-panel exercises, respondents were also
asked to make specific conditional probability estimates for pairs of event* that
4
’•'J
Bli
"11
a?
John Ludlow
panel members had suggested were closely related. First they were to consider
the effects of the occurrence of the conditioning event and then the effects of
the nonoccurrence of the conditioning event (see Fig. 1). One of the objectives
of this procedure was to encourage panelists to reexamine their estimates for
individual events in the light of the influence and probabilities of related
events. Analysis of all individual responses reveals that a relatively high
percentage of respondents altered their final estimates for those developments
included in the set of events which was subjected to conditional probability
assessments. Since this was the third iteration of feedback and reassessment for
many of these developments, it is not unreasonable to assume that the change
in estimates primarily resulted from the evaluation of relationships among
events—relationships which previously had not been fully considered. This assumption is
further supported by the fact that these respondents made almost no changes in
their estimates of other developments, which were not subjected to the specific
routine of estimating conditional probabilities (but were given the benefit of the
feedback of all of the other types of information used in these exercises). In view
of the fact that the relationships among events were stressed throughout these
exercises, any movement in the final estimates as a result of the consideration of
specific conditioning effects is believed to be significant.
Ivvi-lopnients ;md Events that Respondents
Have Suggested Are Interrelated
Probabi1i ty
1971-80
501. Probability
Date
>-32 Requirement by the state, calling for tertiary
treatment of municipal sewage for Traverse City___ ;
Your Previous Estimates_____________
:
Panel Estinates, Round 3_________________________
80
75 (50-85)*
1977
1977 (1975-80)
1978 (1975-80)
Those Who Rated Conpetence 2 3____________
Your Next Estimates for D-32
>
'3| Construction of a spray irrigation system for waste
water disposal in the Grand Traverse Bay region___
Your Previous Estimates__________________________
83 (62-95)
Panel Estimates, Round 3_________________________ i
Those Who Rated Competence > 3___________________'
50 (50-50)
1980
1980 (1980-90)
50 (15-50)
1980 (1978-80)
50
Your Next Estimates for D-31_______________ '
If you were certain that D-32 would occur before 1980, your estimates for D-31 would be j
If you were certain that D-32 would not occur in
1971-80, your estimates for D-31 would be
General Applications:
Delphi Inquiries and Knowledge Utilization
107
An analysis of the estimates of the technical panel showed that some
respondents appeared to have considerable difficulty making probability esti
mates both for a fixed period (1971-80) and for fixed levels of probabilh. (25,
50, and 75 percent). In some cases inconsistent estimates were made (for
example, the probability of occurrence during 1971-80 was estimated to be
greater than 50 percent, but the year associated with a 50 percent probability
was later than 1980).
Fixed probabilities of 25, 50, and 75 percent were selected for personal
probability assessments by the broader-based panelists for several reasons:
(1) . There was strong agreement among the three groups involved in the
exercises—technical, behavioral, and decision makers—on the words and phras
es that they associated with the numerical probabilities of 25, 50, and 75
percent |3].
(2) Individual distributions provided the decisionmakers with more informa
tion than single probability estimates and were believed to be helpful to the
estimator in making assessments that were consistent with his judgment [4].
(3) The 25, 50, and 75 percent levels of probability were ideal for using a
betting rationale, that is, systematically dividing the future into equally attrac
tive segments.
(4) It was believed that group medians associated with these fixed probabili
ties would provide an easily identifiable consensus range.
Since it was likely that many of the decisionmakers would have had little
experience with the notion of personal probabilities, a guide for making
personal estimates of probability was sent to all members of the broad panels—
researchers as well as decisionmakers. The guide presented a systematic method
for arriving at the timing estimates for each technical and social development.
The assessor was asked to visualize a movable pointer below a sequence of
numbers representing years, as in the diagram below. He was asked to move
the pointer mentally so as to divide the future into two periods in which the
development was equally likely to occur. •
1
1
1
9
9
9
7
8
9
123456789012 34567890 Later
A
t
50% Probability
■ Interquartile Range
Fig. 1. Example of interrelated developments.
If the result appeared as it does in the diagram above, 1983 should be entered
as the 50 percent probability date. It could also be described as the “l-to-l”
odds or “even chance” date. If the pointer came to rest beyond 1990, “Later”
would be recorded, and the assessor would go on to consider the next develop-
I
6
tFl
■ TW
ill
■(Bi
■W
I
■; j
John Ludlow
ment. The assessor was then instructed how to divide up the results to estimate
the 3-to-l and “ 1-to-3” odds.
Because of the interest in technology transfer and knowledge utilization in
u.
4)
X
T
patterns of the technicians and decisionmakers, which were displayed as in Fig.
2. For each round the panel medians (connected by a solid line) and the
W
spondents in each group who rated their competence in the area being
considered relatively high were indicated by asterisks. For most items generally
each group’s median estimate lor the final round was very close to the median
/
estimates of those who considered themselves relatively competent in the
subject. Also, the consensus—as measured by the interquartile range-
>■ 5
I
2 ac
7 .2 .£
_ X
S£
••
qj
U
U
i?
4/
'j
3
3
i
narrowed and the average estimates of the two groups tended to come closer
together. Some of the other patterns, while not ideal from the standpoint of
I
I
I'
-*
<>
»
■=i
/I
4»
a
/
yi
g O
1
/
• ir?-
2
3
I
i
i
a
interquartile ranges (connected by dashed lines) were shown. The rounds were
numbered from left to right for the researchers and from right to left for the
decisionmakers, to facilitate the comparisons. The average judgments of re
7> "O
>> 3
CM
i
o
I
4
u
- i i
\
\
H
I
o
05
a. 3 a
A crucial consideration in planning for intelligent management of water
tvsources is the identification of the most important sources of pollution. In
making their judgments, panelists were asked to assume a future social and
oo
x
X
O'
X
O'
M*
X
O'
Q
4)
X
E
I
(A
4)
<S
X
<u
4>
(M
X
O'
O
X
1
r
I
/
X
X
O'
O'
£
2
i
requested to add other items that they felt would affect a body of water
I
comparable to Grand Traverse Bay in the next twenty years. The collated
responses identified seventeen additional sources of pollution and eighteen
additional pollutants for the panel to consider. Since there were too many
x
CM
i
i
i
i
\
45
%
ro
I
I
I
V
CM
H
05
>-
I
I
o5
5 Q
'* *
I J. J—L J__ L 1_J_
X
O'
zo
O'
2
>
m
tn
3
C (55
(fl x
2 45
a)
x £
C
in
tn
o
So
o
—
.*0
7)
fO
05
*
rr
c
Ctf
v>
•I
i
’2
I
1
/'
-•i
i
, r -F
CM
&
'£
I
MS
.2 » S «
(M
00
O'
o
x
X
X
'T
O'
O'
O
3
_
O
45
C5
05
G5
n
>
u
£ «
Q ?
l
i
I
relative importance, for two future periods. The information feedback for the
followmg round provided statistical summaries for Group A, all respondents;
3
•—*
tn
cn
4^
■
■o
45
i
3
7)
'j>
/
u
3
O
0
E
1
do1’
7)
3
d
-
—I
£
0£
X
■w
3
3 3 £
3
narrow the number of alternatives. The evaluation matrix of round three
presented the ten most important sources of pollution as determined by a
statistical summary of the estimates made in round two. Panelists were asked to
distribute 100 points among the sources of pollution, according to each one’s
I
tn
05
7)
tion of several evaluation factors, the primary objective of round two was to
4>
I1
,0
'o T5
T5
w X
7)
45
W
alternatives to present in a matrix designed to encourage the careful considera
1
•- 5
15
•C x
C
cg 4)
<g
3
•
c
On the first round the technical panel was provided with a list of sources of
pollution and specific pollutants thought to be important. Panelists were
x 1
i
i
i
i
I /
I /
S.'o
4)
w
0
X
T
03
that concurrent Delphi inquiries regarding important developments and requisilt technology would influence their estimates.
c <y
0
X
o
7)
political environment consistent with present trends. However, it was expected
O
E
s
o
0
05
o5
O X
in 0
I
Group B, those who rated their competence on sources of pollution relatively
ci
£
i
d £ h
(M
2
a>
cu
\
movement toward a narrower consensus, provided a decisionmaker with infor
mation as to a course of further inquiry.
I
109
\ '
\ I.
1
'
S
Michigan’s Delphi inquiries, there was a special interest in the judgment
Sources of Pollution
T 1 1 u,
1
^■ i :
r
E
O'
,
> H
J
; 14
a
i iU
John Ludlow
x
■M
Q. O
<XJ
s?
M
o
<:
g
w
X
o o
g =1
s
Ous
4<
M
>.
m
>>
_
r-<
MM ft)
-rt
W x 3 ft) o 3 '-<3 0
M U > Z O woe
U
II II
U
II
X _____ CM
CO________
TT
UO
I
.■g
It
3
xo cr
o
0 G r3 - re
4)4)C
x
X
>,
O
E G re 73 re
Z ft) 0
u
G
O
G >. X X ft) >»b£ 2
O re «-) 73 0 0 73 •u ■’W 3
W (J x
3 3 O •-< —t o
W W X W X Z x x a
II U
U
U
II
___ 7"
CM
CM
cn
re
CM
co
«r uo
CJ
O
X
X
‘ajW
E
co
CT)
X
x
cn
X
CD
iSg
5
I®
c
5c
4)
co
m
CM
O
in
CT)
CO
o
TfWI
CJ
CD
u
T
CM
H
T
CO
£
*
Tco
co
no
CM
CM
CO
co
co
in
T
h
co
in
co
X
CM
O C
Im re
ffl a
CM
uo
CO
co
N
UO
C?
CM
CM
T
co
CO
co
E r—
X
X X "JF"
in
uo
in
CM
Yco
co
co
co
in
co
co
in
co
uo
T
co
T
CM
re
co
co
m
it
.2^
Jj
CM
n
T
CM
in
co
m
a> •
co
CO
Tco
X X X
■So
WO
•— m
•r- (D
4- > 4->
5- 03
03 E
3 'rcr+J
S- uo
CD CD
uo
Tco
A
'?
CM
CM
X
U0
•§»
<v
*»
TCM
l—M
o
Z r-4
O'.
O
H CJ -4 r-
Om
7! i
O
CD
r~
■o
TD
x a.
’U
= O E -O re
8reG->,5)^'t£
z 2
CD
o
T>
re 4>
O U
U re
tj
J
£
V)
in
ft) c —i ■o
o -tx x: 4) G>
E
o o c '' «
CO
0 ft) re re
rEQ. G
O
X
CT)
E CO
G ’re
r—<
>■
w
CT)
X
r-< tX C
4) C
COG
re E 3
CX re ■u
sX
O
CM
o c >-<
~ Q
m a.
o
U
Many communities in the Great Lakes basin are confronted with decisions on
waste-water treatment and disposal systems that will have important con
sequences for the future socioeconomic development of their region. This is a
highly technical and complex issue, and decisionmakers must intuitively assess
the judgments of experts in many specialized areas.
A systematic consideration of the available alternatives and the identification
of areas of agreement and disagreement within and between the three general
groups involved in these exercises may aid planners from this region as well as
those from many other communities in the Great Lakes region facing similar
problems and decisions.
Included in the technical panel s round-three information package was an
evaluation matrix that listed six alternative waste-water treatment and disposal
systems. 1 anel members were asked to suggest other alternatives and to
evaluate each of them in terms of two different starting dates for the construc
tion of the necessary facilities. Variances in the estimates were to be attributed
to assumptions about the technology that would be available at the two starting
dates. Panel members were instructed to give 100 points to their first choice for
each time period and a portion of 100 points to the remaining alternatives
according to their value relative to the first choice.
The round-four information package provided panel members with a
summary of the estimates made in the third round. The evaluation matrix for
O
CT>
C3 C
u rM
Q,
r-1 O
•U
CJ O CO
X X s
G O -J
O E
3
4)
E ft> .0
ft)
<0
Final
Fstimntfts
4) Q. u
G E u
® -< Q.
I
X
cj
O
-n: re
x --
-r. m
CT>
5
OEM
J= re
o U 5 4-< >> a,
4-* U
0) C --< X
o M•C ft) ® U E
o u c -> re
cm
C
-j x u
15^tc
X
CT)
CD
X
CT)
G
♦J (fl
X ft)
in
G re
a a.
?
-j o.
0
E
■o <-•
CJ ® CM
O C ■-<
^2?
>
p
11.1J
in
o c
high, and Group C, respondents in Group B who were also relatively familial
with the Grand Traverse Bay watershed area. Although Group B differed
considerably in size from Group C, the average estimates of the two were
remarkably close. 1'his finding might suggest that technical competence is a
more important requisite for panel membership than familiarity with a specific
region, an idea that could have important implications for interdisciplinary
programs such as Sea Grant, in which research methodologies developed for a
subregion are to be applied to a larger socioeconomic system.
In the broad panel exercises the evaluation matrix for round two was similar
to the final matrix used in the technical panel. The evaluation matrix for the
following round provided statistical summaries of the estimates of both the
technical panel and the broader-based panels (Fig. 3).
A significant difference in the final estimates regarding the relative impor
tance of the effluent from the I raverse City sewage system suggests that a series
of estimates conditional on specific social, political, or technical developments
could be used to determine the assumptions on which the evaluators based their
estimates. I he reason for the differences in estimates could also be sought
I hi oug11 intel views <ind othei 11iv.ins ol < <>iiiiihiiik .i(i<hi.
Recommended Waste-Water Treatment and Disposal Systems
4
F ina 1
Es t ima tes
G
in
co
in
in
CM
CO
CM
iQ
in
uo
m
K
3
co
in
in
n
r-<
in
CD
VO
CD
cd
5^
C CD
•2’
LI
QJ 4J
C +-’
^x
0)
X
X
<D
0)
b£
x
o
5
O
X
CJ
B
X
3
re
o
a
e
5
o
rM
-<
3
o
a
>
re
a.
2
if
J
G
gs
X
X
z
o
.2
X
re
6
$
X 3
<D o
□
a
■rt
x
X?
O
O
CJ
□ (fl
n
E
U
X re
O
s 2
is i
O 4-»
c
3
X rl
X
o
w
o
re re
tx
CD
c o
c
s- S
E
>.
CD vo
1
s
re
X
•2
so
X
>5
03 XL
(H X
4-1 4J
•r- C
-M CD
S- XJ
05 C
3 O
o a
cr o.
I re
3 *"
i- vo
CD <D
fc, O
® a
c
• o
re fh
03 <D
*= 2- •rX> E
I I I.J-!U«
£ u S
n.
tx
11
CD 8
0)
(fl
□
c
CJ
•g
k 2
X
I(
i 1 ’’
X
G
i
E
3
C
Y
cm
m
8
’9
a I e
J
It
: MS
' 'I
iJ
fl
: ’1WM
I
l i '4
'
John Ludlow
tn
that round (Fig. 4) requested two evaluations for the six alternative waste-water
treatment and disposal systems for two different starting dates. In the first
evaluation the respondents were asked to consider all factors, in particular the
technology available at the start of construction; in the second evaluation they
were to consider only ten-year operating costs.
The broad panels used the same evaluation matrix as the technical panel in
their final round of estimates, and they were also given a summary of the
results of the technical panel’s evaluation of all factors except for cost estimates.
The broad panelists were advised that the technical panel probably emphasized
technical factors in making their estimates. They were also told that the
recommendations applied to a region similar to the Grand Traverse Bay area
and could differ significantly if the technical panel had considered a specific
situation.
A comparison of the average estimates of those on the technical panel who
rated their competence relatively high with the average estimates of the
respondents on the broad panels showed a very close agreement for both
planning periods. This agreement was evident when panelists considered all
factors and also when they considered ten-year operating costs, although the
values assigned to each alternative relative to operating costs varied con
siderably from the values assigned when all factors were considered.
The judgments of the technical experts are believed to embody risk con
siderations applied to a general situation, whereas the judgments of the broad
panels are thought to be more oriented to the benefits of alternative approaches
for their specific region. Cost estimates include operating costs only; the
consideration of investment costs and financing methods could be equally
important to the decision maker.
The waste-water treatment and disposal system issue was undertaken
primarily to educate the participants and to explore the problem of gathering a
representative group of people and interesting them in the problem. The results
could provide important material for gaming techniques and background
information for deliberations using a variety of methods of information ex
change and analysis.
E
g >2
s^5 “
.
<p
rC
2
CQ
O
*->
G
W
X
O'
"O ni
-J
-U «
? re
u
<.
o
a
x
S « H S
re g
•
2
3u. -1/1
<
re
re
5? 4> c
3 (D (D|_ 0
cr
I
I«» > g>
S
- s 1
ai
2
2 „
re —.
”
I
4)
*
x SE
< x
tn
k
in
co
4)
re
CD
?s is -s»
re
k g a
?A)i ■ore
§—
H
re-
re
x
k
re
u
€
'h
V
:
I
».
$
k.
x
X X
■^1
*-> c
k
'
i
i
m
oo c*4 m
<n co fx
co
o re
b &
u
< ffl
CM fM
<
o
m
<
CD
*k
0
0^
>
CD
C
Q.
*u
y
’«
<D
-
o -k
,rew
w
tn
u
<
o
X
aa
«
- w>
4)
5
o
o
4J
QU
k
V
re
x
1
f—<
u
4)
' >i>
K —
k *->
00
s 2 s. -
u. m
u x o
CL •'*
2 5 X OU X re4) re 2
o
2 2
4E
4)
OU
k
M
k
B
re
re
5 k
Q
< <
cn
<
O
S
I! x re
• u 4> o
b
•I x i -o 'g
<
£
i
W
b
o re
«? £ !
2 -E a S
Eo 3>
CQ
u
. i .is
T 53 ? I
5 I?
- E£
■hili
0 sS •• 5 ou
b U £ O <
ii
ii
it
ii
■
ii
H Sx
H
•
«■ §
O CH
□□□□□
s ssss
re
re b
w
cH
W
b
>.
to
u w
X u
ECL
H
Z
W
cH
H
s=
o
k
X
ft
reCD
o
k
V
Q
k
8
£
51
OU
S
V
u
u
a
v
'U
4)
«
rK
E
45
*-• 4) k
3 3CD4> ok :flj s h oo S” tr »
re
k
.2
re *- 2
0 re » fl
> X M £
xSt^reaiX.-u
34) .£ □ °■ «« 33 Oo gS
■O u p
i
z
k - .x 4
° x Xre* Su , • 4> —
CD
CD
E5
3s
J £4>e ° O
4> c
U S
a
° “ 2 tn O
2 s 2 ° ~ 8<DJ (0
3 S £J ■S -5 —“
6 >
Q- -w
4>
°
3
o
Ik
■Y
ou
C
4
k
•- k
«
2 v o
u
V
ui
■
<D X
>—
E r ?re*3 • ** Lo
? E 2 3 > CQ^O jU)tnX;„ k
« 2 re>- *-•03 QU o-u Xx4) •-5C Ore u
re
”s
w
re
ALTERNATIVE
v
2 «
re
f §-k
I.- >11; ui' ■■
h
CH
'o 2
E
0 * M
a •>
t
v a
4)
w x
.2O k
L
a ” re- -re £u xX
X vO 4) 4) re X o
J f > a w u
£ X * X
re
x
s
3
£
5
* CD X
re 4) U
k «J
o
O s CM
U O ~
o>
Z
t V
w5 reE
re
00
O'
v
re
?
co
re E 2 4)
tn
■"
*O
0
X
"
T O
00 o
E
re
|.i>- \
H X
«-■
4)
m
f
‘ i-
c
re nj
5o 1re
pj M
'QC *-•v tn.E kX
«l
re -H
«£
"
s
6
gf E
GJ
O
E
o
w
v
o
'6
nJ
in
i ><4
&
(fl
v
♦-»
O
re '
"•g 2 i
0. re O
re
X
—
O
z
Q
W
Q
Z
H
>.
•3co
w
X
re u
O
• r-
C CM
co
co
4)
k
'‘
is
113
U O
re
re cr X u
!
O
V
u
'D
Regional Opportunities, Problems, and Planning Strategies
A Delphi methodology was used to generate and evaluate suggestions regarding
regional opportunities, problems, and planning strategies. The group
summaries represent initial individual judgments in terms of a Delphi metho
dology in that these items were suggested on one round and evaluated on a
subsequent round, but not subjected to iterative cycles of reassessments based
on statistical feedback. However, many of the assessments were influenced by
prior consideration of the following in other phases of the Delphi exercises:
CO
O
'’ ' 1
«
-
c >v re
-l|
reil 0 .2 8. 5
o « 01 <D °
If
re •>
s
*4)
re
re
2x
o X
Q
3 V
I x 8
2 •2 i re 2
H
“
1N t «x- 3 * C re ®
; £2 -~~ g so ’G)O « £ EJ S
J
xI
CD C
CL
U C r
“
■ r ■ A
4) ?
k
o
o
I.’A |
2c
i tj
a
re
k
4)
X
I
‘>S
■
’o
re
□□□□0 4.
e 5 S 5sS
£
• •
■ •
w
I'.U?
; i
John Ludlow
1) the trends of statistical measures which have traditionally been used to de
scribe social and economic development; (2) the probabilities and importance
.ssociated with potential technical, social, economic, and political develop'tents; (3) the relative importance of future sources of pollution; and (4)
alternative waste-water treatment and disposal systems.
On the final round a list of suggestions regarding opportunities, problems,
■nd planning strategies was presented to the broader-based panels. Panel’
members were asked to indicate whether an individual item should be singled
ut for special consideration by regional planners using a six-point scale and
•ssociated descriptive words as shown in Fig. 5. In interpreting the group means
. value of 3.5 was viewed as the neutral point. The boundaries for the
esc iipuvc phrases arc as shown.
strongly
disagree
disagree
1
2
1.5
somewhat
disagree
somewhat
agree
agree
strongly
agree
3
4
5
6
2.5
3.5
4.5
5.5
Fig. 5. Singling out individual item for special consideration.
Although the primary interest in these <exercises was to identify areas of
sagreement and the underlying reasons for them,, a Delphi inquiry provides
u accounting of the complete set of items that was considered by the
■spondents—an important concept when an interdisciplinary team of rearchers is involved.
7. Evaluation of the Methodology
here js far from universal agreement on the merits of the Delphi techniques
Kand believes that Delphi marks the beginning of a whole new field of
.search, which it labels “opinion technology”[5]. However, a paper presented
the joint statistical meetings of the American Statistical Association in
\ugust 1971 described the Delphi techniques as the antithesis of scientific
recasting and of questionable practical credibility |6].
According to a recent Wall Street Journal article, the Delphi technique is
.uning rather widespread use in technological forecasting and corporate plan.ng, although the same article cautions:
General Applications:
Delphi Inquiries and Knowledge Utilization
115
I
?
It’s easy enough to see the shortcomings of the Delphi procedure; it’s much harder
to rectify them, as many are struggling to do. Remedial work must be done if the '
method is to be used in good conscience [7].
I he Sea Grant Delphi exercises offered
an exceptional opportunity for a
critical evaluation of the Delphi techniques in an operational environment.
an operational environment,
The panelists—the main resource in evaluating the methodology—were interested in tthe
’
improvement of techniques5 to
to integrate
integrate the
the judgments
judgments of
of a
a
^multidisciplinary research team and to convey its informed insights to society.
Their evaluations were not biased by a strong emotional involvement in the '
strong emotional involvement in the ■
success of the Delphi exercises, as has been true with many of the individual
assessments of the method that have been published. From both a program
budgi iing siandpomt and demands on researchers’ time, the Delphi exercises
competed with a wide variety of other methods for securing and disseminating
information.
°
I he primary instrument in evaluating the effectiveness of the method and its
potential in other applications was a formal questionnaire. It was developed
almost entirely by the respondents themselves using the Delphi technique of
feeding back collated individual suggestions to generate additional suggestions
This procedure somewhat reduces the vulnerability of the questionnaire to the
biases and shortcomings of the investigator. The six-point scale and associated
descriptive words shown in Fig. 5 were used to quantify degrees of agreement
and disagreement. To supplement the formal questionnaire, over thirty-five
interviews with panelists were conducted.
Summaries were made for the three general groups participating in the Sea
Grant Delphi exercises: technicians (Group I), behaviorists (Group II), and
decisionmakers (Group III). For some issues the summaries for technical
panelists under forty years of age and panelists with previous experience with
t ic Delphi method were shown. Using the sample results, tests of significance
were made to test the hypothesis that the distributions of the judgments of the
Delphi method are homogeneous across the groups (the test procedure was
based on the chi-square test statistic) and to test the null hypothesis that the
means of the judgments of the population represented by the groups are
identical (based on analysis of variance and the F-test). The results of these
tests vvere used to support the discovery of basic differences in judgments made
by different groups which had been formed on the basis of similar backgrounds
and experiences. Their evaluations provided evidence that the method is
effective not only in its designed role but in two other roles that
are important
and challenging from a imanagement
---------•
standpoint:
encouraging greater involvement and facilitating communication between
-------- 1 researchers and de- I
cisionmakers. The evaluations also showed that
among the carefully selected
samples of people the techniques were more highly regarded among groups
which were formed on the basis of broad ranges in training and experience j
than among technicians—the group most administrators of the techniques have
focused on.
.. Jj
I
ski
I'M
tl
li
'I
' i
I
11
John Ludlow
General Applications:
1 he reliability of the method was demonstrated by the fact that the perfor
Delphi Inquiries and Knowledge Utilization
• *
I he greatest difference in judgments based on experience
and potential was
found in tthe
................................
behaviorists’ group, and the least among the decisionmakers. On
the basis of the
.l.j iaverage judgments for all respondents the Sea Grant Delphi
mance of the respondents, as measured by group statistical summaries, was
similar for the three groups. Respondents from all three groups were generally
willing to suggest future developments, sources of pollution, and research
priorities; to utilize scaled descriptors to quantify subjective judgments; to'
the Opinion of Informed People
Table 2
Comparison of Evaluations Based on Experience in Sea Grant
I
Technicians
Exercises with Evaluations
Based on the Potential of the Delphi Method
I
Experience
Potential
.11
.05
.14
.05
.08
.16
.44
.39
.19
.30
.03
.05
36
57
3.6
4.0
1.340
1.172
II
Experience
Potential
.00
.00
.00
.00
.10
.04
.65
.15
.15
.42
.10
.32
20
24
4.3
5.1
.696
.859
111
Experience
Potential
.00
.00
.00
.00
.00
.00
.33
.13
.37
.50
.30
.37
30
30
5.0
5.2
.809
.679
II, & III
Experience
Potential
.05
.03
.06
.03
.06
.09
.45
.27
.24
.40
.14
.19
83
111
4.2
4.6
1.218
1.101
Respondents
under 40
Experience
Potential
.07
.05
.02
.00
.07
.09
.54
.21
.22
.45
.07
.20
41
56
4.0
4.6
1.172
1.218
Previous Delphi
Experience
Experience
Potential
.00
.00
.00
.08
.11
.49
.17
.31
.45
.13
.28
39
47
4.5
4.9
.941
.938
Technicians
Behaviorists
Decisionmakers
All Respondents
strongly disagree.
strongly agree.
.00
o
1
2
Experience
Potential
. 11
.05
.11
.05
.06
.11
.50
.42
.22
.32
.00
.05
18
19
3.6
4.1
1.290
a
i.nh
‘I?
II
Behaviorists
Experience
Potential
.00
.00
.00
.00
.00
.00
.72
.43
.29
.43
.00
.14
7
7
4.3
4.7
.488
.756
III
Decisionmakers
Experience
Potential
.00
.00
.00
.00
.00
.00
.22
.10
.22
.50
.56
.40
9
10
5.3
5.3
.675
.675
I, II, & III
All Respondents
Experience
Potential
.06
.03
.06
.03
.03
.06
.47
.33
.23
.39
.15
.17
34
36
4.2
4.5
1.274
1.108
a
Respondents
under 40
Experience
Potential
.06
.05
.00
.00
.00
.00
.69
.21
.19
.58
.06
.16
16
19
4.1
4.7
1.025
1.098
1
j
Previous Delphi
Experience
Experience
Potential
Potential
.00
.00
.07
.00
.00
.06
.47
.19
.27
.50
.20
.25
15
16
4.5
4.9
1.060
.085
f
Combining Groups II and III:
I
_A
Distribution of Judgments
(proportion of total)
3
4
5
6
n
x
Group/Agreement Code«
2
i|
Table 3
Effectiveness of Delphi in Obtaining, Combining, and Displaying
examining a cumulative summary of the evaluation of the effectiveness of the
method in three specific roles shown in Table 2.
1
■ “
treatment, and there is a similar spread in the judgments of the subgroup which
had previous experience with the method.
Some insight into the nature of the difference between judgments based on
panelists’ experience in the Sea Grant Delphi inquiries and the panelists’
conception of an ideal application of the Delphi techniques can be gained by
Distribution of Judgments
(proportion of total)
3
4
5
6
n
x
:■
|
exercises corresponded rather closely to the panelists’ conception of an ideal
accept a statistical aggregation of weights supplied by a group; and to reassess
their judgments on the basis of feedback of information supplied by the group.
Group/Agreement Code =
117
1: strongly disagree.
6: strongly agree.
P(F> 4.6233) = .0165
P(X2> 8.3588)-.2130
P(X2> 3.959) — .0464
II
J
—
A summary of the evaluation of the effectiveness of the Delphi method in
obtaining, combining, and displaying the opinions of informed people is shown
in I able 3. It indicates that the technicians, on the average, agreed somewhat
that the method was effective compared to alternative methods. However, there
I
I
fl
I
John Ludlow
. considerable dispersion in their estimates; some respondents strongly dis~ec that the method was effective in this role. T he behaviorists agreed that it
is effective in the role, and the decisionmakers displayed strong agreement,
persion in the estimates of the two latter groups was much less than it was in
technicians’ estimates. An analysis of variance on these data gives a value of
•?.l I lol the /■ sl.uislK . I he pi <>l >.il >i h t v <»f <>bl.lining .in /■' v.ilnr l.ugci lli.tn
<1 when th<' gloups .uc idrnlii.il is l)lh'» \ < hl sqn.iir ,in.il\sis gives a
'tie loi i 111-scpi.11c of 8..I.>88. I he piob.ibilils ol olil.iiniiig a l.ugci value,
■ ii m fact the chstribuliuns come hum a homogeneous population, is .‘2130
descriptive level of significance. A combination of the judgments of the
isioninakers and the behaviorists resulted in I\\2 > 9.165) = .0025.
• .valuation of the method’s effectiveness in encouraging greater involvement
Sea Grant activities provided similar results. The average judgments of
t'liveness were A7 =4.0, A/y = 4.7, and AZJV/ = 5.1, and the dispersion in the
.mates of the technicians was much greater than for the other two groups.
descriptive levels of significance are .0026 based on the F distribution, and
!<»0 based on the chi-square distribution.
• crage judgments regarding the method’s effectiveness in communicating
'imation to regional planners and decisionmakers were A7 =4.1, XH = 5.4,
=
Die judgments of the behaviorists were exceptionally high and
. reflect this group’s special concern for the psychological and sociological
' iers associated with alternate methods. The descriptive level of significance
0003 based on the F distribution, and .0108 based on the chi-square
i ibution,
‘ or all three roles there appears to be little difference in the average
• mates aggregated according to the respondent’s age and the average estics of all respondents. However, respondents with previous Delphi exience showed substantially higher average estimates of the method’s effecncss in obtaining, combining, and displaying subjective judgments and in
ouraging involvement.
\verage judgments regarding specific applications that are appropriate for a
iphi methodology are shown in I able 4. The support of specific applications
generally strongest among the decisionmakers and weakest among the
hnicians. Only the behaviorists and decisionmakers evaluated the method as
aid in decisionmaking, and both groups supported its use in this role.
i the evaluation of positive and negative aspects of the method that had
n suggested by respondents, the panels agreed that there should be more
• phasis on the idea that an expert should “not feel obligated to express an
lion on every issue.” However, the Sea Grant Delphi inquiries stressed the
icepts of a systems approach and multidisciplinary teams. Therefore it was
ued that each respondent consider all items and attempt estimates on those
i which he had some familiarity. /\ self-evaluation index was provided so
• t a panelist could assess his competence regarding each item. The compeiCe index was a control factor in developing the statistical summaries that
General Applications:
Delphi Inquiries and Knowledge Utilization
119
were part of the information feedback. This procedure allows an informed
person to evaluate such things as relative importance and desirability— evaluations which he can make without being an expert in the area—and gives
the administrator additional assurance that panelists considered items outside
their specialized areas. In addition, there was some interest in comparing the
rsiini.iirs of experts and nonexperts on specific issues.
8 Si
•
I
|
I
Table 4
Suitable Applications for a Delphi Methodology
Technicians
Mean a
Behaviorists
Mean a
Decisionmakers
Mean a
Develop Criteria to
Recruit Industries
4.3
.996
4.5
1.130
5.3
.675
Establish Program Priorities
within a Committee
3.9
1.393
4.6
1.453
5.0
1.080
Long-range Planning by
a University
3.8
1.393
4.1
1.453
4.8
1.080
Education of Practicing
Politicians
4.1
.793
4.3
.707
4.6
1.424
Budgeting of an
Interdisciplinary Program
3.9
1.353
3.7
.866
4.8
.789
4.5
.943
4.3
1.165
4.5
.707
5.1
.835
5.0
1.054
Application
Situations Dealing with
the Future, Uncertainty,
Conflicting Views
■
i
■
Decisionmaking Aid
The suggestion that the method “can result in a manipulated and arbitrary
consensus” received a neutral judgment from all three groups, perhaps indicat
ing that the respondents felt this danger to be no greater than it would be in
alternative techniques for securing group judgments. However, it is this
administrator’s opinion that the Delphi techniques could be a powerful tool for
manipulating opinion and policy [8].
IV.
i
■
•V.
Summary of Findings and Recommendations
I he design and administration of a Delphi exercise in which the concept of a
multidisciplinary team and a systems approach is desired can best be handled
as a project within a professional research organization. The scope of the
exercise is generally determined by the respondents, and, as interesting and
unexpected issues are suggested, flexibility is needed in designing evaluation
matrices and in determining the composition of the panels. Experts knowledge-
1
John Ludlow
able in specialized areas should be available on an adhoc basis to formulatTl
an ad hoc basis to formulate
questions and collate responses in order to minimize redundancy and ambiguity. 'Phe demand for their services in the
of a Delphi exercise is very ,
— course
-------- „f
uneven, as is the need for designing, editing, typing, and distribution services^
There are significant start-up and learning costs associated with the Delphi
techniques that can be justified only if the technique will become a routine
management tool to be used on a continuing basis. 'Phis is particularly true if
lhe benefits of computer processing are to be realized.
1 he following are some general observations that are consistent with the
items suggested and evaluated by respondents in the Sea Grant exercises and
with the information gained in personal interviews with panelists.
» Respondents will be more receptive if the techniques are tailored to specific
gioups on the basis of their training and experience.
« I he administrator should consistently emphasize the distinction between
lhe characteristics of a Delphi interrogation and those of conventional
questionnaires and polls.
« Panelists particularly those with technical backgrounds—must be convinced
that judgments often have to be made about issues before all facets of the
problems have been researched and analyzed to the extent they would like.
(Tor these situations they must be persuaded that their subjective judgments
may be a decisionmaker’s most valuable source of information.)
1 here are several procedural recommendations that may
may be
be helpful
helpful to
designers and administrators of future Delphi exercises.
« Interpersonal techniques, such as interviews and seminars, should be in
terspersed with the rounds of questionnaires and information feedback.
« 'The source of a suggested item should be identified (for example, panel
member number and basic biographical information), taking care not to
compromise the anonymity of specific inputs.
• Standardized scaled measures should be available to a respondent so that
he can qualify his response io specific questions. Such measures are relative
competence in a technical area . familiaiily with .1 I’eot’raphical region, or
confidence in an estimate.
a If a multidisciplinary approach is desired, respondents should be en
couraged to consider all items but to make estimates only on those scaled
descriptive phrases with which he feels comfortable. For example, in these
exercises it was helpful when respondents indicated their familiarity with a
specialized area or the importance of an item even though they did not make
probability estimates.
• I he panelists should decide through their suggestions and evaluations what
aems should be considered. The criteria for retaining an item for further
■ valuation should be made clear at the outset of the exercise.
—‘
General .Applications:
Delphi Inquiries and Knowledge Utilization
121
• Personal comments and arguments submitted by respondents should Be
part of the information feedback.
The Delphi inquiries have complemented the Michigan Sea Grant gamingsimulation activities by providing the following types of inputs:
I
• Data which can be helpful in describing social, economic, and political
forces affecting the region’s development during the next twenty years.
• Regional planning strategies, listed in order of preference for both univers
ity researchers and regional planners.
• Problems and issues which provide the link between the simulated regional
area and a set of decision roles which are gamed [9].
Integration of a Delphi methodology with the Michigan Sea Gram gamingsimulation exercises will give them a more dynamic aspect and provide greater
motivation for the participants. Some particularly interesting applications
would be in cost-benefit analyses similar to those used in the Delphi inquiries to
evaluate waste-water treatment and disposal systems, selection of research
projects through an evaluation of effectiveness in terms of basic objectives and
risk factors associated with various levels of funding, and the development of
alternate scenarios for a region such as the Grand Traverse Bay watershed
area.
According to Michael [10], Delphi inquiries and gaming-simulation exercises
are techniques for introducing customers in a nonthreatening way to a more
complex way of thinking and a better way of perceiving their needs in terms of
the kind of knowledge we have. Knowledge utilization depends upon discover
ing the nature of the awareness of the problem among potential customers both
as a set of variables and as a system of interrelationships; getting new knowledge absorbed by individuals and then by the organizations these individuals
O V£*
in ’
KA 7-4
z-4
rz* 1 z-t v-x ■
xw
---- — 1— ? 1
_1
"
1•
. •
«
■
are in; and developing a capability and inclination to plan rather then
employing an ad lib approach.
Concerned citizens were included as panelists in the Delphi inquiries not
only for the purpose of informing them but also to accord the other panel
members the benefits of the citizens’ knowledge of the area, to take into
account political and institutional considerations, and to communicate findings
in such a way that the acceptance and implementation of policies and actions
on which there appeared to be a reasonable consensus would be encouraged.
The behavioral sciences provide support for this type of approach in effective
communication [11,12].
lhe Michigan Delphi inquiries have provided some carefully formulated
judgments of a multidisciplinary team of researchers and potential users of
research data regarding: the importance and effects of technical, social.
......
’
»
economic, and political developments; sources of pollution and recommended
waste-water treatment and disposal systems; and regional ©ppeftunities, pF0b*
lems, and planning strategies. More important, 4 critjcgl evaluation of th?
I
W
H
I
I
II
flI
11
II
I
'
J
1
■ •' a
^2
John Ludlow
General .Applications:
method has shown the potential of a 1 )clphi inquiry for improving the dialogue
between researchers and regional problem solvers. It has also provided empiri
cal evidence to support further investigation of several innovations which may
bring the methodology closer to Janisch's idealized concept of a forecasting
technique, wherein exploratory and normative components arc joined in an
iterative cycle in which informed citizens can work with researchers in planning for the future [ 13].
Delphi Inquiries and Knowledge Utilization
.
I
123 O
■
Other References
Examples of the key instruments used in the Michigan Sea Grant Delphi
inquiries can be found in “Substantive Results in the
t
"" ‘
z of Michigan’s
University
Sea Grant Delphi Inquiry,’’ by John D. Ludlow,. Sea
Sea Grant
Grant Technical
Technical Report
Report
No. 23, University of Michigan,
Evaluation of<MO
Michigan. Ann Arbor,
Arhnr 1972,
1Q79 and in “
«r
Methodology in the University of Michigan’s Sea Grant Delphi Inquiry” by
John D. Ludlow, Sea Grant Technical Report No. 22, University of Michigan,
Ann Arbor, 1972. Complete sets of the information packages can be found in
“A Systems Approach to the Utilization of Experts in Technological and
Environmental Forecasting,” by John D. Ludlow, Ph.D. dissertation for the
University of Michigan, 1971. Available through University Microfilms, Inc.,
.Ann /Arbor.
....
References
..
....
ra.
■
1. Murray Turoff, “Delphi and Its Potential Impact
(
on Information Systems,’’ Paper 81,
Proceedings of the Pali -Joint Computer
Conference,
Vol. ..39.
AFIPS PresZ W^hingt*
■
--------•
....................... .. •
• • uo.uncion, D. C.,
November 1971.
I he techniques and procedures used in this series of interrogations and information feedback
arc similar to those described in “Some Potential Societal Developments—1970-2000’’ by
Raoul De Brigard and Olaf Helmer, IFF Report R-7. Institute for the Future, .Middletown
Conn., April 1970.
3. In one phase of the Delphi inquiries panelists were asked to assign numerical probabilities to
commonly used words or phrases to indicate the likelihood of an event. The Delphi technique
of reassessment based on the feedback of a group response was extremely effective in narrowing
the dispersion of the estimates. Verbal phrases associated with numerical probabilities were
believed to encourage respondents to think about a probability scale in similar terms and might
be more appropriate than numerical probabilities in expressing the likelihood of socioeconomic
developments.
4. The Michigan Delphi inquiries provided empirical evidence that the feedback and reassess
ment techniques which are inherent in the basic Delphi method reduced the number of
inconsistencies in personal estimates of probability as the rounds progressed. It also indicated a
tendency for a learning “curve’’ for respondents with respect to the technique itself.
>. “Forecasters Turn to Group Guesswork,” Business Week, March 14, 1970.
6. Gordon A. Welty, A Critique of the Delphi lechnique” (summary of paper presented at the
Joint Statistical Meetings of the American Statistical Association, Colorado Slate University,
Fort Collins, Colorado, Aug. 23-26, 1971).
7 "Futuriasis: Epidemic of the ’70s,” Wall Street Journal, May 11, 1971.
8. Fur a discussion of the dangers associated with a Delphi devoted to policy issues see Turoff’s
article, above (Chapter III. B. 1.). For a discussion of deliberate distortion see “A Critique of
Some Long-Range Forecasting Developments,” by Gordon Welty (paper presented at 38th
session of the International Statistical Institute. Washington. D. C., August 1971).
J. I he gaming-simulation concept for the Sea Grant Program is presented in “Developing
Alternative Management Policies,” unpublished report, University of Michigan Sea Grant
Office, 1971.
10. Donald Michael is presently program director, Center for Research on Utilization of Scientific
Knowledge, Institute for Social Research, University of Michigan. Theseare a summary of his
unpublished remarks to a site visit team of government officials and academicians in Ann
Arbor, Michigan, March 4, 1972.
u. Douglas McGregor, “The Professional Manager” (New York: Harper & Row, 1967), p. 153:
"...My conception of a two-way communication is that it is a process of mutual influence. If
the communicator begins with the conviction that his position is right and must prevail the
process is not transactional but coercive.”
12. Peter F. Drucker, “Technology, Management and Society” (New York: McGraw-Hill Book
Co., 1970), pp. 22-23: ... They must understand it because they have been through it, rather
than accept it because it is being explained to them.”
<3. Erich Jantsch,
Technological Forecasting in Perspective.” Organization for Economic
Cooperation and Development, Paris, 1967.
if
■
‘. '5-?
■
i
■
!S
■
' II
I
inc iNaiional Drug-Abuse Policy
Delphi: Prog ress Report
and Findings to Date
IRENE ANNE JILLSON
Introduction
Rationale
Although the abuse of drugs has been recognized in this country for nearly one
hundred years its popularity as a national problem has resurfaced relatively
recently. In 1968, former President Nixon declared drug abuse “public enemy
number 1 ; from 1969 until 1974, some S2.4 billion in federal funds were
obligated to combat the problem, and an industry
industry was created. This expendi
ture represents funding of programs that were
were almost exclusively aimed at
teroin abuse, rather than the broad spectrum of drug abuse. Since 1968, drug
abuse has been the subject of intense public and private debate. Controversy
over the government’s appropriate response has ranged from debate regarding
dtug laws (to what extent different drugs should be controlled, and in what
'“TTJ’ “u ' buaS1C qUeS'iOn °f Whe‘her °r not alcoho1 Programs are to be
included with other ’ drug programs on the federal level.
During the past ten years, the increased concern and expansion of drug
abuse prevenuon programs has resulted in a swelling of the ranks of pro
fessionals who have developed expertise in this field; however, the use of these
.xperts m policy advice and formulation has been sporadic and unsystematic
the same time, numerous research and evaluation sludies of drug-abuse
prevention programs themselves have been carried out. The degree to which
csultant data from these studies can be. have been, or should be, used in
• lectsion-making at the nation level has never been resolved.
In the fall of 1973, it was clear that the problem of drug abuse had
minished in priority, and that substantial reductions in federal funding were
mmunent. This may be attributed primarily to the apparent abatement of the
nerom epidemic, which had served as the stimulus for increased concern and
■rogram funding m the late 1960s. Although the crisis associated with the
heroin epidemic may have passed, it is by no means clear that the broader
problem of drug abuse has been resolved: polydrug use and alcoholism appear
t be mcreasmg; and the abuse of prescription drugs, once the “hidden drug
} 1 b Cin) 1S surfac,nS ln many communities. Such a time of decreased public
0-e201-S293-2nd Appii“,ionS’ Haro,d
‘-“o- and Murray Turoff (eds.)
’opyright © 1975 by Addison-Wesley Publishing Company, Inc., Advanced Book Program
"msmiued^In t°
“l'1"8 pubHcation may
reproduced, stored in a retrieval system or
nsmtlted, in any form or by any means, electronic, mechanical photocopying recording or
.herwtse, wnhout the prior permission of the publisher.
g' reCOrd,n8' °r
General Applications:
National Drug Abuse
1^'
125
j
interest and funding for a yet-existing problem calls for careful consideration of
the basic issues, and deliberation of the strategies to be followed, in order to
maximize effective use of resources available.
Viewed from this perspective, the procedures utilized by Policy Delphi
studies seemed most appropriate to explore national drug-abuse policy options
for the next five years. The volatility of many of the issues, most of which
involve fundamental value and sometimes moral choices; the diverse
backgrounds of those who make or influence policy; the apparent differences in
the positions held by various experts and groups; and the apparent inability of
past policy studies to aid decision-makers led to the conceptualization and
implementation of a national drug-abuse Policy Delphi study. The unusually
high response rate, the degree of participation achieved to date, and the interest
on the part of federal and state decision-makers has borne out this initial
hypothesis.
q- :- .
ill
■ ■
1
History
I
The study described in this chapter was originally conceived in 1973, and
designed during the fall of that year. Implementation began in December 1973;
the first questionnaire was disseminated in March 1974. The first two
questionnaires were developed under a contract funded by the National Insti
tute of Drug Abuse.1 Analysis of the data generated by the second
questionnaire, and the further development of the use of the Delphi procedures
in the exploration of national drug-abuse policy, as originally conceived, was
sponsored by the National Coordinating Council on Drug Education.
The study analyzing the first two rounds was completed and published in
December 1974.
I.
i I
M
Objectives uj This Study
There are three primary objectives for this study:
• To develop a range of possible national drug-abuse policy options
• to explore applications of the Policy Delphi methodology to this and other areas of
social policy
• to explore the possibilities of applying the technique on an as-needed basis and on an
ongoing basis
Since the level of drug abuse in the United States is presumed to be both
endemic and epidemic, and since strategies to respond to changes in use
patterns need to be both immediate and long range, this study is concerned
with ascertaining the feasibility of utilizing the Delphi technique to meet these
needs of policy formulation and planning.
• On an as-needed basis. This would involve the use of a panel of experts who would
respond to queries sent as the need arises. For example, if a decision-maker were to be
‘NIDA Contract Number B2C-5352/HOIMA-2-5352.
i
ri
■
■ y
: ■-u
Irene /\nne Jillson
informed that there was a dramatic decline in the number of patients entering
treatment programs, a Delphi would be developed to determine the opinion of
.selected experts with regard to this particular trend.
On an ongoing basis. If one agrees that there is an endemic level of drug abuse then it
JT™ aPPrOPriale lo dcvd“P
ongoing Delphi study of indefinite duration,
his Delphi would be implemented such that questionnaires would be distributed at
regular intervals. Since the panel would be of indefinite duration, membership might
be fluid. Current trends in the field would be incorporated into the questionnaires, so *
that there would be a continuous flow of information for the use of the policymaker,
and those operating programs of various disciplines in the field.
Study Design
A number of advisers—experts in the field of drug abuse, policy planning and
analysis, and the Delphi technique in particular—assisted in developing the
study design. I o date, they have continued to provide assistance in all aspects of
the study. Included are Dr. Norman Dalkey, Engineering Systems Department,
Diversity of California at Los Angeles, who originally was instrumental in
developing the technique in the late 1950’s and who continues to explore its use
as part of his decision-theory research; and Dr. Murray Turoff, who has
developed the Policy Delphi and is co-editor of this book. Dr. Peter Gold
schmidt has assisted in planning and management of this study, and M.
Alexander Stiffman has assisted in developing the analytic approach and
designing the computer analysis. Dr. Raymond Knowles, Charlton Price, and
Anthony Siciargo have assisted in pretesting the questionnaires.
I he final study design was based on the premise that policy may be
[simulated from a number of different perspectives. In designing this Policy
Delphi study we are exploring the formulation of national drug-abuse policy
from three perspectives:
• the “top-down” approach—establishing national drug-abuse policy objectives to be
achieved over the next five years, based on the respondents’ value systems
• a ’ bottom-up approach—identifying factors which control the transition between
general population and various degrees of drug use: deciding which of these arc
important and can be affected by national drug-abuse policy; and determining
appropriate policies to affect them
. an issue-oriented approach-deriving policies from issues which are the subject of
current controversy or debate
lhe underlying thesis in this design is that different decision-makers may
formulate policy predominately using one or another perspective, and that this
results in distinct types of policy options and considerations which may appear
attractive from one perspective, but turn out to be counterproductive from
another. For example, setting an objective using the “top-down” approach may
•esult in its achievement becoming a political issue, while an objective that is
lormulated as the result of a political issue may never be achieved because it is
echnically impossible even though its achievement is valued.
General Applications:
National Drug Abuse
127
I his top-down, bottom-up approach speaks to
the general concept of the
Policy Delphi, in that alternative policy options are drawn from a number'of
'■
Addh'111 V||nUSe P°‘ntS (U ‘S nOt Simp'y a statement °f objectives, for example)
Addtnonally, we are not mterested in consensus per se, but rather, in exploring
Lakernatives, and pro and contra arguments (or the alternatives
S
■or these reasons the outputs from the three approaches used in this study
vv 11 be brought together so that the panel can identify more appropriate
alternatives. We anticipate that this will prove more fruitful in terms of
exploring national policy options than either approach alone. A description of
the approach, round by round, is given in Table 1.
Respondent Population
A list of more than one hundred highly selected potential respondents was
developed from among the most notable “experts” in the field and from those
who directly impacted on the field (e.g., police chiefs). Invitations to participate
m the study were sent to forty-five persons; the remaining names were held in
reserve as a second series of invitations was anticipated in order to secure
twenty-f.ve participants. In fact, thirty-eight individuals (84%) responded posi
tively. Since that time, three respondents have withdrawn from the study owing
to a change in career orientation from drug abuse. Needless to say there
was
no necessity for a second series of invitations.
As the study progressed past the first two rounds, several additional respondents were added. These additional respondents were selected to represent
:
•
s
I
areas o interest which had developed in the study, but for which respondents
had not been initially selected. Experts in alcoholism were added to the panel
or example, because a significant proportion of existing panelists expressed the
vicw^that a national drug-abuse policy could not be considered separately from
alcoholism. 1 he addition of such experts will allow their views to be added to
those of the present panelists, and so provide an appropriate additional
perspective. 1 he present panel consists of thirty-nine persons.
1 O(Ur/eSrundent gr°UP represents some of the most respected authorities in
te be d. I hey include the Deputy Director of the Alcohol, Drug Abuse and
Mental Health Administration, a former director of the Bureau of Narcotics
and Dangerous Drugs, officials from the Office of the Secretary, and ( mce of
the Assistant Secretary for Health of H.E.W., notable researchers, treatment
administrators, law-enforcement officials, and policymakers in the field of drug
abuse. It should be emphasized that participation is voluntary, and that no
tionoianum is paid to respondents.
The Questionnaires
First Questionnaire
nrenUml^r
ap,p™aches were explored during the developmental phases
.P™.P
.°ry ? R°Und O”e- The final draft of 1116 resultant questionnaire was
;
i
■
j||h
UiP?
General Applications:
=4
il
1.
-111
2.
iin
n!5
mH
3.
4.
-3
z :: ?
1 IhH
i
.h
3 SS
J ifi
II
z
I.
h
?.
ni
:S5
HI
H-
.
L r
Iih hll
= =
j
l-M =
Pi fl
j J
-6
I
£z
O 5
5
r .5
f
i
'
Development of Objectives. The respondents were asked to develop up to five national
policy objectives in the field of drug abuse, given a five-year time frame; and to list
up to three key indicators for each objective.
Transition Matrix. Respondents were given a simplified transition model which
depicted the flow from one state of drug involvement to another and were asked to
list the factors which promoted or inhibited the movement of people from one state
to another. (A detailed description of this matrix is provided in a
subsequent
section.)
Policy Issue Statements. Twelve issues, in the form of should/should not statements, ■ ' I : ft
of should/should not
s ’
were posed. These had been culled from a potential
potential list
list of
of twenty-five
twenty-five issues
issues felt
felt to
to ; ; 1 ‘ •:' •1
be current and controversial.
Additional Items. These included a self-rated expertise Question- two
1 -i
pertise question; two questions relating
■ft.
to expectations and objectives for participating in the study; a request for responses
■' ;
to a list of definitions; and a request
request for
for feedback
feedback on questionnaire design and/or
content.
°
Sil
i :h’
5
a.
1- ‘
129
lift
hn
Ji
I i ■
that the time to complete the questionnaire was decreased considt >.tbly by
deleting this section.
I he first-round questionnaire, disseminated in mid-March, consisted of three
primary sections:
-scHi Js
J
National Drug Abuse
:P I’
ill
I wenty-four of the study’s thirty-five respondents (69%) actually completed
the Round One questionnaire. The breakdown
------------- of
„f returns by category of
respondent is shown in Table 2. It
should
be ipointed
------------------—out that mailing posed
serious difficulties both in disseminating and in returning the questionnaires. In
several cases it took two full weeks for the questionnaires to arrive by air mail r
to their destinations; several never arrived, and duplicate packages had to be Mill
sent out. For this reason, tthe deadline for completion of the first-round
?
questionnaires was extended by two weeks.
|
'
The absolute range of time for completion of the first questionnaire was
<
if teen minutes to ten hours. The interquartile range was one to three hours; kffi
the median time for completion was 2{ hours, which was approximately the
v median we had anticipated. The respondents included in our predesignated
policymakers subpanel category spent a median of 3^ hours; the median for all
i' '?r
the remaining participants was 2| hours.
In addition to the substantive-issue questions, respondents were asked to
self-rate their expertise in ten drug-related areas (see Table 3). These data were
then used to cross-check the categories (subpanels) into which respondents had
been placed, and to further analyze responses to particular items. There is one
particular point of interest to be noted. As critical as evaluation is held to be in
the formulation of national policy, not one of the policymakers rated himself
herself as expert in this area.
r
Respondents were also asked to indicate their expectations for the study, and
.a
■
■
J
jl
I
=
II
L
K
li
<5
to list personal objectives with regard to participation. Eighty-two percent of
those responding reported that they expected the study to be of direct benefit.
■J
1 able 2
PARTICIPATION AND RETURNS BY CATEGORIES OF RESPONDENTS (SUBPANEL)*
Invited
Agreed to
Participate
Formally
Withdrew
Delphi
Panel
Completed
Round One
Completed
Round Two
Policy Makers
9
9
0
9
5
5
Researchers
12
9
0
9
9
9
Treatment Program
Administrators
11
11
2
9
4
b
Criminal Justice
Adminis trators
9
6
1
5
3
3
Other
4
3
0
3
3
2
Total
45
38
3
35
24
25
Subpanel
* As of 1 August, 1974.
I
Does not include panelists added after that date.
I
Tabic 3
o
A SUMMARY OF SELF-RATED EXPERTISE BY AREA AND SUBPANEL*
2
Ct
Subpanel:
3
Percent Who Rated Themselves "Expert"(l)
Area of Expertise
Policy Makers
Researchers
C.J.S.(2)
Adminis
trators
National Drug Abuse
Policy Planning
100
33
33
Prevention
20
22
33
T.P.(2)
Adminis
trators
Other
All
Respondents
3
0
100
50
I-
50
0
26
z
p
Intervention
0
33
33
75
0
30
Treatment
40
78
0
100
0
57
Law Enforcement
20
11
100
0
33
25
2
CTQ
Research
20
100
0
50
0
50
>
cr
Evaluation
0
78
0
50
33
44
Training
0
33
33
50
0
26
Education
TO
56
33
50
0
39
Pharmacology
0
11
0
0
0
4
Number of Respondents
5
9
3
4
3
24
S’
□
—
□
s
* As of 1 August, 1974. F
... panelists added after that date.
Does not include
(1) Percentage of those who responded.
(2) C.J.S.
'■ Criminal Justice System; T.l . = Treatment Program
. Hr
1
Irene Anne Jillson
There were twenty-eight different narrative responses to this item. Twenty-two
saw positive utilization of the results for policy planning, idea exchange,
consensus development, etc.; three were uncertain of usefulness, and two were
skeptical of its utility. I here were fifty-four statements of personal-study
objectives; these were distilled to twelve clusters, and are shown as Table 4.
Respondents were asked to comment on a list of standard definitions which
had been prepared for use in the study. Twelve respondents commented on
these definitions. As anticipated, there was little agreement with these defini
tions on the part of those who commented. Apart from giving each respondent
a common base line, one of the reasons for including the list of definitions was
to determine what interest would be aroused. In most of the social-service areas,
and particularly in the field of drug abuse, there is much dissension even
among policymakers, regarding critical definitions (e.g., drug abuse, modality
types). Clearly it is not sufficient to gloss over this issue continuously in the
hopes that at some time in the future standard definitions will somehow be
devised and agreed upon by a reasonable majority of those in the drug-abuse
held. A Delphi study specifically lelaled to the formulation of standard
definitions is presently being planned.
i
'fable 4
i
133
respondents’ objectives for participating in the study
i ■ | j
■I
T_o test the Delphi technique; determine the value of
the jprocess in sharpening views in social policy fields
or in bringing forth practical
ideas3 or new insights.
-- ---
• ]
To explore the limits of possible public policy formujation; to learn more about policy formulation.
O'
j
.- H J
To be involved in the formulation and development of
drug abuse policy; influence policy through identifi
cation of critical policy distructions; to share in a
process that may lead to wiser process than we now
have in the drug abuse area; help develop policy view
in a nonpolitical forum.
.
■
To see if jany
---------consensus is possible in drug abuse
policy; see if
— the group can solve the problem or give
direction.
Second Questionnaire
Preparation of the second questionnaire began shortly after the first completed
Round One questionnaires had been received. It was decided that, because of
the complexity and time required for completion of the transition matrix, this
section would be deleted from the second questionnaire and included as part of
a subsequent round. The second questionnaire and a summary of the first
round results were disseminated in mid-May.
I he second questionnaire included two sections:
1.
2.
National Drug-Abuse Policy Objectives. Respondents were asked to rate fifty-five objec
tives on the basis of feasibility and desirability, and to rate the importance of the key
indicators associated with them.
Policy Issue Statements. Respondents were asked to re-rate the original twelve issue
statements; rate the narrative comments associated with them; and rate the fifteen
new issues suggested by respondents.
1 here is usually a decrease in response rates for the second round of a Delphi
study, particularly those involving voluntary participation. For this reason, and
because of the length of the Round I wo questionnaire, we anticipated a
response rate of approximately 45 percent to 50 percent. In fact, twenty-five
respondents, 71%) completed the questionnaire; a most unusual and gratifying
response rate, and one higher than that for Round One. (See Table 2.)
1 he absolute range of time to complete the second questionnaire was 1 to
81 hours; the interquartile range was 2J to five hours; the median time to
complete was three hours. For the policymakers' subpanel, the median was
5}
hours; for all others the median was three hours.
To distill and synthesize the collective thinking of
sgme of the best minds in drug abuse; obtain the bene
fit of the ideas of the others as a stimulus to my own
thinking; to learn what the group knows about, and
applies in responding to drug abuse.
To assess the extent to which my views in drug abuse
coincide with or differ from those of my colleagues;
check my own opinions against those of the group.
To clarify my own thinking pertaining to drug abuse
policy; get a broader perspective.
To test ideas in a multidisciplinary* quasianonymous
environment.
go, develop priorities for the organization or agency
in which I work.
To make a useful contribution to knowledge; help solve
a problem; provide ideas that may arise from my spe
cial knowledge and experience.
To gain personal enjoyment: by responsive discussions.
To satisfy my sense of duty to participate in such
studies as an anN'vP rpCDarnhar
1
J
I
I
!,
Irene Anne Jillson
Table 5
Third Questionnaire2
Por this questionnaire, panelists were asked
lions:
I.
2.
135
FEASlBILITY/PRACTicALITY SCALE
I
. i
i
to respond to two scries of ques-
Scale Reference
Xaliunal Policy Object. There were twenty-five objectives included in this sectionthese had to be re-rated because there was a broad distribution of voting responses’
deferences ,n vottng between policy experts and policy nonexperts, or because
original objectives had been combined or divided.
'transition Matrix. In this section, the transition factors
suggested by respondents in
the first questionnaire were further developed.
1.
Definitely Feasible
Can be implemented
No research and development work required
(necessary technology is presently avail
able)
Definitely within available resources
No major political roadblocks
Will be acceptable to general public
2.
Probably Feasible
Some indication this can be implemented
Some research and development work required
(existing technology needs to be expanded
and/or adopted)
Available resources would have to be
supplemented
Some political roadblocks
Some indication this may be acceptable to
the general public
I he policy issues were not included for consideration in this round in order
to maintain expected time for completion to a reasonable level. The data from
this third questionnaire, and information gathered during previous studies of
Definitions
Ins type, w-ill be used as a basis for developing policy and program options in
uture rounds. Fhe respondents were again sent two copies of the questionnaire
and an introduction and summary volume which included results of the
previous questionnaire.
11
il
Results
3.
Objectives
First Questionaire. In the first (questionaire respondents were asked to list
------- to list up to
five national drug-abuse policy objectives (for
x ’ a five-year time frame), and1 up
to three
t‘
key indicators for each of these. They listed at
' '
total of seventy-eight
such objectives; from this list, fifty-five different objectives could be discerned,
I he 187 key indicators listed by the respondents
s were culled to 153.
Even after wetdistilled
............... the original seventy eight objectives to fifty five, there
May or May Not be
Feasible
!
■
Contradictory evidence this can be im
plemented
Indeterminable research and development
effort needed (existing technology ;'ay
be inadequate)
Increase in available resources would be
needed
Political roadblocks
i
Some indication this may not be acceptable
to the general public
still were similarities between them. Rather than risk
a possible misinterpretanon of the objectives as stated by the respondents, we decided not to distill the
4.
Probably Infeasible
Some indication this cannot be implemented
Major research and development effort
needed (existing technology is inadequate)
Large scale inci^ase in available resources
would be needed
Major political roadblocks
Not acceptable to a large proportion of the
general public
5.
Definitely Infeasible
Cannot be implemented (unworkable)
Basic research needed (no relevant tech
nology exists, basic scientific know
ledge lacking)
Unprecedented allocation of resources would
be needed
Politically unacceptable
Completely unacceptable to the general
public
objectives any further. It is difficult to be precise in a Policy Delphi, particu
larly one in a field as complex as drug abuse; we have therefore emphasized
development and creativity.
Second Questionaire. The respondents were asked to rate the fifty-five objectives
derived from their responses to Round One Objectives Section. The two rating
scales used were feasibility and desirability. They were also asked to rate the 153
indicators relating to each objective; in this case, the scale used was importance.
1 hese scales are shown in I able 5. 1 he objectives were presented in arbitrarily
defined categories (prevention, treatment, law enforcement, organization, e.g.)
simply as a means of ease in completion; these categories had no other
significance.
2Firsi Round
National Drug Al.>UM- Policy Del phi.
■
I
i
ifti
Ii
■«
Irene Anne Jillson
General Applications:
1 able 5 (Continued)
1.
2
3.
4.
5.
Highly Desirable
Desirable
Neither Desirable
nor Undesirable
Undesirable
Highly Undesirable
137
Table 5 (Continued)
DESIRABILITY/BENEFITS SCALE
Scale Reference
National Drug Abuse
s
IMPORTANCE SCALE
De fini tions
Will have a positive effect and little
or no negative effect
. Social benefits will far outweigh
social costs
Justifiable on its own merit
Valued in and of itself
Will have a positive effect with minimum
negative effects
Socialbenefits greater than social costs
Justifiable in conjunction with other
items
Little value in and of itself
Will have equal positive and negative
effects
Social benefits equals social costs
May be justified in conjunction with
other desirable or highly desirable
items
No value in and of itself
Will have a negative effect with little
or no positive effect
Social costs greater than social benefits
May only be justified in conjunction
with a highly desirable item
Harmful in and of itself
Scale Reference
Definitions
1.
Very Important
A most relevant point
First order priority
Has direct bearing on major issues
Must be resolved, dealt with or treated
2.
1mport ant
Is relevant to the issue
Second order priority
Significant impact but not until other
items are treated
Does not have to be fully resolved
1
I
'.d
■'’=i
3.
4.
5.
Moderately Important
Unimportant
Most Unimportant
May be relevant to the issue
Third order priority
May have impact
May be a determining factor to major
issue
Insignificantly relevant
Low priority
Has little impact
Not a determining factor to major issue
No priority
No relevance
No measurable effect
Should be dropped as an item to consider
■i
II
.I
1
1
Will have major negative effect
Social costs far outweigh any social
benefit
Not justifiable
Extremely harmful in and of itself
I
1
i
■
I
I,
licne Anne Jilhon
Genera! Applications:
139
National Drug Abuse
IH.
'The feasibility and desirability ratings of the objectives were analyzed so as
to develop a summary list of objectives, and determine to what extent there was
consensus or polarization.
Respondents’ ratings on the feasibility and desirability of objectives were
Group Score
Less than 1.80
Equal to or greater
than 1.80 but less
Feasibility
Highly feasible
Feasible
X
05
H
CO
c
1—I w
X o
00 T5
2
o
o
o
o
o
o
o
o
o
<u
Desirability
Xi
05
X
05
5-<
W
I lighly desirable
Desirable
«
Q
<U
c
C
X
w
||
05
1 I
x
ii
oj
C
X
Greater than 3.40
but less than or
equal to 4.20
o
C
•r4
X
than 2.60
Equal to or greater
than 2.60 but less
than 3.40
I
n
J
<V
translated into group scores by summing the scale values and dividing the total
by the number of ratings. This procedure, it should be noted, treats nominal
scales as interval data. The feasibility and desirability scores were used to
categorize objectives as follows:
May or may not
be feasible
Probably in
feasible
Neither desirable
nor undesirable
B<y xoi
w
03
<y
Undesirable
I-I
o
o
<D X
T) W
C <D
o
>>
X Q
01
<U
!
X
01
C/5
'
I
I 11
L'l
&
o
o
in
w
<u
Greater than 4.20
Definitely in
Highly undesirable
H
O
M
Objectives were first grouped on the basis of their feasibility and then sorted
X
o
on the basis of their desirability. This produced the rating of objectives depicted
in Table 6.
The twenty-five objectives which scored Highly Feasible and Highly Desir
4
<u
>•
u
Ixxx
o
Q
X
03
x
m
CM
O
CM
o
cn co w
M x <u
• IS
JII
X Q
CU
able, or Feasible and Highly Desirable, are shown as 'Tables 7 and 8, respec
M
CO
tively.
No objectives were rated “Definitely Infeasible” and none was rated as either
§
<
O
“Undesirable” or “Highly Undesirable.” These results indicate that the major
indeterminable, either because there was polarization (with some respondents
rating an objective feasible while others rated it infeasible); a broad distribution
'..I- J
■ "4
>
feasible
ity (55%) of (he objectives listed were rated al least “I'easiblc” and “Desirable.”
I he following ilcms or sets of Hems deserxe spe< lai attention because of the
distinctions in rating patterns. I he feasibility of Iwcnly-one objectives was
J
Q
co
w
g
Q
X
<v
X
05
H
3o
H
g
X
ol Jn
C x
g
X
i
<u
’h
X
00 03
•rl 01
52 X
<u
X
u)
ol
<U
is
<U X
X
0) U)
T3 01
C O
X X
<u
<u
X
X
(0
►» w
r-4 05
s
’f - 'Mi
x <d
a
i
Table 7
OBJECTIVES VOTED ’’HIGHLY FEASIBLE" AND "HIGHLY DESIRABLE"
(in decreasing order of desirability)
Desirability
Score
Feasibility
Score
Objective
—
S o
<r
o
c n
e o
** H
•— O
To conduct research on treatment modalities
and effectiveness.
1.50
1.14
To train in-line treatment personnel to
enhance their skill in helping the drug
dependent person.
1.77
1.27
To have available more adequate epidemiologi
cal estimates of a) prevalence, by drug type;
b) incidence, by drug type and individual
characteristics; c) discontinuance, by drug
type.
1.73
1.41
To develop, validate and disseminate infor
mation on efficacious programs in vocational
rehabilitation, and early intervention.
1.79
1.46
To increase research into brain chemistry
and psycho-social correlates and clinicalsocial treatment.
1.70
1.70
H
►*» H
►*• o
H 22
Vi
P
S-
|S I’ S3. 5!
I 3I
=” ? $ 2 r
e
21
ft) ft)
3 Q.
Q.
•s " -u
5;
’I
C Q. C
S n cr
v r.
** 00 tcr
n
10
«
(A
e s"
—* n
?
K- rt
S Q
rt
□ <*
<
n
ft)
ft;
O X*
<t
ft>
ft>
•
<ft)
£
?
3
3
3
’ 2
I
n
a
ft) C- ft) r- Q.
C r- ft) rt
*< ft) O (A ft)
r- O S' <
•s <A S' t* o
«A
O O rf
CA •—
Vi
n C
O r^ O
CJ rt
rt n frt K. S'
r- S ft)
< 00 r»
rt
r-» rt
o. n •*,
rt o rt,
C 3 O
00
n
(A ft) rt
Q. <A
O <
to
k
rt
t- Q. U
.
rt r,
rt
rt
•
rt
T5
Q_
rt
rt rt
3
<
"O rt rt
S' rft) «< rVi
t*
t* O O
(A rt, a
*1 rt cu
n *1 o •o ft)
n (a rt s ft) s
r»
O
3“ t* rt t* rt
ft) 3 n n
CT rt O C rt
t* CL t* r- ft)
rt
3 ft) rt
<a ft)
n 3
r* ft)
rt
0
3 £X 3
rt, K« Q. rf »-•
O
3
C
rt, C CL 3 T- rt
V. r- C
3
r» rt < r»
•
3"
tCT
t*
3
ft)
ft
t*
O
3
rt o « o
V
O
to
to
to
to
o
b
£
H
O
n
rt
ft;
►*—
§ n * zr 2
O 3
O
(S
<5
2
33
O
w a
rt ra
§
Q. »-•.
Q. ft)
O' r»
•— n
M- ft)
O
*
3
if
M O
o' ***
2.
-
rt
S' W
n e
n>
Vi tft
"O
O
-□ C
S' ft)
ft) rt
<r. rt>
4 ‘
vi
n
I
I
I
ft)
Q
U)
rt,
3
to
s:?
:3i hs
.
Si.?
””
3 § c
'
(/>
Pi
Fo'2 <
|j F
II
Q.
i. s If
2?~
s
rt (2
° O
Q. U)
Il
2’S
H"5’
ss
2,
2.8ir
S 2
ft)
S' (A
rt o
•
<
? ?■
30
O
C
1 f
to
to
to
£
g
rt,
7.
n
H
r0
3
r*
ft
t*
rt
<A
d
t—
3
t*
3
(JO
Q.
rt, oc
— rt -100
—
*-• **
n
S'
s ft) o.
oo c m ft
(A C
r* n 00 rt,
o
rt
s' ft> n
S' rt cr rt
t*
Ct*
« S' w <
to rt rt
3 v
rt
(A rt
rt CT o n
cl rt
rt
(A rt rt ft)
•
3 S' rt
ft) 3
Q. rt rt
rt
5
•u
»5
O
00
n
ft)
3
(A
k I
I ?I
to
•g s
.
h
i
H
to
o
u>
00
to
to
§
25
°|
ift SIs
cr:
to
h
a
■li
DJ
to
•—•
Fh
11 J
I
F h
p
o
n
a. t; o
cr
C rt
(A rt
rt 1
rt (A
(A
*1
rt
Hl
a-
[
g
8
5
h
i KJ
gI g
st5."I
2g "
S'
35 ?
3 5? ?
t* o c
3 n
rt c 3 rt
B '■
r*
n rt «a 3
•o^o
o
tA n
ft;
i T O C
rt
sa •
c o
rt, H
ill
t*
rt, 3
o
*1 rt
rt
O' V)
3
ft) Q.
-O
rt
ft) n
3 O r- O
rt O r- <
3 3
rrt rt w-, Q.
r»
-5
ft)
r»
O
00
t*
rt
(A
(3 -O
rt
n rt o o
rt rt *i
1
F. o
-
I
CO
n <r
o
n w
to
■?
2
i
in u
n a
O U>
n *-•
a m
O'
i'
o
■S
.■
L./- ;
k-J
00
u>
to
U)
v»
ro
L-)
—
ft>
CT
2
...
——-------- -d
--.‘J.
'■
.
L.
.
■
■
.
-
-
-
-
I 12
Irene Anne Jillson
General Applications:
National Drug Abuse
143
Table 9
Table 9 (Continued)
OBJECTIVES WHICH EXHIBITED POLARIZATION OR A BROAD DISTRIBUTION OF RESPONSES
(Feasibility and/or Desirability)
OBJECTIVES WHICH EXHIBITED POLARIZATION OR A BROAD DISTRIBUTION OF RESPONSES
(Feasibility and/or Desirability)
Objective
Objective
Scale Value; Percent voting
2
1
3
4
0
38
25
25
12
To develop a national public discourse and standards on use and.abuse of drugs.
14
27
27
23
9
To reduce non-prescribed use of psychoactive drugs.
8
34
21
29
8
To reduce the number of persons engaged in the non
medical or recreational use of dangerous drugs.
0
33
22
33
12
23
4
23
13
23
35
26
35
5
13
32
18
To provide voluntary drug treatment available to
anyone who wishes it, without government regula
tions .
35
To reduce legal-political-governmental control
of treatment, including urines; dosage; age;
identification requirements.
To reduce/eliminate criminal penalties for
personal use and possession of drugs currently
defined as "illegal".
4
5
9
23
18
5
23
14
23
18
27
14
27
14
35
13
17
13
22
3
!
30
9
26
13
22
r
To increase med-social research -- hallucinogen
for eco-systems and positive social values
through use (Ritualistic, etc.)
36*
0*
36*
21*
7*
To develop a group with a primary interest In the
problems people have with drugs.
32
14
27
9
18
Indicates that over one-fifth of respondents did not respond to
this item.
To establish the principle that whether or not an
individual uses a mind altering substance is a
matter of personal choice with minimum governmen
tal interference.
9
26
13
30
22
To organize an effective, coordinated approach to
research, prevention, treatment and rehabilitation
among all appropriate Federal, State, local and
private agencies in the health and social services
areas, thus developing the mechanism tor timely
government responses.
23
18
27
27
5
To recognize that all substance abuse planning,
programming, etc., should be administered together.
5
24
19
47
5
To minimize damage caused by government reaction
to drug use.
0*
39*
22*
33*
6*
★
To utilize outreach and coercive techniques to en
gage those not entering treatment on their own.
To establish the principle that whether or not an
individual uses a mind altering substance Is a
matter of personal choice with minimum governmen
tal interference.
To utilize outreach and coercive techniques to engage those not entering treatment on their own.
To reduce/eliminate criminal penalties tor
personal use and possession of drugs currently
defined as "illegal".
2
Polarized or broad distribution on Desirability
To achieve improvements in public education which
will enhance the quality of the individual's life,
strengthen his preference for leisure-time acti
vity not involving the use of drugs, and enhance
his capacity for meeting personal problems with
out resorting to the non-medical use of dangerous
drugs.
To consider social action approaches, as alterna
tives to treatment, including concern with housing,
employment, education, and counselling, in addition
to other novel approaches.
1
5
Polarized or broad distribution on Feasibility
Scale Value; Percent voting
Indicates that over one-fifth of respondents did not respond to
this item.
•w!‘J
(with respondents voting approximately equally for four or more of the five
scale values); or truly indeterminable (with the modal response being “May or
may not be feasible”). In only eight of the twenty-one objectives which scored
“May or may not be feasible” was this the modal response; in the case of
eleven objectives the reason was that the voting was either polarized or broadly
distributed, as I able 9 shows. All seven of the objectives which respondents
scored “Neither desirable nor undesirable” were either polarized or of a broad
distribution.
By reviewing the frequency distribution and the scale scores we were able to
identify objectives in which there was a significant voting difference between
those who rated themselves experts in national drug-abuse policy and those
who did not. Table 10 lists these items. Some of the differences were in items
that could be of major importance in the formulation of a national drug-abuse
policy; most of the differences had to do with the feasibility of attaining
objectives.
Table 10
OBJECTIVES WHICH EXHIBITED VOTING DIFFERENCES BETWEEN SELF-RATED
POLICY EXPERTS
AND NONEXPERTS (Feasibility and Desirability)
Objective
Policy expertise
Scale value; Percent voting
1
2■
I 3
4
5
Feasibility
Score
Desirability
Score
Differences on feasibility
To develop adequate alternative
models in prevention and phased
intervention.
To reduce the supply of illicit
drugs available for abuse.
Expert
9
27
55
9
0
2.64
15
69
8
0
8
2.15
9
64
18
0
9
2.36
8
25
25
17
25
3.25
1.5:
0
27
46
27
0
3.00
1.73
Non-Expert
15
39
0
31
15
2.96
Expert
27
9
55
9
0
2.46
Non-Expert
32
23
15
15
15
2.62
Expert
36
19
36
9
0
2.18
27
55
9
0
9
2.09
Non-Expert
Expert
Non-Expert
To reduce non-prescribed use of
psychoactive drugs.
To reduce prescribed use of p-y
psychoactive drugs/diminish misuse
by physicians.
To incorporate drug treatment into
standard health delivery systems.
Expert
Non-Expert
To establish an effective social
rehabilitation system for drug
abusers who have become dcsocial•ized.
To consider social action
approaches, as alternatives to
treatment, including concern with
housing, employment, education,
and counselling, in additioni to
other novel approaches.
To develop a g-' ’ a primary
group with
interest in the problems people
have with drugs.
1.2-7 J
2. ' 5
1.7 2-
I
Expert
0
30
40
20
10
3.10
1.20
Non-Expert
46
9
36
0
9
2.18
1.29
Expert
0
20
0
80
0
3.60
1.80
Non-Expert
25
25
34
8
8
2.50
1.75
Expert
30
10
40
0
20
2.70
3.50
Non-Expert
42
25
8
17
8
2.25
2.00
Expert
40
60
0
0
0
2.00
1.60
Non-Expert
100
0
0
0
0
1.58
1.00
Expert
10
10
30
20
30
2.70
3.50
Non-Expert
jU
17
25
0
8
2.25
2.00
Differences on desirability
To train in-line treatment
personnel to enhance their
skill in helping the drug
dependent person.
To develop a group with a pri
mary interest in the problems
people have with drugs.
-•-
-J-
Irene .Anne Jillson
In five cases, the modal response of the policy experts was “May or mav not
be feasible," while nonexperts voted the same objective feasible. These related
.o major strategies such as “to develop adequate alternative models in prevenon and phased intervention," to reduce presenbed use of psvchoact.ve drugs/
Jmiimsh misuse by physicians." and even “to incorporate drug treatment into
.tandard health delivery systems." Since these objectives were all held to be at
least des.rable, and since one is unlikely to propose objectives one is unsure are
achievable, bringing to light this additional information may broaden the
>ohcy options available to decision-makers. Alternatelv, it could be that the
slew of nonexperts in these cases is overly optimistic. In one case (“to consider
.oeial-action approaches as alternatives to treatment... ") the modal response of
he nonexpert was "May or may not be feasible": the police experts were sure
K was "Probably feasible.’’
Policy experts and nonexperts differed on two objectives which represent a
najor effort in the present national drug-abuse prevention strategy. Police
spoils scored to reduce the supple of drugs available for abuse" as 'T'easule," while nonexperts were less certain, scoring it “Mav or may not be
■asible.” The reverse was true of the objective “to establish an effective
social-rehabilitation system for drug abusers who have become desocialized."
obey experts were not sure if this objective was attainable and scored it “May
r may not be feasible"; nonexperts, on the other hand, scored it "Feasible.”
Only one objective exhibited a voting difference between policy experts and
anexperts on both feasibility and desirability. This objective (“to develop a
■loup with a primary interest in the problems people have with drugs") was
cored "Undesirable” and “May or may not be feasible by policy experts, but
)csirablc and "Feasible” by nonexperts.
Although there was a big difference in the desi,ability scorn between policy
spells and nonexperts on the objectiv,- "m train in-line treatment personnel to
ihance their skill in helping the drug-dependent person." this was mostly m
gicement with 40 percent of experts and 100 percent of the nonexperts voting
ms objective as “Highly desirable."
Objectives that score "Probably infeasible” or "May or may not be feasible”
md this scale value was the modal response),, were dropped from considera•.on, unless there was a significant difference in voting between policy expert
id nonexpert.
■ mtd Qiiestionaire. in
were listed
In this questionaire.
questionaire, twentv-five
twenty-five objectives
(' ’
hich required revoling
(desirability
and
feasibility).
Objectives
are
presented
oting
icvoting because of polarization on the part of the panel; because there was
> broad distribution of voting responses; or because there were differences in
ting between policy experts and policy nonexperts. In some cases, original
■bjectives were combined or divided after respondents’ comments had been
viewed; in this instance, voting was required on the newly developed objec-
National Drug Abuse
|
<
|
General Applications:
147
live. The remaining objectives will be held over until a later round. Considera
tion of the key indicators associated with objectives rated at least feasible and
desirable will also be held over to a subsequent round.
Transition Model and Matrix
First Questionnaire. Social policy is the result of multiple interacting forces,
Policy is often seen as being based on advocacy rather than derived uom a
careful analysis of em;piric findings. Although policy may have to be developed
even in the absence of information, a rational examination of the bases on
which policies have been built is a fruitful way of providing the policymaker
with insight to develop more appropriate
riate nolicies
policies.
In the case of drug abuse, the factors which cause people to pass through
various states of drug dependence can be systematically examined. Such
examination allows the policymaker to estimate the importance of specific
variables, and the extent to which they are subject to his influence. A systematic examination of factors also allows any counterintuitive effect of the
variables to be brought to light.
In this part of the study we hoped to elicit from respondents factors which
control the rates of flow from general population through the various states of
drug abuse. The simplified model shown in Fig. 1 developed for this purpose
was intentionally simplified to allow for examination of the five transition states
included in the matrix. More complex (and probably more realistic) models ■
would have diverted attention from the question at hand.The five transitions in
whlnK
____ _________
which we are particularly interested
are:
• General Population To Potential User
• Potential User To Experimental User
• Expcrimcnial User To Occasional Abuser
• Occasional Abuser 'Io Drug-Dependent Person
• Drug-Dependent Person To Formerly Drug-Dependent Person
Respondents were asked to list the factors which affected each of the
transitions and state whether a specific factor increased (promoted) or de
creased (inhibited) the rate of flow of individuals from one state to another.
1 he number of factors listed by respondents ranged from a low of two to a
high of over forty; we identified a total of 128 distinct factors.
Using the criterion that a factor must register three votes for a single
single
transition, twenty-five significant factors were identified from our total list of
128 factors. The twenty-five are shown in Table 11, the vote is shown by
transition state. The table shows the number of respondents who thought that
the factor increases the transition rate from one state to another, the number
who said it decreases the rate, and a residue who did not indicate direction.
Because the number of votes is small, and also because the interpretation of
respondents indication of direction was sometimes difficult, reference will be
’
1
• rf
IM
i i
I -J
iwl
j
i
Ii
■ m
I
I
1
uc
1■ i
I
1
149<
DRUG INVOLVEMENT TRANSITION MODEL
3
Table 11
THE TWENTY-FIVE TRANSITION FACTORS RECEIVING AT I EAST THREE VOTES ON A SINGLE TRANSITION
(presented by factor category)
'’W
Factor
GENERAL
POPULATION
1
POTENTIAL
USER
EXPERIMENTAL
USER
age
Response
Transition
*l(General
Population
to Poten*
tlal User)
Total
Inc
Dec
EW
4
0
0
LV
0
#2 (Poten
tial User
to Experi
mental
User)
4
^(Experi
mental User
to Occa
sional
User)
5
^(Occa
sional User
to Drug De
pendent
Person)
#5 (Drug
Dependent
Person to
Former De
pendent
Person)
4
4
peer pressure
0
3
0
0
2
0
0
0
1
0
2
3
4
6
5
0
8
2
0
0
4
0
0
4
0
0
Total
Inc
Dec
EW
religious faith/
training
Total
Inc
Dec
EW
availability of effec
tive stress-relieving
alternatives to drug
use/lack of boredom
Total
Inc
Dec
EW
3
2
0
2
0
Experimental user to
Occasional user
opportunity for meaning
ful social definition/
availability of adult
roles for adolescents
Total
Inc
Dec
EW
1
0
0
0
1
0
Occasional user to
Drug dependent person
feeling of social in
adequacy or alienation/
alienation from stan
dards of adult-society
Total
Inc
Dec
EW
inability to resolve
frustrating personal
prob liras
Total
Inc
Dec
EW
personal stress/
situational crises/
increase in intensity of
day-to-day pressures
Total
Inc
Dec
EW
2
0
I
protracted adolescence/
youth
Total
Inc
Dec
EU
0
0
OCCASIONAL
USER
General population to
Potential user
Potential user to
Experimental user
DRUG DEPENDENT
PERSON
Drug dependent person to
Formerly drug dependent person*
State of drug involvement
Rate of transition from one
state of drug involvement
to another.
one should not presume that this is a permanent state;
it indicates that point at which, either through
treatment or self-denial, one is no longer drug
dependent.
FORMERLY DRUG
DEPENDENT
PERSON*
!
poor school achievement/dropplng out of
school
Total
Inc
Dec
EW
meaningful and
satisfying life role/
econixnlc opportunity
Total
Inc
Dec
EW
5
5
0
0
2
0
2
0
o
0
2
0
o
5
0
0
1
2
0
0
0
1
0
3
3
2
0
3
3
2
0
2
0
0
2
2
2
0
0
14
14
(12-)
0
’8
•I
(23-)
|
14
0
2
(12-)
I
1
3
3
2
2
0
0
Lsa
1 14 I 1
I <«->|
0
1
2
0
1
2
2
0
0
o
1
3
2
1
1
0
0
0
2
0
1
0
2
1
2
21
(25)
2
0
0
0
0
1
0
1
3
i,
2
3
4
2
10,
(5)
2
0
1
0
2
0
0
2
0
2
1
0
2
0
i
21
<5)
2
3
3
3
(2)
3
0
0
2
0
1
»
2
0
0
3
2
3
0
3
2
0
o
0
0
0
0
6
4
5
3
1* I •
0
4
0
J
2
>
3
2
3
4
3
•
,
0
5
0
:
■
-
(«)
8
0
proportion of
drug users Ln
peer groups
Transitions:
1
Total
Inc
Dec
EW
K.
(Rank)
2
4
0
0
■ breakdown
2
Tot*l
.
•«or«
2
0
11
(IS-)
I
!
Inc ■ Increase
Dec - Decrease
EW - Either Way
I
I MJ
Irene Anne Jillson
General Applications:
Table 1 I (Continued)
Response
specific drug effects/
pharmacological
properties of drugs
of abuse
Total
Inc
Dec
EW
availability of (illi
cit/euphoria producing/
anxiety reducing) drugs
Total
Inc
Dec
EW
supply of drugs/
heroin
Total
Inc
Dec
EW
advertising of drugs
In the mass media
especially on TV
Total
Inc
Dec
EW
dissemination of re
search results/reported
research data (eg.
effects of drug use)
Total
Inc
Dec
EW
drug abuse education
programs
Total
Inc
Dec
EW
Transition
fl (General
Population
to Poten
tial Eser)
92 (Poten
tial User
to Experi
mental
Vser)
2
sional Vser
to Drug De
pendent
Person)
*5 (Drug
Dependent
Person to
Former De
pendent
Person)
2
Total Impact
Score
and
(Rank)
15
0
0
0
10
(11)
13
13
9
0
12
0
12
0
1
2
0
0
0
0
0
2
0
0
0
0
0
10
10
8
0
8
0
(1)
20
0
2
0
0
0
1
0
(7)
2
2
0
0
2
0
0
6
8
(23-)
2
11
0
(18-)
0
2
24
3
(4)
availability ot
cuusauulty baaed
t reatment/access to
treatment centers
Total
Dec
EW
0
0
severity of legal
sanctions/laws against
use/legal prohibition
of drug use
Total
Inc
Dec
EW
3
0
enforcement activities/
law enforcement pressure/applleatIon of
legal sanctions
Total
Inc
Dec
EW
public acceptance of
legal drug taking
behavior
Total
Inc
Dec
EJ
economic social
deprivation
Total
Inc
Dec
EW
2
0
0
2
0
0
Total
Inc
Dec
EW
3
O'
0
3
0
0
number of persons
using the drug
«3(Experi
mental '.'ser
to Occas ional
Vser)
1-.
0
0
(10)
17
0
0
0
0
0
(9)
6
0
25
2
0
0
2
0
2
1
0
0
2
0
0
0
0
0
0
2
0
2
(3)
0
2
3
9
(22-)
A
10
2
2
0
0
0
0
3
0
0
0
1
0
«?
(20)
3
3
0
0
13
(16*)
151
made only to the total number of votes indicating that a particular factor
affected a given transition. It should be noted in passing, however, that for
some factors respondents appeared to disagree on the direction in which a
factor affected a transition rate. A wide range of factors made the listing; some
of them were interwoven into the fabric of society, some were clearly interde
pendent, and in other cases any relationship between factors was less clearcut.
The dominant factor was “the availability of drugs,” which scored half again as
many votes as the second factor, “peer pressure,” which in turn scored almost
half as many votes again as the third factor, “enforcement activities/lawenforcement pressure.”
‘
The top ten factors, arranged in rank order of voting across all transitions, h [
together with the number of votes cast 'for the
affecting
' factor
..... j
. a not
particular.'
transition are shown in Table 12.
- A
‘ : I:
It is important
to note that the_ votes
no -----effect” were obtained from
,
------ ---of “—
respondents who completed the matrix and who specifically recorded the fact that a !' |
THE TVEm-FIVE TRANSITION FACTORS RECEIVING AT LEAST THREE VOTES ON A SINGLE TRANSITION
(presented by factor category)
Factor
National Drug Abuse
)
particular transition was not affected by the factor they had listed. This means other ; r t
respondents may also not think a particular factor affects a transition, and so in
in ",
Ejgik
order to balance the picture the number of nonvotes is also shown. Nevertheless, it is interesting to note that even for the most influential factor (“avail
ability of drugs”) some respondents specifically said it did not affect transition r
#1 (general population to potential user) and transitions #4 and #5 (occasional user to drug-dependent person, and drug-dependent person to fformerly
\
dependent person). The factor with most disagreement was “drug-abuse educa
tion" (all transitions). We presume that this item refers to drug education as it
is practiced now, rather than what its potential impact might be if practiced
correctly. It is particularly noteworthy that no less than eight respondents
specifically said that “availability of commynity-based treatment” did not
affect the transition from general population to potential user, and from
potential user to experimental user. This number of “no effect on ” votes was
twice as high as for any other factor, and particularly interesting in that not a
single respondent stated that the factor actually affected these two transitions.
The wide range of impacting factors and the fact that some of them are
closely related to the kind of society in which we live illustrates the problem of
1
creating specific drug-abuse programs.
t r\ If
;2 ( ® -- i' * /Sli
S
Inc ■ Increase
Dec • Decrease
EW • Either Way
¥
L
06870;
Second Questionnaire.
The transition matrix was not considered in this round.
Third Questionnaire. In this questionnaire we examined the variables that
influence the transition from one state of drug involvement to another. Re- i
spondents were asked to rate the twenty-five variables that were most
frequently cited in response to the first questionaire; ratings included estimat
ing the importance of each factor in controlling a particular rate of transition,
and the extent to which these factors can be influenced by the national
drug-atbuse policymaker (manageability). These data ,m then be used by
( able 12
np TEN FACTORS ACROSS ALL TRANSITIONS
'ac tor
Response
Transition
#l(General
Population
to Poten
tial User)
#2 (Poten
tial User
to Experi
mental
User)
/^(Experi
mental User
to Occa
sional
User)
#4 (Occa
sional User
to Drug De
pendent
Person)
#5 (Drug
Dependent
Person to
Former De
pendent
Person)
vailability of illicit/
•uphoria producing/anxiety
•educing)drugs
Affect
No Affect
No Response
10
3
4
13
0
4
13
0
4
10
2
5
8
4
5
■eer pressure
A f1ec t
No Affeet
No Response
6
1
10
8
0
9
8
1
8
7
2
8
6
3
8
nforcement activities/law
nforcement pressure/
pplication of legal sanctions
A fIcct
No Affect
No Response
4
2
11
5
1
11
5
1
11
5
1
11
6
0
11
rug abuse education programs
A f feet
No Affect
No Response
6
2
9
7
1
9
5
3
9
2
6
9
4
4
9
reportion of drug users in
Jer groups
Affect
No Affect
No Response
5
0
12
4
0
13
4
1
12
4
1
12
4
1
12
failability of effective
iress-relieving alternatives
' drug use/lack of boredom
Af feet
No Affect
No Response
5
1
11
5
1
11
5
1
11
3
2
12
3
3
11
^pply of drugs/heroin
Af feet
No Affect
No Response
2
1
14
4
3
0
14
2
1
14
9
0
8
Affect
No Affect
No Response
4
0
13
11
4
1
12
3
2
12
2
3
12
vereity of legal sanctions/
ws against use/legal
ohlbition of drug use
Affect
No Affect
No Response
4
1
12
5
0
12
3
2
12
1
4
12
4
1
12
ailability of community
sod treatment/access to
eatment centers
Affect
No Affect
No Response
0
8
9
0
8
9
2
5
10
5
3
9
9
0
8
0
13
5
1
Ln
W
Irene Anne Jillson
General Applications:
cspondents to reformulate priorities for national policy, and develop programs
uom the objectives and policy-issue-staiements sections of this study
Respondents were also asked to rate the critical.ty of the five separate
lansitions from one state of drug involvement to another, in terms of the
priority each transition should receive as part of the national drug-abuse policy
Respondents did this by allocating one hundred points over the five transitions
o as to reflect their priorities. One of the intents of this question is to develop
dteinative methods of determining program-funding priorities.
Policy Issue Statements
:-'trsl Questionnaire. For the first questionnaire, j___
panelists were asked to respond
io twelve should/should not statements, constructed
--—J on current policy issues in
ic <drug-abuse
...........................
held, and to indicate the importance of these statements,
Respondents were also asked to list i:issues they thought were sufficiently
mportant to be included in the study.
ho mid
arguments lor lliese issue stmemems ue,e m have been elieiied
' l-.ul ul lhe second q uesl lot m.m. ; m f.ui, di. lespondents used dun fust
.oesuonumre comments regarding <-a< h of die stale,m ots to do just that \
iial of 184 separate comments were made, and were fed back as part of the
•cund questionnaire; respondents were asked to rate their importance.
A total of forty-six additional policy issues was suggested by respondents in
V hrst questionnaire; once again, these seemed to exemplify the diverse and
implex nature of this field. From this list we were able'to formulate an
additional sixteen policy issues in the form of should/should not statements.
(Jiu-stionnaire. The issue statements section of the second questionnaire
eluded a statistical summary of responses from the first questionnaire, and a
the narrative comments made by respondents from each issue statement,
-lie respondents were asked to revotc the issue statements, and rate the
irrative comments with regard to importance. In addition, thirteen of the issues
■osed by respondents themselves in the first questionnaire were fed back in the
im of should/should not statements for initial voting. It should be noted that
ice of the original issue statements were rewritten for clarity, following the
cd back from the respondent panel.
In preparation for this questionnaire, the analysis or the policy issue statethe policy issue statems was confined to comparing the group’s response in Round One and
<ound I wo on those items which were common to both
rounds (no major shifts
re observed in the group’s position); and to <comparing the Round Two
spouses of ^elf-rated policy experts to those of
_f nonexperts. Both rating
spouses and importance ratings were compared.
Respondents were almost unanimous in thinking that “Treatment programs
‘U.d „ eF trealment for m°re than one type of dependence within the same
ici ity and that “The federal government should allocate funds to community
ilth centers so that these centers can offer treatment for drug abuse.” In this
I
National Drug Abuse
I'
155
round, substantially more irespondents
’
- • that
felt
the private insurance plans
should be encouraged by the federal
r------—1 government
to provide coverage for the
treatment of drug dependence than. in
Round
----------— O:
One. In Round One, 42 percent
of respondents felt that marijuana ?should
’
be legalized. This support dropped to
32 percent in Round Two, but 81
-1 percent of respondents felt that “The
personal use of marijuana should be decriminalized” (a new item).
Issues were
re classified according to their importance, determined by means of
the group importance score. Table 13 lists'all thirty-one issues in order
—------.• of
importance; the most important being
bea national
w “There
— should/should
—— •—/ ~••»****»^ notwrC
registry of drug-dependent persons...Also shown in Table 13 is a compari
comparison of the way self-rated policy experts and nonexperts voted on the Round
I wo issue statements. Really major differences were observed on only three
items. I he greatest difference was in relation to the issue “Patients who enter
treatment should/should not be required to remain in treatment for a minimum amount of time”; 70 percent of policy experts voted should while 77
peri em of nonexperts voted should not. Only 50 percent of policy Xpert's voted
shoul,/ compared to <)(> percent of nonexperts on the issue “When an individual
IS referred (01 treatment in lieu of incarceration, he should/should not have the
right to choose the treatment he prefers.” On the other hand, 100 percent of
policy experts voted that “Regulations should be passed by Congress prohibiting
any and all alcohol and tobacco advertising via any media,” compared to only
58 percent of nonexperts.
There were four issues that exhibited marked differences between policy
experts and nonexperts in the importance of issues (see Table 13) Policy
experts scored the importance of the issue “Treatment programs should/should
not offer treatment for more than one type of dependence within the same
facility” 2.60, while nonexperts thought it considerably more important, scoring
it 1.67. A similar situation pertained to the issue “The manufacture and sale of
cigarettes should/should not be outlawed.” Although both policy experts and
nonexperts considered the issue not to be particularly important, experts scored
3.40, while nonexperts scored 2.42.
Maiked differences were also observed between self-rated policy experts and
nonexperts in the importance scores of these issues:
i
i
■ j
■ -'w
•I
i
II
. II
■ "’fl
"■1
I
• “All drug use or possession for personal use should/should not be decriminalized.”
(policy experts = 2.30, nonexperts = 1.50)
•
A greater proportion of the funds available for drug abuse should/should not be
allocated to basis research.” (Policy experts = 2.30, nonexperts= 1.55)
• “Persons who are identified as drug-dependent should/should not be required to
receive treatment.” (Policy experts =■ 2.40, nonexperts = 1.64)
•
Minimum mandatory prison sentences should/should not be imposed for certain
drug-trafficking violations.” (Policy experts = 1.90, nonexperts = 2.64)
•
1 he personal use of marijuana should/should not be legalized.” (Policy experts «
2.60; nonexperts ■1.83)
1 o'’'
i
AM?
/I
! i
u.e.
STATm.-.hM
RESPONSES
Table 13 (Continued)
M
157
POLICY 1SSIE STATlDdENTS:
RESPONSES AND IMJ’ORTANCE RATINGS BY SELF-RATED
POLICY EXPERTS AND NONEXPERTS (in decreasing order of group importance)
ss-e Stajer.en:
Percent voting "should" (I)
Policy
Policy
Experts
Nonexperts
Ht i
Ii .-re SlkH LD SHOl LI.' Ni'T be a national registry e:
Jr.ig-Uependent persons (including ident i: •.tr.g in• 'i-mation such as the social security numb, r oi
i ie i nd I vi dua Is >.
1. )11 ••
.’.ere SHOl l.D'SHOULD NOT be an explicit national
;•-licv regarding drug abuse.
I Sere SHOULD'SHOVLD NOT be a maximum amount ot t
l .'t an individual to be maintained on methadone.
•rivate Health insurance plans such as Blue Cross'
nine Shield SHOULD SHOULD NOT be encouraged
iid.r.il government to cover treatment o! drug
2.pendency in their insurance plans.
'■ eison.il drug-dependence SIWI LD'SIWI’I.D Nui'
c : i mi ii.i I o I 1 en.se.
Aiun an individual is referred for treatment
lieu <>l inearcer.it ion, he SHiKT.D'SH II
N il 1
l 1 <• riitht to choose the treatment '.<■ prefers.
i'u national pulley SHOi l.D'SHJi l.D NOT
'>■. a non-political group,
1.
'*1
1. ■>*>
I. ;
■ atients who enter tre itment SHOULD'SHDULD NOT be
required to remain in treatment for a minim nn
amount of t ir^-.
70
23
. 20
2.08
The p ersonal
:
..<e o: marijuana SHOULD'SIWILD NOT be
legal i.
33
31
2.60
1.83
il umg use or possession lor personal
SHOULD NOT be decriminalized.
1.
:< .tment programs SH "I LD SH ’ LU : '7 ■:
r :r
vii: :<>r more than one tvpe o! depend.nci wit
Idl iEDERAI. S •ONSORI NT. AC.f.N.'Y ' 5 I N : !.f. MA.I A; I A f
■ b< responsible : t eval utinc treat-vr:
'W
iM
Non-medlcal personnel who are employed as direct
servicers in drug treatment programs SHOl’LD/SHOULD
NOT be cert i :’l ed to
1
work in that capacity in those
programs.
80
91
2.10
A drug SHOn.Ii S-i ’ 'I NOT be subject to tight cont rols (product i on luotas, export-import regulations, prescription controls, etc.) if it has
potential
se, irrespective of its historical ab
67
82
2.22
The Drug En: .’rce.r.-nt Administration SHL'LD'
SH)' L; Nir bl dissolved (as part of a commitment
to decriminalization of drug use and in an effort
to decrease pair police practice and unnecessary
government expense).
20
9
2.40
2.18*
Regulations SliJU:.i SH) LD NOT be passed by Congress
prohibiting anv and all alcohol and tobacco adver
tising via anv media.
103
58
2.30
2.17
The interim ioial treaty actions and commitments
of the United States (Single Convention and
i’svc otropic Convention) SHO'.'LD/SHJILD NOT be rt'
moved and o ir international stance vis-a-vis
drug be radicall, altered.
44
17*
2.11
2.63*
Practicing p .sicians SHV.LD'SHOULD NOT be re■|iired to itt.n! continuing education in psycho*
phirmtcologv or .e rejuired to pass periodic tests
regarding ps.c op"irmicology.
70
62
2.50
A ci t li en-go.'.-rnment-p’-arm-ace'-itical council
SHDiLD'SH.i'. 1:;.H be iormzd to reco.nmcnJ guidelines
to the p’u civic i it leal iniaitry for its goals in
psychoictiva irug development and for its practices
in vales to ph .■sicians.
80
80
2.50
Licensi ! medical personnel who are emnloyed in
drug treatm.iit programs SHOULD/SHOULD NOT be
cert i lied to ..ork in those programs.
90
80
3.03
Special teams >■: reieral. State and local agents
SHO.'LD 'SHO u) N >1 be termed to operate across
municipal and state lines tor the purpose of sup
pressing local drug peddling.
78
Ikinuiacture .mJ -.ill of cigarettes SH ).'LD/SH)J-D
NOT be o.i tl i... i.
0
I. ■<') • • •
Personal use of marijuana SHDILD'SIK'I'L;’ Xi'T be
i ‘.ml nail zed.
The federal government SHJ'.'I.D «!»> : ’• NOT allocate
funds to communitv •.valtb centers so t at t
centers car oiler treatment tor drug .ibu». .
Importance Score (1)
Policy
Policy
Experts
Nonexperts
2.33
2.55*
2.11
11
' r.-trial detention SH i. Lh'SHi
virtain drug tratticking cases.
2.67
.
1.
8
I
2.22*
3.40
■'illinium mindatory prison sentvncis ;
'.•A be imposed :or Certain druc t r.i:
(I) Percent
those responding
Indicates that over one-fifth of respondents did not complete this item
2.11
i ’ > '.at Iona I I nst it ;ti
he combined
tor
e
I ' < lilted States SH 1
or influence other ►.>
o: crops s-.ich as opiur
(li ''ereent
iiulic.ltes
those responding
2.11 •
. iihl]
1.1
K •. I . <■ I
i|
11
1 jU
Irene Anne Jillson
General Applications:
National Drug Abuse
■
159
Preliminary Epilogue
Considerable time should be spent in coceptualization of the study design
and development and pretesting of the questionnaire. Ini any area as complex
and diffuse as drug abuse, the study-design team needs to allocate substantial
effort to this ph ase of study development.
objectives were met, and to what
extent the
the respondents
' particihat extent
respondents’’ objectivt
objectives for
pating in the study were met.• The preliminary steps in this evaluation have
already been taken: in the first questionnaire, respondents were asked to list
f°r PaniciPati^
the study, for example. At the conclusion of
the study, the respondents will be asked to measure the degree to which their , Ki
oD
’
bjeciivc5 have been reached and whether they might have developed* other
I V
...
.L
'
.....
objectives during the course of the study. In addition, they will be asked to
evaluate the study on the basis of questionnaire design, content, and other
relevant areas. The results of this evaluation will be utilized in developing an
ongoing interactive policy planning system which the author is presently
designing, as well as other specific studies which are expected to stem from the
present effort.
I*...
he evaluation of
oi the
inc impact
impact of
OI a
a study
study such
such as
as this
this is
is a
a much
much more complex
problem, but one which we believe is ultimately of more significance. We have
just begun to develop plans foi
' >r a long-term evaluation of the study. This will
include, for example, as assessment of the degree to which study results were
reviewed and considered iin the formulation of national drug-abuse policy.
Applications
A cknowledgments
The relative success of this National Drug-Abuse Policy Delphi has resulted in
considerable interest in utilization of the technique not only in the drug-abuse
Held, but in other social policy areas as well. The opportunities it affords for
idea exchange among diverse professionals and interest groups; and the con
tinuous flow of significant data for policy review are but two of the positive
attributes of the method. The potential for its application is extensive; as this is
the first study of its type, all of us who are interested in its future application
can profit from the lessons learned from this effort.
The process of conceptualizing and analyzing policy options is supremely
complex; it may be that the Delphi policy method will be a significant advance
in the field of applied decision theory and policy analysis, as it relates to the
social policy area in particular.
I should like to acknowledge the extraordinary gratis assistance of Dr. Norman
Dalkey and Dr. Murray Turoff; and that of Dr. Peter Goldschmidt and M.
Alexander Stiffman of the Johns Hopkins University; without the support and
dedicated efforts of these individuals, and the 39 respondents, the study would
not be possible.
I should also like to thank the National Coordinating Council on T
Drug
Education for sponsoring the continued development of the study desig]
m as
originally conceived and implemented.
/It sou rce Requ irements
Wil
One of the advantages of the Delphi technique as a tool in policy analysis is its
minimal cost for maximum output. The costs for completion of a Delphi studv
such as this one can range from SI5,000 to S40,000 for a nine- to twclve-month
effort, depending upon staff and direct-cost expenditures required. Eor ex
ample, if the effort is included as part of ongoing staff assignments, then staff
and space costs may not be directly chargeable; if computer services arc
available, then a sizable cost category is deleted. I he amount of data which
may be derived, and the opportunity afforded to facilitate a ‘-discussion” of the
issues by divcigcnl experts in the field, rcndci the technique unusually costel feet ive.
■ J
#I
11
■
■
II
;;
I
!
Prospectus
Phis phase of the study was completed in December 1974. During the succeed
ing rounds, the objectives will be further summarized; the policy issues on
which there is significant divergent opinion will be explored; and policy options
will be developed from the objectives, policy issues, and transition factors. .\n
interactive conference will be held at the conclusion of the study; part of this
conference will involve introduction of the computer conferencing technique to
respondents.
We shall evaluate the present effort to determine to what extent the study
I
•I
i‘>, L A Delphi Evaluation ol Agreement
between Organizations
General .Applications:
GHESTER G. JONES
introduction
Delphi [Ij is often used to combine and refine the opinions of a heterogeneous
group of experts in order to establish a judgment based on a merging of the
information collectively available to the experts. However, in this process it is
possible to submerge differences of opinion and thus suppress the existence of
uncertainty. In many situations it might be advisable to run separate Delphis
using more homogeneous groups of experts in order to highlight areas of
disagreement. This paper will report on an activity that did just this and point
..ml several areas in which the types of responses obtained were fundamentally
very different. In some cases these differences were quite unpredictable, and so,
a highlighting of the variations greatly increased the information obtained.
Running one Delphi using a subset of the experts from each group would
probably not have illuminated some of the differences in opinion. The mere
weight of pressure lb move toward the median response [2] would have caused
a joint Delphi to converge toward a middle position. In addition, the presence
ol disagreement is much more significant when large groups share similar
positions. 1 he traditional approach to Delphi generally results in the using of a
small number of experts from any one area.
One concern that is often raised about the credibility of Delphi results is that
individual experts may bias their responses so that they are overly favorable
toward areas of personal interest. 1 his is of particular concern when experts are
asked to evaluate areas in which they arc presently working and when the final
Delphi results could impact the importance attached to these areas. In this
papct results will be presented that indicate that no such bias occurred in the
Delphis reported on. It appears that the particular groups of experts used were
able to rise above the desire to protect personal interests.
Background
1 he United Stales Air Force presently maintains an official list of System
Concept Options (SCOs) in order io indicate to the Air f orce Laboratories
potential future technology needs. This activity is primarily a means of com1 he Delphi Method: Techniques and Applications. Harold A. l.instonc and Murray Turoff (cds )
ISBN 0-201-04294-0; 0-201-04293-2
Jopyright © 1975 by Addison-Wesley Publishing Company, Inc.. Advanced Book Program.
All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or
transmitted, in any form or by any means, electronic, mechanical photocopying, recording, or
otherwise, without the prior permission of the publisher.
Agreement between Organizations
161
municating to the laboratory planners the thinking of Air Force System
planners. However the number of potentially worthwhile systems possibilities
and thus the number of technology needs, exceed the resources available to
fulfill all the possibilities and needs. Clearly the Air Force Laboratories needed
a means of establishing priorities for the System Concept Options. Thus it was
decided to undertake a program of Delphi evaluation. This program was run,
by the Deputy for Development Planning, Aeronautical Systems Division, and!
was limited to considerations of those SCOs that fell under the Deputy’s'
jurisdiction. Thirty SCOs were evaluated. They covered'a rather large spread
m need for technological support as well as proposed mission use. Some'
concepts represented a rather straightforward extrapolation of present technologv-, while others would require substantial technology development pro
grams. I he missions represented included most of the areas of interest to the
Air Force including many strategic and tactical possibilities as well as systems
intended to meet support and training requirements.
It was decided to conduct separate Delphis utilizing personnel from various
Air Force organizations, in order to determine how closely the organizational
opinions agreed. In this way it was believed that not only would a I ..isis for
prioritizing the systems be obtained, but in addition, the results would help to
indicate areas of communication problems between organizations. If organiza
tion viewpoints in a particular area differed greatly, there would appear to be a
need for increased communication about the area.
Delphis were conducted within the following four USAF organizations:
Deputy for Development Planning, Aeronautical Systems Division (ASD/
XR); Air Force Avionics Laboratory (AFAL); Air Force Aero Propulsion
Laboratory (AFAPL); Air Force Flight Dynamic Laboratory (AFFDL). The
experts chosen were senior managerial and technical personnel (both civilian
and military), and were selected so that representation of most if not all of the
major departments within the organizations was present. A total of sixty-one
experts look part in the evaluations which involved three rounds of questioning.
The above organizations are of two different types. The Deputy for Devel
opment Planning is a systems planning organization having responsibility for
identifying promising aerodynamic system concepts and defining them to the
point where development decisions can be made. It has no direct responsibility
for research activities. The three laboratories are responsible for developing
technologies in their assigned areas which will improve system capabilities. The
Avionics Laboratory is concerned with electronic systems, the Aero Propulsion
Laboratory with atmospheric engines, fuel, etc., and the Flight Dynamics
Laboratory with aircraft structures, controls, aerodynamics, etc. Thus the four
groups that were asked to evaluate the list of SCOs are quite different in their
areas of expertise. In particular it should be emphasized that the laboratory
groups were being asked to compare SCOs some of which required considerable
support from their particular laboratory, others of which required little or no
.‘i-
I
3
J-
’ I
ioi
I,
!
a
■ i
I
I
I
i
3
160
•!
J
Chester G. Jones
support. All of the participants were, however, senior Air Force personnel and
were thus knowledgeable of activities at other Air Force Laboratories.
Results obtained for three of the questions used will be discussed in this
paper.
Question /. Please rank-order the SCO list of systems on the basis of where
tuiicnl Air I'orce Laboratory Programs will make the greatest contribution
toward success of the system.
Question ■/. Given that each system becoims a technological success, rank
order the SCO list in terms of importance of each system to National Defense.
Question 5. Considering technology, timing, and system importance, rank
order the SCO list according to where you think the Air Force Laboratories
can make the greatest contribution to National Defense.
Lach of these questions involved a complete ranking of thirty items, which
proved to be a trying but not impossible task. It should be noted that
succeeding round changes in answers often required a large restructuring of the
list. 1 hat is, a change in the answer or rank of one system generally changes the
ink of other systems (however, the participants were allowed to use a limited
number of ties if necessary and thus a few participants avoided this problem).
1 his interrelation of answers tends to make convergence difficult, since dis
agreement in one area impacts other areas.
General Applications:
One indication of the effect of a Delphi experiment is the amount of conver
gence caused by the iteration process, where convergence is a measure of how
much more agreement is achieved on succeeding rounds as opposed to the
first-round response. In this effort, one measure of convergence was the change
in the spread between the lower and upper quartile values for a given question
and a given SCO. In all of the Delphi experiments the spread between the
lower and upper quartile values generally showed considerable reduction
during the course of the efforts. However, as indicated in Fig. 1, the average
amount of convergence varied considerably from group to group for some
questions.
All of the groups achieved basically the same degree of convergence for
Question 1. However, the convergence on Questions 4 and 5 follow significantly
different patterns. In particular, the z\SD/XR group achieved less convergence
on Questions 4 and 5 than that achieved by the laboratory groups. The
ASD/XR group was the only group primarily composed of system planners
and so this group’s failure to converge as well as the other groups on Question 4
would seem to be quite important. The ASD/XR group should be the best
yuited to serve as experts concerning Question 4. Thus, for Question 4 the
greatest uncertainty is associated with the most expert group. If one Delphi had
I
163
*
3
AFAL-
■w
■AFAPL
? 2
8
AFFDL
(t
I
ASD/XR
Question 1
V/-1--------- J---------- 1______ l
i
l
Average Spread between Quartiles
3
AFAPL
.A (liI
I
ASD/XR
2
AFAL
AFFDL
g
cr
1
Question 4
•
y/-1---------- 1—-—i----------- 1------------ 1_______l
•1
Average Spread between Quartiles
3
AFAPL \
AFAL-
? 2
Convergence
Agreement between Organizations
Y\ AFFDL
X
ASD/XR
8
or
1 -
Question 5
-//J------------- 1------------- 1________ i________ i________ i
Averoge Spread between Quartiles
Fig. 1. Average interquartile spreads for questions 1, 4, 5.
been run combining experts from each of those groups, it appears possible that
the greater convergence between the laboratory experts might have caused a
considerably better overall convergence than that shown in the ASR/XR
result. Thus the relatively less expert participants might have caused the
creation of a false sense of expert agreement.
Correlation between Questions
In reviewing the results, it was obvious that some groups tended to give SCOs
similar rankings for different questions, while other groups changed many of
the SCOs rankings drastically from question to question. Table 1 shows the
■
i
I
:. ‘ if
Chester G. Jones
•Spearman rank correlation coefficient for each Delphi for each combination of*
questions.
Fable 1
Spearman Rank Correlation Coefficient for Each Question Combination
QUESTIONS
ASD/XR
ARAL
AFAPL
AFFDL
Q1-Q4
Q1-Q5
Q4-Q5
+ .295
+ .315
+ .746
+ .788
+ .863
+ .925
+ .571
+ .904
+ .579
+ .448
+ .698
+ .844
Clearly the ASD/XR answers suggest a greater change in laboratory
emphasis (as shown by the low correlation between Questions 1 and 4, and
between Questions 1 and 5) than that indicated by the other three groups. The
system planners thus indicated a greater need for laboratory redirection than
the laboratory personnel. Again we have an area of disagreement that might be
camouflaged had one combined Delphi been utilized.
Il is interesting that the AFAPL results indicate the least correlation between
Questions 4 and 5. Although it might seem that the answers to these questions
should correlate closely, there are several possible reasons to explain lack of
correlation:
:\SySlem may be iinPonant but not need substantial laboratory support.
(2) I he necessary laboratory support might best be supplied bv non-Air
Force Laboratories.
(3) /\ system might be important if technologically feasible, but the
neeessary technological develop!mcnts might not be considered likely in the
near
future.
Thus there might be a logical explanation for this lack of correlation.
However, the data are surprising enough to indicate the desirability of a more
<Ktai<(l !(\icw of the A1APL results. A subsequent review of the AF/M’I
answers indicated that many of the comments used to justify the apparently
imonsislcnt results did involve considerations such as those listed above How
ever this example shows the value of looking for correlation between answers
and then, highlighting comments that justify departures from expected correla
mm
tion.
Bias by Time Period
Figure 2 shows the average evaluations for Question 5 when the SCOs are
grouped according to date of estimated technological feasibility. Obviously the
General Applications:
Agreement between Organizations
165
system plannersJ ASD/XR) with their more futuristic interests attach greater
importance to the far-term,. more
advanced
----------------—I systems. This might be a result of
the planners’ greater awareness of the possible benefits these futuristic systems
offer. However, a possible reason for the laboratory viewpoint might be a
greater appreciation of the difficulty associated withi solving the technological
problems.
i
24 1974-1976
20
1976-1980
1980-
cn
c 16 -
c
o
cr
12 -
cr>
o
< 8
;
cr
4
Q
<
X—IxCL: :: SQX
i
■ II
j;‘ .A
■cr
Ila
•o
■CL!
:cn
_
Fig. 2. Average responses for Question 5 when SCOs
are grouped by year in which
engineering development could start.
■i!
Again the results suggest the possibility of a communications gap. Both
groups should benefit from an exposure to the reasoning that led to such diverse
results. This type of exposure might best go beyond a Delphi-type exchange
(which is generally limited in the amount of information transferred). Such a
transfer of information is essential if the potential value of the SCO list is to be
achieved. It is often not enough to establish priorities, unless all parties
concerned accept and understand the logic that led to the priorities.
I
I
• J'
Laboratory Bias
There was some concern before the laboratory efforts were started that the
: .I
•
Chester G. Jones
General .Applications:
icsults might tend to be biased. Although the laboratory participants were
instructed to rank the laboratory efforts by the total efforts from all the Air
Force Laboratories, it was hypothesized that the participants' greater knowl
edge about their own laboratory programs and the natural tendency' to
promote one’s personal interests would lead to a bias in favor of their
laboratory s efforts. In order to test this hypothesis, the rankings obtained on
Question 5 for the SCOs that received crucial support from each of the
laboratories were compared.
In mid-1972, each of the laboratories published reports that reviewed their
Technology Planning Objectives (TPOs) and the relevance of each TPO to
each SCO. I he top relevancy category indicated a TPO that the laboratory
felt was essential to a given SCO. I able 2 shows the average ranking given for
Question 5 to the groups of SCOs having a top relevancv match with the
various laboratory' 1 POs, respectively. The lowest number in each column
indicates the organization placing the greatest emphasis on that laboratory’s
piogiess. 1 hvrefore, bias would be indicated if the lowest number in a given
laboratory's column was on the row corresponding to that laboratory’s Delphi.
I he Delphi conducted in Laboratory B gave poorer (larger numerically)
rankings to SCOs that were felt to be essentially related to one or more of their
1 POs than any of the other groups, while the Delphi conducted in Laboratory
A gave neither the poorest nor the best rankings to SCOs that were felt to be
essentially related to one or more of their TPOs. Although the Delphi con
ducted in Laboratory C did give the best (numerically lowest) ranking to SCOs
considered to be essentially related to their I POs, the average ranking is not
too diffetent from those obtained in the other Delphi's.
I he results discussed in this paper indicate information that was obtained by
comparing several Delphi experiments utilizing experts from different organiza
tions that probably would not have been obtained had one Delphi been rut}
utilizing a subgroup from each of the groups of experts. Clearly differing
organization viewpoints were identified, despite the fact that all of the groups
involved very senior Air Force personnel who shared access to a considerable
common information base. That is, all of the organizations had detailed
knowledge of many of the same programs.
A noticeable difference in the amount of convergence was observed where in
one case the apparently more expert group showed the poorest convergence.
Disagreement was also apparent concerning the question of whether or not the
laboratory programs should be redirected (as well as the related question of
whether laboratory efforts should be directed toward near-term or more
futuristic technology needs).
Comparisons of results were also made to determine if th.' laboratory group
gave answers that were biased to support their own program. This investigation
failed to show the presence of any real bias. This finding is very encouraging,
for it suggests that at least these groups of technical experts were able to place
their professional ethics above the common desire to promote personal gain.
Had this not been true, the worth of this activity would be greatly reduced.
References
' X ™ No.’ HOcXO1I959)."Cher’ "°"
2' CoZa.ioCn,
____________ Relevancy Match with_____
Laboratory A
Laboratory’ B
Laboratory C
12.7
12.0
15.1
11.6
15.1
17.6
22.3
20.0
15.6
15.9
15.8
14.9
I
I bus, the hypothesis that a given laboratory Delphi lends to indicate biased
rankings for SCOs that receive crucial support from that laboratory’s effort
I
' '^1
i
•!
Summary
Average Answer to Question 5 for SCOs That Are Related to Programs of
Particular Laboratories
ASD/XR
Laboratory A
Laboratory B
Laboratory C
167
does not appear to be valid. The answers obtained from Question 5 do not
indicate the presence of laboratory bias.
Table 2
DELPHI
Agreement Between Organizations
-.“T-T
,naaa
R“d
1
r-1
I
.ill
I
ti. C. 1. Delphi Research in the
Corporate Environment
LAWRENCE H. DAY
Introduction
I he Delphi technique has become widely accepted in the past decade by a
broad range of institutions, government departments, and policy research
organizations (• think tanks' ). I hese applications are described elsewhere in
tins book, 'fhe use of the Delphi approach in the corporate environment will be
discussed in this section. Corporate utilization of Delphi is perhaps one of the
least-known aspects of the technique’s application. This is a result of corpora
tions regarding the products of their Delphi exercises as proprietary and, hence,
icstricting their distribution or description in professional literature. A review of
the long-term planning and futurist literature has revealed that few of the
corporate efforts in this field have been documented in any detail.
1 he first part of this analysis will examine some uses of the Delphi technique
in industry. This general review will be supplemented by an analysis of the
application of the methodology in six Delphi studies conducted by the Business
Planning group of Bell Canada. The Bell Canada experience will be followed
by a description of some of the problems and issues that arise when using
Delphi in the corporate environment. Phis review will conclude with some
comments on the potential future of the Delphi technique in the business
environment.
Examples of Corporate Delphi Research
General Applications:
Delphi Research in Corporate Environment
169
of high-level managers and executives in business. Thus, in this situation,
corporations are the consumers of Delphi research rather than users of the
technique itself.
Parsons & Williams Inc., an international consulting firm, conducted a
Delphi study entitled “Forecast 1968—2000 of Computer Developments and
Applications in 1968.[1|. This study was undertaken for a conference on
computer file organization held in Denmark sponsored by the International
Federation of Information Processing Societies. This study examined future
computer applications in business, the home, government, and institutions and
projected the future of specific computing and technological developments [2].
Another recent Danish conference has also used a Delphi study as an input to
panel discussion. This study, for a conference on long-term trends in personnel
management called “Delphi 71-80”, examined thirty-seven areas and predicted
“how far society would be moved in certain directions by 1980” [3].
Sponsorship of Delphi studies by groups of firms are generally examinations
of the future of an industry or an industrial segment. Current examples in this
area include reviews of the cosmetics, recreation, and insurance markets. A
“Delphi Panel on the Future of Leisure and Recreation” has been conducted
by Social Engineering Technology Inc. (SET Inc.) [4]. This multiclient study
was conducted by SE 1 for a group of companies interested in future market
opportunities in the recreational area that could develop through the impact of
cultural change.
A Delphi study on the life insurance and other personal financial services
markets is being conducted for two life insurance companies by the Canadian
consultants Ducharme, Deom & Associes Ltee. The objective of this study is to
“design a picture of the Life Insurance market and other personal financial
services in the 1980’s in terms of external environmental variables... ” [5].
■ -is
I
S
I
Industrial Grouping or Professional Association Sponsorship
Delphi studies sponsored by corporations can be classified into i'_
three categories.
1 he first category includes those studies sponsored bv an industry■ association or
a professional association.
I hese studies are usually of a broad nature and are concerned with project
ing the future of an industn- or perhaps even some broader societal field. The
logistics of this application usually indicate that the study has to be contracted
out to an independent consultant or research organization. While this type of
application does not result in the day-to-day use of the methodology in
business, the results of these Delphi studies are often exposed to a broad range
Mcthod: Techniques and Applications, Harold A. Linstone and Murray Turoff (eds)
ISBN 0-201-04294-0; 0-201-04293-2
'
Copyright © 1975 by Addison-Wesley Publishing Company, Inc., Advanced Book Program.
All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or
transmitted, in any form or by any means, electronic, mechanical photocopying, recording, or
otherwise, without the prior permission of the publisher.
i »;n
Individual Corporate Sponsorship
I his second category of Delphi research is similar to the first. The grouping
includes individual corporations who sponsor Delphi studies at research or
ganizations on subjects of general or specific interest. The Institute for the
Future (IFF) has conducted the largest number of these studies on this basis. In
the case of IFF, the study results are in the public domain [6]. Several of these
studies have been concerned with the impact of the computer/communications
revolution:
(1) The Future of the Telephone Industry; sponsored by the American
lelephone and Telegraph Co. (New York, N. Y.) [7].
(2) The Future of Newsprint; sponsored by MacMillan
Bloedel Ltd.
(Vancouver, B. C.) [8].
(3) On the Nature of Economic Losses Arising from Computer Based
Systems in the Next Fifteen Years; sponsored by Skandia Insurance Co.
(Stockholm, Sweden) [9],
i
-I
$
Lawrence H. Dav
Another IFF study sponsored by Owens-Corning Fiberglas Corp., examines
Some Prospects for Residential Housing by 1985” [10]. The research program
by Owens-Corning also produced an IFF study on "Some Developments in
1 lastics and Competing Materials bv 1985" [1 11.
An interesting IFF study sponsored by Genera! Telephone and Electronics
Corp., examines “Some Prospects for Social Change bv 1985 and Their
Impact on Time/Money Budgets” 112). Hence the various corporate-sponsored
studies at I FI- fall into general research categories (e.g., GTE, Skandia, and
Owens-Corning) and specific industrv research studies (e e AT&T and
MacMillan Bloedel).
‘
’
Delphi research has also been sponsored by corporations in other research
organizations. 1 he Danish study referenced above on personnel management
was extended by the consultants (Management Training Division of the Danish
Institute of Graduate Engineers) to three groups of employees from the printing
firm CON-FORM [13], The Pace Computing Corporation sponsored a study
by marketing research consultants to determine the potential demand for its
services [14].
1 he Delphi technique has recently conic to the attention of other marketing
research consultants. This should lead to an <expansion of corporate sponsorship
of these studies, as the market researchers will
....1 promote the technique with
customers who might not normally become exposed to Delphi or other longerter m planning techniques. One Canadian market research organization has
m entioned the technique in its periodic newsletter to clients [15]. As noted in
the fiist category, this sponsorship leads to senior management exposure to the
technique even though the corporations do not conduct the studies themselves.
General Applications:
Delphi Research in Corporate Environment
171
Ltd. on the future of the British Chemical Industry in the 1980s. This has been
discussed in several articles by Parker of Hercules [18] and used as a case
example in the book Technological Forecasting by Wills [19]. Other U. K.
experiences with technological forecasting and Delphi were referenced in a
recent article “Technological Forecasting in Six Major U. K. Companies” by I
Curill [20]. While he does not name specific companies, he notes that Delphi
has been used by: a “Glass” Company, a “Consumer Goods” Company, two i
“Chemical Companies,” and an “Electrical Engineering” Company and that ;
this is one of the most popular techniques of those companies utilizing tech- L
nological forecasting methodologies.
The medical field has been explored in a U. S. study by Smith, Kline and
French, a major pharmaceutical manufacturer [21]. Three other large U. S.
pharmaceutical companies are reported to have conducted studies as well [22].
Industries undergoing rapid change have been frequent targets of Delphi
research. The merging computer and communications fields are an example of
this phenomenon and a significant number of industrial studies have been
conducted. IBM has conducted an internal study on future computer applica
tions. ICL in England have also sponsored a Delphi study. In addition to
sponsoring the IFF research, AT&T conducted a study, “Communication
Needs of the Seventies and Eighties” (internal document) [23]. Bell Canada
has undertaken six studies projecting technological and social trends in four
main areas: education, medicine, business information systems, and “wired
city” services (all proprietary) [24]. The Trans-Canada Telephone System
conducted an internal Delphi study on future data service needs. British
Columbia Telephone is conducting a Policy Delphi with senior managers.
Corporate In-House Delphi Research
Summary: Corporate Examples
I Ins final category includes Delphi studies conducted bv research or planning
groups within the corporation itself. In this case, members of the corporation
staff become very involved with Delphi as they must master the technique as
well as use the study results. This category includes most proprietary uses of the
studies and their results, and they usually have not been published or distributed widely.
I he best-known example of corporate Delphi experience is that of TRW.
While the I R\V Delphi studies are unpublished and proprietary, a number of
papers have been published by North and Pyke on the technique’s use and
selected study results [16]. TRW’s modification of Delphi has been named
1 ROBE. The initial study was started in 1965 and resulted in a fifty-pagc
document containing a set of 401 forecasts published in June 1966 [17]. This
has been refined in a second study called “Probe II.’’ The reader is referred to
the papers noted above for an elaboration of the TRW experience with Delphi.
A Delphi study was conducted in the U. K. by the Hercules Powder Co.
I he discussion of various forms and examples of corporate sponsorship of
Delphi studies is not intended to be all-inclusive. It merely attempts to outline,
on an international basis, a few of the known examples of the scope of Delphi
usage in industry. The next discussion will center on our experiences in Bell
Canada as one example of how corporate Delphi studies have been conducted.
’S
M
• S1
* *-4
I
Delphi and Bell Canada
Background
Bell Canada is an operating telecommunications company serving the pro
vinces of Ontario and Quebec in Canada. In addition to offering voice, data,
and visual telecommunications services, Bell Canada owns a large manufactur
ing subsidiary (Northern Electric) and an R & D subsidiary (Bell-Northern
Research). There are also several other subsidiaries in the telephone, directory,
I
172
Lawrence H. Day
General Applications:
Delphi Research in Corporate Environment
and electronic components manufacturing fields. T he Business Planning Group
in Bell has the responsibility for identification of corporate opportunities (or
Table 1
threats) that will arise through changes in society and/or technology' in the
VALUE CHANGES IN NORTH AMERICAN SOCIETY
1970 - 2000
next decade or two.
The communications field is
is •in the midst of rapid change which will have a
significant impact on its intermediate and long
j term future. Highlights of these
changes include:
• merging computer and communications technologies
• regulatory changes introducing new competitive elements
• emerging visual telecommunications markets
• perceived and projected social changes
• increasing costs of investment options
173
SIGNIFICANT
SLIGHT
NO
SLIGHT
INCREASE
SIGNIFICANT
INCREASE
CHANGE
DECREASE
DECREASE
traditionalism
I
W"
HARD WORK AS A VIRTUE
_______________________________________
I he Business Planning Group surveyed these various pressures in the late
1960s as it was developing a study plan to evaluate future trends in the visual
■■
.M ? -c.
and computer communications fields. There was a distinct lack of qualitative
data on potential futures for these fields, especially in the Canadian environ
ment. An examination of various potential technological forecasting techniques
indicated that the Delphi technique would fill the perceived information gap.
Bell Canada Delphi Study Development
■
AUTHORITARIANISM
■
' i ■ ■ I??
1 <■'
MATERIALISM
1
REWARDING WORK AS A VIRTUE
—
- I
I he individuals involved in designing, conducting and managing the Business
Planning Delphi efforts have generally had a marketing background. Tins
l>«i< kgtound includes both a< adcinit training and professional experience,
I licsc background factors were iinpoitani dcicrminanls of the approach fol
lowed.
Initial steps relied upon the basic marketing approach of defining the
“market segments’’ that will have the most iimportant impact on future applications of visual and data communications. These
.
segments were chosen after
r
INDIVIOUAI ISM
|
INVOLVEMENT IN SOCIETY
A
preliminary studies of potential segments and taking account of the time and
resources available. I he final choices were future applications in the educa
tional, medical, information systems, and residential markets.
I he basic philosophy in the studies was to examine the future of applications
in these segments from a user point of view, not from the direction of
technological imperatives. 1 he initial questionnaires were prepared after ex
PARTICIPATION IN
DECISION MAKING
ACCEPTANCE OF CHANGE
the experts to suggest the most important areas of interest. Since the panelists
were actively encouraged to suggest new questions or modifications to existing
ones, the potential for significant study bias by the designers was low. This
approach also helped reduce the number of rounds required for the studies and
hence saved time for the participants and study managers alike.
’I
SELF EXPRESSION
tensive literature reviews of potential developments in each of the chosen areas.
The approach in questionnaire design was to guide the discussions in some
basic areas of interest in a segment rather than start with blank paper and ask
-I
NOTE:
■- I
I
The shaded areas above represent the median
responses from the five Bell Canada Delphi studies
noted in the footnotes. Shading over two areas indicates differences in opinions between the
various panels.
17 i
Lawrence H. Hay
General Applications:
Delphi Research in Corporate Environment
175
Next, innial questionnaires were pretested with groups of readily available
tables, threshold market penetration values of 20 percent and/or 55 percent
experts.
Ins proved to be a very valuable step, as poorly worded questions or
confusing questtonnaire design were largely eliminated before the errors eould
be tnfhcted upon the Delphi panel. This step adds time to the study and ntav
were used for the technologies. This gives the panelists and readers some feeling
for the scope of service acceptance in the markets under consideration.
be somewhat ego deflatmg at times for the study managers; however, it pays
,
good dtvtdends m higher quality, less ambiguous results, and happier panelists.
Systems, and Terminal Usage. Table 5 illustrates some of the summary results
from the medical study [28], The format and adoption thresholds were similar
Delphi Study Results—Education, Medicine, and Easiness
I he initial studies in education, medicine, and business followed a similar
to those in the Education Delphi.
The Business Information Processing Technology study examined trends in
Management Information Systems, Mini and Small Computers, Terminals and
Data 1 rocessing. fable 6 summarizes the median contusions of the panel on
fotmat. I he first part of the questionnaire asked the panelists to project their
V.ews on the long-term (thirty years! future of some basic North American
values. 1 he purpose tn asking these questions was more to help the panelists get
tn a societal frame of mind when answering the rest of the questionnaire than
to obtain the societal trend data itself. When the social trend views of the
Pcmem acceptance of various technologies both in business and in the home
29], It should be noted that this Table, like the earlier ones, summarizes only
various groups of experts, as shown in Table I. were compared after all of these
studies were completed, it was interesting to note how similar the results were
considering the diverse background of the 165 mdividuals in the various panek
( there was no interpanel communication during the studies).
Other areas of each study also explored nontechnological developments as
well as the adoption of systems to serve various applications. Table 2 illustrates
some of these nontechnical factors considered in the three studies p5]
Table 2
MEDICINE
the median statistical conclusions of the panels. Panelists were sometimes split
.
into schools of thought” on various issues. These opinion splits were often not
reflected m graphic presentation of the results. In these cases the panelists were
encouraged to debate their differences in writing through the rounds of Ithe
various studies. These differences are reflected in the reports along with
supporting assumptions and comments of the panelists. It was found that the
panelists comments and their analyses t
_____often very important modifications I
were
of the statistical projections shown in the
L — above
_ 1______ tables.
it
•I
I
Design of the Future of Home Services Delphi
Nontechnological Factors Considered in the Bell Canada Delphi Studies
EDUCATION
I he Medical Delphi explored acceptance of a number of developing medical
technolog.es. These mcluded: Multiphasic Screening, Computer Assisted
Diagnosis, Remote Physiological Monitoring, Computerized Medical Library
BUSINESS
1.Value Trends
1 \ aluc Trends
1. Value Trends
2. Evolution in School
Design
2. 'Trends in the
Medical Profession
2. (Changes in Business
Procedures
3. Changing Role of the
Teacher
3. (Changes in the
Medical Environment
3. Trends in Business
Physical Environment
The three studies outlined above provided
an important new source of input to
the Business Planning group. However,
one market area was still largely
unresolved: the future of communications services in the residence market,
Each of the above studies asked a few questions on home services but their
combined answers still left a large information gap. Determining hoy to obtain
the additional information reopened some important internal differences of
opinion on the value of Delphi for this» p--purpose.
1 he main issue revolved around the definition of what is an “expert” in the
1 he Education Delphi examined potential adoption of three basic types of
educational technologies: Computerized Library Systems (CLS). Computer
Aided Instruction Systems (CAI), and Visual Display Systems (including
nstant Retrieval 1 elevision). The summary forecasts of the panel are
shown below in Table 3 |26|. The projections on the use of terminals for input
and output purposes for CAI and CLS are shown in 'Table 4 |27|. In both
residential field. This question had developed in creating the earlier expert
panels as well. In two cases (Education and Business) the question was whether
selected industry specialists within the telecommunications industry were as
knowledgeable as experts in the above fields when it came to projecting the
future. I his question was resolved by conducting two studies in the Education
3
and Business fields. In both cases, independent panels of “internal” and
"outside” experts were used.
The question in the residential study was whether housewives or researchers
and planners were the best experts on the future adoption of communications
services in the home. In this case the study was totally service- and not
I
1
I
Lawrcnce H. Day
'I’ablc 3
General Applications:
Delphi Research in Corporate Environment
177
!'■ Th
LIKELY TIMING OF ADOPTION OF
TECHNOLOGICAL SYSTEMS
i.-
'■ -1
COMPUTERIZED LIBRARY SYSTEMS
20%
I
Primary
55%
I
]M
20%
M
55%
L
Table 4
Secondary
TERMINAL
TO BE USED FOR
CAI AND/OR CLS
J M[
■I m H
20%
Pos t-secondary
INPUT
CAPABILITY
WILL BE IN
20% OF ALL
70-75
76-80
M[
55%
81-85
86-90
91-99
LATER
never
Homes
1975
1985
1 995
CAI SYSTEMS
T ouch-Tone
Type
20%
M
P n mary
■ I
P. Schools
Tele phone
1
S. Schools
M
Home s
20%
M[
Ty pewri ter
Keyboards
] M
1975
1985
1995
J
P. Schools
■is
or
HmHI
“ Light"
55%
■
BHBi m
Homes
E lectromc
Primary
J
S. Schools
Pen s
]
—
S. Schools
P.S. Schools
IRTV & VISUAL DISPLAY SY STEMS
20%
P.S. Schools
20%
Secondary
■■■ M
Estes m
Home s
55%
1 M
P. Schools
M
Human
20%
Post-secondory
Voice
55%
S. Schools
M
] M
P.S. Schools
1975
1985
M
1 995
MEDIAN
PERIOD OF SYSTEM REFINEMENT AND EARLY ADOPTION (INTER QUARTILE RANGE!
Spec ia I ly
PERIOD OF EXTENSIVE ADOPTION (INTER QUARTILE RAnGEI
SOURCE
OOrLE AND GOODWILL, EDUCATIONAL TECHNOLOGY,
The median Symbol. •
INDICATES ’ ■ E • > E A w
LOC A T (ON T
ThE SA m E • .
Ch AG ’
I
P. Schools
Type
■■■■■■Im
55%
I
M
Se con dory
Pos t-secondory
-1191
J
P.S. Schools
55%
.
Keyboard s
55%
20%
rr- 1i
F ormed
64
tme fcllowing ones only
; i an and the first ob Third Quag t i
N SOME CASES.
P. Schools
M
Printed
' ITS EXACT
MAY BE IN
Characters
S. Schools
P.S. Schoo Is
M
M
♦
♦
Lawrence H. Day
General Applications:
Delphi Research in Corporate Environment
179
Table 4 (continued)
Table 5
PERIOD OF SYSTEM REFINEMENT AND
EARLY ADOPTION OF TECHNOLOGICAL
SYSTEMS (20% UTILIZATION)
TERMINAL
TO BE USED FOR
CAI AND OR CLS
e
OUTPUT
WILL BE IN
CAPABILITY
20% OF ALL
70-75
76-80
81-85
86-90
Homes
LATER
NEVER
TECHNOLOGICAL
INSTITUTION
mMM
P. Schools
Computer
♦
91-99
SYSTEM
■Mi MMM
S. Schools
in
P.S. Schools
81-85
86-90 I 91-99
MS
M
CAD
M
CLS
mIIm
LATER NEVER
Solo Proctice
M
0
e
76-80
Physicians
Voice
Repl y
70-75
Homes
P. Schools
Printed
♦
MS
M
Physicians in
Page
S. Schools
Group Practice Who
P.S. Schools
e
CAD
M
CLS
M
-
Share Facilities Only
Homes
Trodi fipna I
e
P, Schools
Ml
Televi sion
I
Screen
Display
Practice Who Share
P.S. Schools
MS
M
CAD
M
Physicions in Group
S. Schools
•i.
M
Records & Incomes etc.
CLS
e
e
Large Flat
P. Schools
Screen
. -if
M
Telev is ion
S. Schools
MS
M
CAD
M
CLS
M
MS
M
Acute
M
Display
Genero I
P.S. Schools
e
e
’Iv
M
Homes
*
M
I
Hospito Is
Homes
Audio
M
P. Schools
V idea
Recorder
S. Schools
Record i ng
___
___ _
M
P.S. Schools
m HMB
Homes
CAD
M
CLS
wag M
Med ic ine
M
Screen
Di splay
P.
S.
P.S.
Source
h fe
Occupot lono I
e
Pi cturephone
M
CAD-
SCHOOLS = PRIMARY SCHOOLS
CLS-
SCHOOLS= SECONDARY SCHOOLS
MMI
- MULTIPHASIC SCREENING
V//////&
- COMPUTER ASSISTED DIAGNOSIS
pt I
I
r
■
. COMPUTERIZED LIBRARY SYSTEMS
SCHOOLS - POST SECONDARY SCHOOLS
Doyle and Goodwill, Educational Technology, P. 41—2
Source
I
Doyle and Goodwill, Medical Technology, P. 60
I
1 <jL)Ic 6
An Exploration of th<- Future in Business Information Processing !►- —. ogy
1970-75
1976-80
1981-85
DEVELOPMENTS*
THE WORK LOCATION (WHI RE WHITE
COLLAR AND CLERICAL EMPLOYEES WILL
PERFORM THEIR JOB DUTIES)
Of f ICE CfNTE*
management information systems
MIS FOR I ARCE
MINI AND SMALL COMPUTI RS
MIS FOR Ml DIUM f IRMS
INTEGRATED MIS
FRF F-STANDING COv°L ’ERS
COMMUNICATION CO‘-’P._ Tf RS
DUAL-PURPOSE v.*C-‘NfS
TERMINALS
1-T II IE PHON(BUSINESS
TYPE WRITER kDb<
(BUSINESS
COMPUTER VO Ct
(BUSINESS
PRINTED P-C(BUSINES"
TRADITIONAL I. SC - t • N
(BUSINE<^
ACOUSTICAL I \ COi
TERMINUS
(BUSINf s>
'PLUG-IN PORTABLE TtP\- naLS
(BUSINESS
1-T TELEPHONE
(HOMES)
PICTURE PHONE FOR
f ACI-IO-E ACE COMMUNICATION
(BUSINESS)
TOUCH-SENSITIVE INPUT
(BUSINESS)
OCR (BUSINESS)
2-WAY LARGE SCRflN
COLOR TV
(BUSINESS)
AUDIO-VIDEO RECORDERS
(BUSINESS)
PICTURLPHONE DISPLAY
(BUSINESS)
SMAIL-SIZE EiRMS N
MANUI ACU RiNC
INDUST
WILL Bl UTIUZ'NC
D P. I ACUITIES
SMAll-SI/l f IRMS IN
SERVICE INDUSTRIES
WILL Bl UTILIZING
D P. FACILIIIES
.• )
DATA PROCESSING
MEDIUM-SIZE IIRMS IN
MANUFACTURING AND SERVICE
INDUSTRIES
WILI BE UTIUZING
D P. FACILITIES
TECHNOLOGY IN THE HOMl
I
ONE-WAY (INCOMING)
AUDIO-VISUAI COMMUNICATION
19W>-90
F
I ATE R
DEVELOPMENTS*
THE WORK LOCATION (WHERE WHITECOLLAR AND CLERICAL EMPLOYEES WILI
PERFORM THEIR JOB DUTIES)
NEIGHBORHOOD R.W.C.
THE MOBILE WORKER
MANAGEMENT INFORMATION SYSTEMS
MIS FOR SMALL FIRMS
HOME REMOII VVORk C:NT{R
3
8
I
DISTRIBUTED MIS
MINI AND SMALL COMPUTERS
TERMINALS
LIGHT PENS (BUSINESS)
HUMAN VOICE INPUT
(BUSINESS)
PICTUREPHONE FOR
FACE-TO-FACE COMMUNICATION
(HOMES)
PICTUREPHONE FOR DATA
INPUT
(BUSINESS)
HANDWRITTEN INPUT
(BUSINESS)
COMPUTER VOICE REPLY
(HOMES)
TRADITIONAL TV SCREEN
(HOMES)
AUDIO-VIDEO RECORDERS
(HOMES)
FULLY PORTABLE WIRELESS
TERMINALS
(BUSINESS)
PICTURLPHONE FOR
DATA INPUT
(HOMES)
SCRIBBLEPHONE
(HOMES)
SCRIBBLE PHONE
(BUSINESS)
PRINTED PACE OUTPUT
(HOMES)
LARGE FLAT COLOR T\
DISPLAY
(HOMES)
LARGE FLAT COLOR T\
DISPLAY
(BUSINESS)
PICTUREPHONE DISPLAY
(HOMES)
TYPEWRITER KEYBOARDS
(HOMES)
LIGHT PENS (HOMES)
HUMAN VOICE INPUT
(HOMES)
HANDWRITTEN INPUT
(HOMES)
TOUCH-SENSITIVE INPUT
(HOMES)
OCR (HOMES)
2-WAY LARGE SCREEN
COLOR TV
(HOMES)
o
=r
O
o
■5
o
B
a
□
□
3<T>
□
TWO-WAY AUDIO-VISUAL
COMMUNICATION
•THE RESULTS INDICATE WHEN THE DEVELOPMENT WILL
MEDIAN ^EXPERT" RESULTS ARE PRESENTED.
7=
cs
ft
W
DATA PROCESSING
TECHNOLOGY IN THE HOME
2=r
REACH A 20 PERCENT LEVEL OF ACCEPTANCE BY THE APPROPRIATE UNIVERSE. THE
*
i •
-
CO
I
Lawrence H. Day
General Applications:
Delphi Research in Corporate Environment
technology-oriented. I his issue was resolved bv establishing two <
competing
panels to forecast the future in this area. One panel consisted of hous^wive;
(experts through experience) and the other of experts through research or
planning for -'wired city" services. The study design and steps followed are
shown in Fable 7.
'Fable 7
183
Table 8
SHOP-FROM-HOME SERVICE
In predicting which types of products will be purchased through a Shop-from-Home
system, the housewives and experts disagreed on a number of items. The summarized
answers are presented below with the answers for the expected costs of such a service. •
Some typical comments are presented on the facing page. There appear to be significant
differences in such items as produce, small and large appliances.
i
Study Design: Future of Communications Services into the Home
TYPE OF PRODUCT:
1.
2.
4.
5.
6.
7.
8.
9.
%
%
over
5-20%
0-5%
YES
NO
20% more
more
more
E
Literature Search
H
Assemble Panels of ‘’Experts' and Housewives
Design Draft Questionnaire
Pretest Questionnaire
Print and Distribute Revised Questionnaire (Identical
F
48
____ 58_ ____ 42
52
48
produce
Design, Pretest, Print and Distribute 2nd-Round Questionnaire showing:
a) Ist-Round Statistical Results from Each Group on C
One Page
b) Ist-Round Supporting Comments from Each Group
p on Opposite Page
c) Ask for Resolution of Answers within Each Pgnel
d) Highlight Differences between Panels and Ask for Resolution
Prepare Final Analysis
__ y
■e
□ hD
■e
hH I
52
F'X
□hD
83 ___ 17
0
100
groc. dry goods
___ 96
58
42
cloth in g
48
II
■ eBHI
___ 96
90
drugs ond cosmetics
cations Delphi [30], The importance of obtaining feedback
comments from the
panelists is illustrated in the table.
large appliances
grammed education in the home. The study also explored the future of ten
types of information retrieval services that may be offered to homes. Table 9
illustrates some summary results of the study |31 ].
4
10
$
E
___ 75 __ 25
47
53
□hD
7« r
□hD
2i
r |i
NOTE: Shaded area represents significant differences between panels
E = Expert Panel: median or percentage response.
H - Housewife Panel: median or percentage response.
Do you wish to change any response? Could you comment on the differences between the
two groups? Do you have any concluding comments
ROUND II COMMENTS
Summary: Bell Canada Delphi Studies
^8
i-i^i
II
I
Business Planning efforts in the six studies outlined above have resulted in an
important increase in the availability of qualitative data for planning purposes.
Experience with the technique icsullcd in significant modifications from the
oiiginal RAND approach, especially with the emphasis on
analyzing the
oil
panelists comments and establishing thtcshold levels of acceptance. 1 he use
:■
sma 11 appl ionces
8 in 'Fable 7. The results should stimulate debates between t’
____ L i.'
i the panels if this
---------approach is going to derive the maximum benefits from both panels. Table 8
shows atypical two-page feedback and question set from the Home Communi
1 his study examined future acceptance of electronic shopping from the
home, remote banking, electronic home security services, and electronic pro
.
■□hD
25
74
The important steps that are different
C"'
from normal Delphi studies are 7 and
5-20%
ovor
less 20% loss
CZZ~1h
0
100
(dairy, bread, etc.)
less
E
5
____ 75
other peri shables
0-5%
52
me a I
to Both Groups)
Prepare Statistical Analysis of Ist-Round Answers
I repare Analysis of Supporting Comments from Each Group
same
of
(Cont’d.)
I
4
I
I
I
■ r
..' ’fc J
Lawrence H. Day
General Applications:
Table 8 (continued)
I
I
SH0P-FR0M-H0ME (CONTINUED)
EXPERTS
The panelists hod the follow.ng generally
an economic
advantage (savings on sales peop le , e • pensive store space, electnc 11 y and di>plu y ,
etc.)”
'fSI
service and its
1.
A useful service for those living in suburbs
rciuule rural ureas whos e access to C it ics
and stores is d.fhcull, also for elder!
y peoPie."
O
2.
"I don't th ink it should cost any more be-
type of product
Assumption - meat and produce
Supp I i ed
from a familiar supplier and good previous
cause it would not need a huge store or
many so le sc ler k s, and certainly shoplifting
would be a thing of the. ,past
----------for
«r anyone
experience."
Assuming a rremote shopping service were available to householders,
Better price and product comparisons con
allow the purchaser to save money as well
3.
by feeling, smelling, and seeing the
1.
specific
items in question and comparing them to
the others available in the display. This
2.
"Small or large appliances (if brand name
Produc e
Insuring quality will raise the price of
perishables above attractive cost,"
3.
"People will expect large cost reductions
for appliances in return for not being able
to see the merchandise. . .Several visits to
warehouses will be typical even with home
shopping
Other Perishables
% NO
75
100
0
25
50
75
100
_J
llI
R1
■
3.
R1
E =0
R2
Z o E =0
For most housewives, shopping is a diver
i
H
•«
R1
Grocery Dry Goods
R2
"Meat and produce - Individual specific
Clothing
Rl
o
<>
R2
"The cost of these items in our doily living
SmalI Appllancet
budget is expensive enough as it is; if it
Rl
I
R2
would cost more to buy these things at
home, nor many people would take ad
vantage of it unless they were unable to get
Drug s and Cosmetics
out of the house. . . Clothing and large ap
pliances would be difficult with this service;
Rl
!
■j
o
R2
with clothing you like to examine the fabric
to see how well it is mode; with large appliances you would wont to discuss with
i
Rl
Large Appli ances
R2
rhe salesman the pro’s-con’s of the appli-
L—
once.”
1
Source:
50
R2
quantities required; bult packaging not
always desirable.’’
tinued existence of meat markets, fruit
ttands, etc.”
I
R2
sion and a break in a routine."
cannot be done remotely, hence the con’
2.
25
f rom-Home.11
"Meat, produce - these items ore bought
[I
Meat
These comments reflect some of the typical reservations aboul the service:
I.
0
R1
is known) present no problems for Shop-
as time.*1
'I
:
would it be used to purchase these types of products?
HOUSEWIVES
participating in the service."
3.
1
How will it be used?
favorable comments on the
% YES
2.
‘■’[II
..
SHOPPING FROM THE HOME
EXPERTS
"Will succeeds only if it offers
|
185
Table 9
HOUSEWIVES
expected costs.
I.
Delphi Research in Corporate Environment
While there seemed to be general agreement between panels that grocery dry goods and
certain perishables would be purchased through a remote shopping service, the housewives
eheved that a number of other products would also be purchased with the aid of this ser
vice. The comments reinforce this pattern ("that housewives would ‘trust’ the store
3 SmaHer trUSt”)’ At the Same time however’ the comments reflect the’ ’
eeling that this service would eliminate a significant element in housewives’ social
activities. It seems that if remote shopping is to become widely accepted, housewives will
need to have recourse to alternate modes of socializing.
There did not appear to be any major change in panelists’ attitudes between rounds. The
largest sht t was the decrease in the number of experts expecting large appliances to be
purchased from the home.
Bedford, Questionnaire, pp. 2-3
(Corn’d.)
i
■ i
3
4
Lawrence H. Day
General Applications:
COMMENTS
HOUSEWIVES
ROUND ONE
"Meat, produce - these items ore bought by
feeling, smelling, and seeing the specific
and stores is d.fficult, also for elderly people.
Clothing would be difficult in my opinion as
there would be no way of trying on and fitting.
Delphi in the Corporate Environment
Endless returns and exchanges would defeat
The issues discussed below are based upon
upon Bell
Bell Canada
Canada experience
experience and
and on
discussions with individuals who have conducted similar studies in other,
corporations. I hese issues will probably be faced by any group in indu>». y that
launches a serious attempt to conduct professional
orofessional nnalitv
quality research in this area.
’he original purpose of the shop-from-home
service. Otherwise, most things could cdFvemently be bought from the home.”
want hands-on experience. The butcher has a
very poor public image. Consumers will want
"Small or large appliances <if brand name
to see how much fat the steak has.”
known' present no problems for Shop-fromHome. '*
"It’s not a function of the product but rather
how o given purchaser perceives the product.
For most housewives, shopping is o diversion
I
and wonts to be there. The cafeter ia operator
"Clothing and large appliances would be
couldn t core less. Since many stores ore our
difficult with this service, w.th clothing you
like to examine the fabric to see how well it is
made; w th large appliances you would want to
discuss
I scuss with the salesman the pro's-con's of
the appli once.
ROUND TWO
"I still don't accept the idea that housewives
will buy meat, produce, and large appliances
via shop-from-hone service. The first time the
Change my response for meat, produce and
clothing to NO because one of the main
reactions to the product is through a confronta
housewife gets burned using S-F-H (re. a poor
tion with it which results in the buying or
steak, unfresh meat, etc.), she will go back
rejecting of it."
to using the store.'
It is rother amusing that housewives would
responses were based on the belief
My 'NO
that on intelligent housewife would wish to
carefully choose meat and produce herself
trust
the store for meat and produce but the
experts hove smaller trust. Go with the houst
wives - we do most of the shopping.”
unless her butcher and grocer were intelligent
people who knew her states and preferences
wel I.'1
I still don f like the concept of shopping on a
larger scale from home. Il strikes me that this
"Experts tend towords NO, probobly because
system is geared to the larger chain stores
which already offer catalogue service os a type
they buy these products as men with a techni
of shop-from-home feature (with regard to cloth
cal bent."
ing, appliances, etc.) But I wonder how such a
service would affect smaller businesses and
Experts seem to ascribe greater wei ght to
first hand observation of product in buy >n g
decision than do housewives. Husbands may
specialty stores which definitely couldn't
operate such a service themselves."
be more similar to experts in this propensity."
Source.
Bedford, Future of Common icotions Services in the Home,
pp. 28-29.
Should Corporations Pay for Basic Delphi Research?
i;. ’-M
l-.i-
and a break m a routine."
The gourmet thinks of food as an art form,
contemporary art museums, the process of
shopping is quite complex.”
J
"A useful service for those living in suburbs
be done remotely, hence the continued
"In regard to meats, the consumer will always
I
or remote rural areas whose access to cities
items in question and comparing them to the
others available in the display. This cannot
existence of meat markets, fruit stands, etc.
187 !
Delphi to evaluate the marketability of services by users rather than predicting
the median
was also helpful An
..i dates of potential technological development
develop!
analysis of completed studies has also revealed comparison information on the!
use of internal panels vs. external panels. Thus, Business Planning has learned^
much about the technique while
while obtaining
obtaining useful
useful information.
information. This
This three-year
three-'
intensive involvement with the technique has also given Business Planners a
realistic view of some of the issues that arise when operating with Delphi in the I
corporate environment.
I able 9 (continued)
EXPERTS
Delphi Research in Corporate Environment
i
I his, of course, is the fundamental question that must be answered. The
emphasis here is on in depth research, since this will often result in a significant
allocation of time and money resources in an area where immediate payoff is
not clear to senior management. Other forms of business research (market
research, operations research, economic research, etc.), have more precise goals
and utilize more understandable techniques. The benefits of Delphi research
will not be reiterated here, but the corporate planner has to recognize that this
is one area not easily understood by busy executives.
This is part of the more basic question on the value of long-term planning in
business. Generally, long-term planning has become an accepted part of
business today. Delphi research is most needed in the long-term planning
function where the conditions of uncertainty are the most evident.
Basic research of this nature is beginning to fall into the general area of
corporate social responsibility. Many corporate decisions made today will have
important secondary effects for decades to> come. The rapid rise of interest in
government, academia, and business in what is termed “Technology Assessment” [32| is one reason for considering this as a part of corporate social
responsibility. Delphi study results can be used as corporate inputs to the
development of technology assessment equations [33].
The North American telecommunications industry is generally privately
owned but regulated by government agencies. Recent studies in the U. S. [34]
and Canada [35] have noted and projected an accelerating trend on the
sharing of planning data between the regulators and the corporations. While
many industries are not formally regulated, none can escape the growing
governmental and public scrutiny of the consequences of their actions and
;-1
I
■H
I
I
■'ll®
Lawrence H. Day
plans. Sharing of basic planning information such
as Delphi study results can
help develop a common asscssux-ni data base on both a
••• •• “’ipoialc and a public
>asis.
ri.e MKiid/pomu-.d
al)iiV(. U|1. t.S|Jt.i ia||y „ll|)i)rlanl w|i(.n
• aluaung the eost/benefn analysis of undenak.ng corporate Uelphi research
owever, this is a more obvious reason for doing this type of research: the
icsults can be used to help make business decisions.
Mng Delphi Results in Business
Many corporate Delphi studies conclude with the publication of a report to the
panelists and management outlining the study findings. The problem of recog-
‘‘TT
k
n , iS reSearCh deVel°pS if 'hat is where the DelPhi studies
T hese basic Delphi reports are important as:
(1) Educational tools to inform senior r.
managers of the panelists’ views of
potential futures or various areas of interest.t to the business.
(2) Trading documents with other plr
anners and researchers.
(3) Environmental trend documents that
...~t can help technological planners in
esearch labs.
1 he use of the Delphi results must then become more directed. One useful
ay of usmg the specific results is to regard them as a data base to be drawn
pun when preparing corporate recommendations in specific topic areas. The
Delphi forecasts should be combined with other relevant material (trend
tirapolations, multiclient study results, market research data, etc.) in order to
•resent a comprehensive estimate of the impact of a forthcoming decision 1361.
. '.ese combinations may be in the form of cross-impact matrices, scenarios
arket analyses, etc. The use of the Delphi data with other material helps
teate confidence in the overall package. It is rare that the Delphi results alone
an help resolve an issue when preparing a recommendation. Of course, this
3proach is useful in the nonbusiness environment as well.
1 he Bell Canada Delphi study results are regarded as part of a data base
•ach of the Delphi forecasts has been abstracted, key-word indexed, and stored
an on-line computerized information retrieval system. Other items in the
ua base are also stored in the same manner. These items may be forecasts
■ am trend studies, material from other internal research, appropriate forecasts
mi studies available from government institutions, policy research institutes,
orporauons, etc. The data base is used in the creation of several types of
■jsiness Planning outputs (Note: the Delphi material is an .nfiut, not an
ipui). These outputs include:
(1) Specific Service and Business Proposals designed
to exploit identified
upoi tunities.
General Applications:
Delphi Research in Corporate Environment
189
(2) Environmental Outlook Reports that identify trends and potential future
cvcnis whit h may impact on the company or a .spv< ific corjxjrate function (i.e.,
m.ii kclint’).
(3) 1 argeted ( hitputs designed to present selected material to various gov
ernment commissions, task forces, as well as other research organizations.
(4) Subject Sourcebooks and Information Packages which combine all of the
available information on a specific field of interest into an annotated document
for use by other planners in the Bell Canada Group.
(5) Methodology Analyses that document what we have learned using a
particular technological forecasting technique.
In all of the above cases, the Delphi research material has been combined,
massaged, analyzed and placed in perspective vis-a-vis other future informa
tion.
Delphi research can also be used in obtaining certain types of information
not usually available from normal marketing research activities. Statistical
polling of consumers can only produce a limited base of attitudinal data.
Feedback and interaction are not possible here. On the other hand, group
depth interviews can run into many of the problems that the lack of anonymity
produces. The modified version of Delphi used by Bedford enables the re
searcher to generate opinions and conflicts between potential consumer groups
for new products or services. This controlled conflict with feedback produces
valuable behavioral information that would not emerge using other techniques.
This data can be used for product or service modification or’ redefinition of
market opportunities.
Delphi research in business must be regarded as a means toward ends rather
than as an interesting intellectual end in itself. Use of the technique as
indicated above can result in an affirmative answer to the question: “Should
corporations pay for basic Delphi research?”
Misusing Delphi Results in Business
Delphi study results can be used to advantage in the corporate environment.
The.reverse situation is also possible. One of the most common situations is for
the results of the study to be viewed as representing a corporate position, policy,
or forecast.
This is not the case for the results of the vast majority of Delphi studies which
represent the combined and refined wisdom of the particular panel of experts
on the study [37]. One of the recurring problems with the Bell Canada Delphi’s
has been the suggestion that the studies represent a corporate position even
though this suggestion is explicitly refuted in the reports.
A related problem is the temptation of corporate public relations groups to
distribute the studies as another P. R. tool. This can be especially problematic,
■I
"w
i
- fl
I i
.. i
i
i
I
j
w
I! 1I
Lawrence II. Day
nee Delphi panelists are assured that their contributions are provided in
onfidence on a professional basis. The use of the study results in this manner
»)uld backlash on the study director, especially if he hoped to conduct future
udics using panelists drawn from the same population. Of course, the value of
he documents as trading vehicles would diminish as well if they were handled
.i this manner.
A further issue is related to the perceived precision of the study results. Many
Delphi studies process the interim and final results using computers. This
ci mils the presentation of statistical results that "appear' \ ery precise to the
tsual observer or individuals accustomed to dealing with the results of
conomic and statistical research. 1 he findings of Delphi studies are subject to
lotc interpretation than arc most research results. 1 he planning group should
-y to ensure that others using the results as a data base are aware of the
arious strengths and weaknesses of the information.
'n-1 louse vs. Consultant-Conducted Studies
\nolher question that must be resolved is whether or not to conduct the study
using in-house or consultative resources. This decision can be analyzed by
jnsidering the following factors.
(1) Single vs. Multiple Studies. There is a definite learning curve involved
vhen conducting Delphi studies. Serious attempts utilizing the technique
icquire an initial time and resource investment to learn how Delphi studies are
ffectively conducted. This investment will pay continuing dividends if a
number of studies are planned. These rewards include the development of a
..tore knowledgeable planning staff that fully understands the strengths and
eaknesses of the data obtained from the studies. On the other hand, conduci
ng a single study with a planning group unfamiliar with the use of the
cchnique may be a costly venture that produces mediocre results.
(2) Study Sophistication. The nporalion may be dealing with a subject
m alter that is changing rapidly and is very compk (e.L’., computer tcch..ology). 'The firm may also want a large number of factors considered in the
udy. in this case, the use of outside consultants who have considerable
. xpericnce in conducting large complex Delphi’s (i.e., LET.) may be more
productive. The use of these consultants will also ensure that the best modificaon of the technique is applied to the company’s problem. Experienced
consultants are constantly learning more about the technique and are modify.ig it as a result of that experience. The time delays that occur before this
xpericnce is reflected in the professional literature may mean that the
orporate researcher is using a somewhat less than optimum version of the
hnique.
(3) Proprietary Research. The problems involved with proprietory research
that are discussed in the next section are also factors to consider when choosing
ctwecn in-house and consultant conducted studies.
General Applications:
Delphi Research in Corporate Environment
191
Proprietary Nature of Delphi in the Corporate Environment
One of the usual descriptors of corporate market research is that the results are
considered proprietary. Many studies are conducted to further a competitive
advantage. Corporate Delphi research is often conducted in this environment
with similar objectives. In these instances the results of the studies are not
designed for outside consumption. This creates problems if external expert
panelists are used in the study. The usual contract with the panelist is a full or
partial payment with a copy of the study results. This usually attracts ,highcaliber panelists who are interested in adding the study results to their own
store of information. The presence of the report in turn results in dissemination
of its contents to the panelist’s professional colleagues, either by photocopy
ing or by requests to the study director for additional copies. This process of
information dissemination through “invisible colleges” usually means that
proprietary studies are not too practical with external panelists.
One solution to this situation is to utilize in-house experts. Of course, this is
practical only if there are a significant number of internal experts in the subject
matter of interest. The penalty of using this approach is the loss of the
independent outside viewpoint.
The use of mixed panels of in-house and external experts creates another
potential problem. The in-house panelist may have access to confidential
corporate market or technological research and use this in making and justify
ing his projections. The study director may have to edit this data out of the
panelist feedback material unless the company is prepared to let the corporate
information out to the external panelists. This situation can create some
intellectual dissonance for the study director, since the secret data could help
resolve specific questions under consideration by the panel. The best solution in
this case may be to try in advance to avoid subject matter in the study where
confidential company research is underway.
1
■'■O
I
1p
• H
" - -"I
: I
• I ' Iff
11
Cunclusions
-fif ' £
1 he preceding list of issues that must be considered when conducting corporate
Delphi research is not exhaustive. The main purpose of this section was to
examine some of the common or most important issues that the business
planner must face when deciding whether or not, or how, to use Delphi
research. As with many situations, a heavy application of common sense when
planning Delphi research will avoid some of the potential problems outlined
above.
1
Future Use of Delphi in Corporations
I he near future should see continued rapid expansion of the Delphi technique
in business. The methodology appears to be currently reaching the “faddish”
stage. Many low-quality studies (which may be mislabeled “Delphi”) will be
I
Lawrence H. Day
General Applications:
conducted. 1 his could result in a credibility gap with those trying to use the
technique to its best advantage. If this credibility gap does occur, there may be
a numerical decline in the number of studies conducted, but a general
improvement in the overall quality of corporate Delphi research.
Widespread use of the methodology will result in continued rapid modifica
tion of the original RAND design. Mini-Delphi’s will be used to develop
specific forecasts or evaluate potential policy changes. The latter area will
itceive further attention with the continued development of interest in tech
nology assessment. I he use of on-line Delphi techniques will spread, especially
as corporate management information systems and remote access terminals
become widespread [38]. 1 he availability of standard packages that permit any
researcher with access to an on-line Delphi system to act as a study director will
also encourage further use of the technique [39].
Delphi will become popular for certain types of market research studies. This
will probably occur more as a result of the promotional activities of market
tesearch firms than from the conscious decision of corporate researchers or
marketing academics. 1 his opinion is held since there is little overlap between
the current professional literature of the marketers and the long-term planners
[10], whereas consultants are presently indicating interest in the technique.
In conclusion, Delphi has a healthy future in the corporate environment.
1 his is a future for a whole family of Delphi-inspired techniques in a broad
tange of applications. Use of the term “Delphi” to describe a monolithic
technique has rapidly become obsolete in this environment. This expanding
family of techniques will be the properly of the market researcher, market
planner, policy planner, systems researcher, etc., as well as the long-term
business planner.
Delphi Research in Corporate Environment
193
I
17. North and Pyke, “Technological Forecasting in Planning for Company Growth,” IEEE
Spectrum, Jan. 1969, p. 32.
18’ K4e’i f°J,,eia/nple: E- F‘ Parker’ “Some Cexperience with the Application of the Delphi
Method, Chemistry and Industry, Sept. 1969. p. 1317.
H
E. F Parker “British Chemical Industry in the 1980’s—A Delphi Method profile,” Chemistry
and Industry,
\970, p. \38.
J
19. Gordon Wills et al. Technological Forecasting, pp. 194-203, 208-9 Penguin Books, London, 1972.
Also discussed in Gordon Wills, “The Preparation and Deployment of Technological Fore
casts, Long Range Planning, Mar. 1970, pp. 51, 53, 54.
20' March U1972 p^"0108^1 Forecastin« in Six MaJor UCompanies,” Long Range Planning,
9. J. R. Salancik, Theodore J. Gordon, and Neale Adams, On the Nature of Economic Losses Arising
from Computer-Based Sen-ices in the Next Fifteen Years. R-23, Institute for the Future, Menlo Park
Park,
March 1972.
■
21. A. D. Bender, A. E. Strack, G. W. Ebright, and G. von Haunalter, “Delphi Study Examines
Developments in Medicine,” Futures, June 1969, p. 289.
22. George Teeling-Smith, “Medicine in the 1990’s: Experience with a Delphi Forecast,” Long
Range Planning, June 1971, p. 69.
*
23. Communications Needs of the Seventies and Eighties, Bell System, April 1972 (internal document).
24. trank J. Doyle and Daniel Z. Goodwill, An Exploration of the Future in Educational Technology,
Business Planning, Bell Canada, Montreal, January 1971 (proprietary).
Frank J. Doyle and Daniel Z. Goodwill, An Exploration of the Future in Medical Technology, Business
Planning, Bell Canada, Montreal, Oct. 1971 (proprietary).
Daniel 7 Goodwill, An Exploration of the Future in Business Information Processing Technology,
Business Planning, Bell Canada. Montreal, Oct. 1971 (proprietary).
Darnel Z. Goodwill, Perspectives D’Avenir des Techniques de ITnformatique, Planification du Service
1 elephomque, Business Planning, Bell Canada, Montreal, August 1971 (proprietary).
Michael I Bedford, The Future of Communication Services in the Home, Business Planning Bell
Canada, Montreal, November 1972 (proprietary).
25 Techno^ D°ylC and GoodwiI,» National Technology and Medical Technology, Goodwill, Business
26. Doyle and Goodwill, Educational Technology, p. 64.
27. Ibid pp. 40-41.
28. Doyle and Goodwill, Medical Technology, p. 60.
29. Goodwill, Business Technology, p. 99.
30. Michael T. Bedford, The Future of Communication Services in the Home, Round II Questionnaire,
Business Planning, Bell Canada, Montreal, Ian. 1972, pp. 2-3.
i
•I
10. Sclwyn Enzer, Some Pro^ecb for Ruuknlial Housing by 1985, R-13. Institute for the Future. Menlo
Park, January 1971.
11. Selwyn Enzer, following article (III C 2).
12. Selwyn Enzer, Dennis L. Little, and Frederick D. Luzer, Soi
ime Prospects for Social Change vby 1985
°n Txrne/Money
R-25, Institute for the Future, Menlo Park,
.. ...March
------ 1972.
13. Ole Lachmann, “Personnel Administration,” p. 23.
*,
H. Marvin A Jolson and Gerald L. Rossow, “The Delphi Process in Marketing Decision
Making, Journal of Marketing Research 8, (November 1971). 444.
15. Contemporary Research Center, “Thinking about Tomorrow,” Printout, April 1972.
!6. See, for example: Harper Q. North, Technological Forecasting in Industry, NATO Defense
Research Group Conference, National Physical Laboratory, England, Nov. 1968
Harper Q. North and Donald L. Pyke, ‘“Probes’ of the Technological Future,” Harvard Business
Review, May-June 1969.
Harper Q. North and Donald L. Pyke, “Technological Forecasting to Aid Research and
Development Planning,” Research Management, 12 (1969).
J?arpe5
and
L- Pyke’ “Technological Forecasting in Planning for Company I
Growth, IEEE Spectrum, Jan. 1969.
7
Donald L. Pyke, “A Practical Approach to Delphi,” Technological Forecasting and Long Range
Planning, American Institute of Chemical Engineers, Nov. 1969.
Donald L. Pyke, The Role of Probe in TRW's Planning, Technological Forecasting—An Academic
Inquiry, Graduate School of Business, University of Texas, April 1969.
pnald l,7 Xyke’e *<LOng Range Technol°gical Forecasting: An AppL ation of the Delphi
1968eSS H
Summer Sy™?05*0"1 ofEnergy Economic Division of ASEE, UCLA, June 21,
References
1. Forecast 1968-2000 of Computer Developments and Applications, Parsons and Williams, Copenhaeen
1968.
r
s ’
2. Sample results are shown in “Computers in the Crystal Ball,” Science, August 1969, p. 15; see
also Joseph Martino, “What Computers May Do Tomorrow,” The Futurist Oct 1969 do
134-135.
’
, i. Ole Lachmann, “Personnel Administration in 1980: A Delphi Study,” Long Range Planning 5
No. 2 (June 1972) p. 21.
’
4. Delphi Panel on the Future of Leisure and Recreation. SET Inc., Los Angeles, 1972 (multiclient
proprietary study).
5. Letter from Ducharme, Deom, and Associes Inc., August 18, 1972.
6. See the Articles of Incorporation; the Institute aim is “to make available without discrimina
tion the results of such research and scientific advances to the public”; also quoted on inside
front covers of many IFF reports.
/. 1 aul Baran and A. J. Lipinski, The Future of the Telephone Industry , R-20, Institute for the Future
Menlo Park, Sept. 1971.
8 l^?! Baran’ Tht FutUTe
Institute for the Future, Menlo Park, December
■
•3
I
I
1
3
i
ft
i
’I
'i
$
>
I
Lawrence H. Day
d. Bedford, Communications in the Home, Nov. 1972.
J2. One recent indication of this interest was the formation of the “International Society for
Technological Assessment."
33. Bell Canada has recently attempted a m<xiest pilot assessment: Philip Feldman. J Technology
Assessment of Computer Assisted Instruction, Business Planning, Bell Canada, Montreal, Sept. 1972.
34. Baran, Future of the Telephone Industry.
35. Canadian Computer/Communications Task Force, Hranchtng Out. Vol. I, Information Cianada.
Ottawa, May 1972.
3b. The Bell Canada Business Planning approach has been described in J. Martino,
“Technological Forecasting is Alive and Well in Industry." The Futurist, IV, No. 4 (Aug. 1972k
pp. 167-68.
>«. An exception to this is the Policy Delphi conducted within a corporation.
38. See Chapter VII. For information on on-line Delphi see: Murray Turoff, “Delphi Conferenc
ing," Technological Forecasting and Social Change, 3 (1972) pp. 159-204.
Murray Turoff, “Delphi and Its Potential Impact on Information Systems,” Fall Joint Computer
Conference Proceedings 1971, Vol. 39, Afips Press, 1971.
A. J. Lipinski, H. M. Lipinski, R. N. Randolph, Computer-Assisted Expert Interrogation—A Report on
Cun ent Methods Development, Technological Forecasting and Social Change 5 (1973) pp. 3-18.
19. A. J. Lipinski et al. pp. I3ff.
10. One exception is (he Ixxik by Wills, which lakes .1 maikeling approach when reviewing
forecasting and Delphi studies.
Acknowledgments
Many of the current and past members of the Bell Canada Business
Planning Group have been involved with the Bell Delphi studies outlined in
tins paper. Mike Bedford, Frank Doyle, and Dan Goodwill spent many months
on the research, design, conduct, and management of those studies. Don
Atkinson, Ogy Carss, Ken Hoyle, and Sine Pritchard provided the necessary
management support and maintained a belief that these efforts would produce
useful results.
1 would also like to thank Phil Feldman for his efforts in tracking down
many of the references listed. In addition to some of the individuals mentioned
above, I ony Ryan and Phil Weintraub provided many useful comments and
suggestions on earlier drafts of this article.
L.H.D
III. C. 2. Plastics and Competing Materials by
1985: A Delphi Forecasting Study
I
SELWYN ENZER
1 he application of Delphi to the identification and assessment of possible
developments in plastics and competing materials1 posed a severe challenge to
the technique. Before launching into a discussion of this project it is worth
considering the advantages offered by the technique for this application. Since
the study was conducted with questionnaires transmitted through the mails, it ’
permitted many widely separated people to participate without the difficulty of
having them travel to be co-located at any specific time. It permitted the group
to focus on what they regarded as major developments very quickly and discuss
only those prospects in detail. Furthermore, because anonymity was employed, Bi' d
each participant was forced to judge the
the potential
potential of
of each
each possibility
possibility on
on the
the
basis of his knowledge and the supporting arguments presented. In other words,
the tendency tQ jucjge those developments suggested by the most notable
panelists were eliminated by virtue of anonymity.
This study was originally scheduled to be completed in three rounds of
interrogation. However, as it evolved, only two rounds appeared necessary.
1 his occurred by virtue of the high degree of specialization which appeared in
the first-round responses and became even more evident in the second round.
1 he ability to tailor-make plastics for various applications, enhanced by
■3
growth in understanding of organic chemistry, alloying, reinforcing, etc., plus
the responsiveness of the material itself, have led many researchers to believe
that the types of plastics produced in the future will be determined more by
what is desired (and pursued) than by what is possible. Thus in many ways this
study was more an investigation of material needs and resource allocations than
of technological possibilities.
I he study focused upon possible combinations of material property2 changes
that are likely to affect widespread material usage. A prime difficulty encoun
tered in this study arose from discussing yet unknown (and hence unnamed)
materials. In general, it is easier to discuss improvements in the properties of
J
’5®
steel, aluminum, concrete, boron, niobium, etc., than to discuss the prospects
for development of, and properties of, material X, Y, or Z. Yet in many cases
I
i
I
i
Ia
f * th^p0 ^(j’
in Plastics and Competing Materials by 1985, Report R-17, Institute
properti“,uch"strcngth'den^-
IC20h"04U2C93a2nd
Harold A'
“d
■
T“™« ('*•)
■
Copyngbi © 1975 by Addison-Wesley Publishing Company, Inc., Advanced Book Program.
All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or
transmitted, in any form or by any means, electronic, mechanical photocopying, recording, or
otherwise, without the prior permission of the publisher.
■
i'.'H
Selwyn Enzer
this study had to do exactly that. As a result, it probably tended to focus more
on changes in existing materials than it did on totally new materials.
Since the study focused on material property changes that may be realized in
existing materials as well as new materials and their properties, the number of
alternatives to be contemplated was vast. Io address this challenge a matrix
type categorization of materials and properties was used as the point of
departure. For this purpose a breakdown similar to that presented in “The
Anatomy of Plastics,” Science and Technology (F. W. Billmeyer and R. Ford), was
used. I his matrix of materials and properties was divided into five subcat
egories:
* Engineering Plastics
General Purpose and Specialty Plastics
Glass Fiber Reinforced Plastics
Foamed Plastics
« Nonplastics
I he panel was asked to: (1) review the materials and properties presented,
indicating where they thought changes were likely to occur within the next
fifteen years which would significantly affect the widespread use of that
material; and (2) add and describe the anticipated properties of new' materials
which they thought w'ere likely to evolve and gain widespread use by 1985. In
both of these steps the panel was also asked to describe the new chemical,
physical, or other technological developments that they believed would lead to
the creation of the new' material.
I hese inputs from the first Delphi round were used to prepare a three-part
questionnaire for the final round of interrogation. These parts were: (1) a
.ummary of the assessments of anticipated changes in existing material proper
ties, indicating those selected for more detailed investigation; (2) a listing of
both plastic and nonplastic materials with the nature of the anticipated major
changes described (those respondents who had anticipated these changes were
asked to estimate the new material properties they expected would exist by
1985 and to estimate the 1985 annual consumption by application); and (3) a
list of new materials anticipated by 1985 and a description of their properties
(those respondents w'ho had anticipated these items were asked to estimate the
properties and consumption patterns they expected for these by 1985). All of
lhese parts were open-ended in that any of the respondents could still add
additional items or comment on any item.
General Applications:
Plastics and Competing Materials by 1985
197
'■%
materials which would significantly affect their widespread use by 1985; and
(2) identify new materials (in each of the categories shown) which are likely to
be developed and would be in widespread use by 1985.
Table 1
Existing Plastics
■ w
Engineering Plastics:
ABS
Acetal
Fluorocarbons
Nylon
Phenoxy
Polycarbonate
Polyimide
High Density Polyethylene
Polypropylene
Polysulfone
Urethane
Poly (Phenylene Oxide)
General Purpose & Specialty Plastics:
Acrylics
Cellulosics
Cast Epoxy
Ionomer
Melamines & Ureas
Phenolics
Low Density Polyethylene
Polystyrene
Vinyls (PVG)
San
Glass Fiber Reinforced Plastics:
ABS
Epoxy
Nylon
Polyester
Phenolics
Polycarbonate
Polystyrene
Polypropylene
San
Polyethylene
i
-
i
I
Foamed Plastics:
Polyethylene
Polystyrene
Polyurethane (low density)
PVG
Polyurethane (high density)
1»
I
■hl
.ij
Anticipated Changes in Properties of Existing Materials
1 he Delphi panel w'as presented with descriptions of the major uses, properties,
and proprietary qualities of 37 plastics and 16 nonplastics all currently in
widespread use. These 37 plastics are presented in 'Fable 1. As indicated earlier,
they were asked to: (1) identify likely changes in the properties of these
-Pi
1 he format used for this portion of the assessment is shown in Fig. 1. This
figure is divided into four columns. Column 1 lists the material and its typical
uses. Column 2 describes the properties of that material which are the key to its
current widespread use. Column 3 is divided into subcolumns which contain
specific material properties and the current performance ratine nf each
EXISTING GENERAL PURPOSE & SPECIALTY. PMSTICS
Property
3-
MUTEHIAL &
5u
</
Proprietary qualities
(relative to typical
uses):
(*) Assets
(-) Liabilities
Typical uscb
F
r:
F
CAST MOXY
n!
Print-d clrcults;
potting compounds
C•
h:
!•*!
11 11H
l-'old^d
(♦) strong A flexible
ITT
2
5
t:
2
3
flj
illfI i
J sSi £ 2inr
2
3
1
2
2
3
2
2
2
1
1
.2
2
3
2
1
1
1
1
3
2
1
2
2
£
4>
Aspects of changes in material
properties;
<n
V)
tn
?
♦J
-
r
H
o
6
ACRYLICS
(♦) optical clarityt
Windows; fiber optics
weather reel stance
building panels;
lighting; tubing
(-) abrasion resist.
CEI.I.ULOSICS
(♦) tough; clear
lacknging; film; toys
telephones; Instru
ment glass
(-) abrasion resist.
>4.
•r-4
r*
________________
housewares;
toys; extruded tuba"n‘tp!5heet’lng; P"ckUh EAS
'j.
6
I
Di idles; wood laminater
appliance cabinets;
electrical devices
I'H'TOLICS___________
Appliance cabinets &
parts; bonding
resins; electrical
(♦) transparent.tough
i stance!
st rength■
temp, rnrtge
(♦)
(-)
rance (finlshat'i 1 i *y );
surface hardness
Impact strength;
t>-inp. range
(♦) cost;
strong.hard.rigid;
abrasion resist.
(-) chemical resist.
IOW DENSITY POLYETHYLENE♦) flexible ; cost
Dishes; Lotties;
„ Ji(-) --------Strength
pipes; tubing; film
ueatherabl1ity;
| packaging__ ______
1'lammabi 1 ity
Outstanding In property
indicated; among the
best performers available.
3
2
2
3
3
2
2
3
3
3
I
1
2
’T
2
3
1
J';
3
1
Acceptable performance
In this property; etill
suitable in cost cases.
1
Not acceptable if indicated
property is Important to
Intended use.
Fig. 1. Typical questionnaire for eliciting changes in existing plastics.
S1
n
i ■
S § g
o 3
“ ft
ft
ft
p_
C
•3p
%
§
tiff
=: ° J? 3. §
>
w
ft
o
o
(Z>
c
=: o
2
‘
P
co
T3
O_
2
<z>
o,
£
*<_
g
<
TJ
a3 £
g- oT g
s o3
ft
g?
§
Cd
§
s
< r
*p
o O_
3
0 y’ M -
3
p”
□
Cd
Cd
I
22
2
g
2
8
5
P
Z n >
3
3 E
C" S 3
~
%
2 X
* §
5
CD
C/2
o
1
•fl
a.
no
25’
o'
e
2
p"
o*
□
4f
■8c
5
8-
p
tn
r*
8
I— •
I
3
—I -^rsn—'■
Cd
Cd
C/5
“
s.
Cd
i
Is
»>.
c»
: j. . f
l-
hg
ft
^4..
H
1 3 I-i-
-
,
Pe
ft
CL
3‘
CTQ
3 = p
r
i. ■ -
ft
2
o,
93
p_
o*
O
ft
c^
p
n>
I
o'
S’
4
Cd
(Z>
!■
? §I
ft
ft
T!
’ i-1
? 5141.
3
OQ
S’ C/9
%
;
“O *c
-u
2 X
y_ O tT
p p _ l
-3ft
2 H3 S 5"
ft —
CT
a 2_
F 33 e
□ 3-G>
3 (T>
“O “O
□
ft —
JZ
ft
3
5
3
c
% CTQ • 3
? 3 ftQ. CL• h’ a
ft
£
zr o
O
H
2. CT
: p
c 5- 3_ ° ft
— cn
sr
09
zr 3 3o Pft
oo c.
cr
ft —
3
ft UQ O
ft O, OQ a p
D2.
ft
o
P
rr
. “S'
□ in
ft p ft
O,
s"
cl
cr
§: ?°
O 5
' p !-•
ft” ft
*
rt
9
Q.
p
3
ft 3 5'
g ft
□
5- > o’ B- 3
JO « 3 oq
3“
3n>
c
8
2O
|
3.
o
8- 5- 3 5
3 3 3
-•- 2o
C5 p Er
P 5 ft P Cd p CL J”.
3 H
□
ft
ft FO q 3 a.
p 2 r
3
8 sip s< 35
— • .P
3 p' era
3
rt
3
2 3 rr 3 3 ►—
2, ft
-- gCL F ft £.
ft
a 1=
p
• CL
— fj zr ft
pfto’<"o3ft^^
S- c
p
—.
-i < p
• " ft§ 23 5'
O ft 5
“
£
2
9S
CTQ
n
a> ft ft
2
Qp 2- %2
ft
2
n> 13
ft p
8 >
1 P2
P 2-^
ft
ft
ft
ft B. 77
5’^ C/5 tZ_
C fs ft OQ
8" 8 ft C*
o
<»
ft
*
P
*
3
§
□
<
□
OQ
s § □S c« □. ftCL ftcr
3
P % “ rt>S
ft
cn 2- 2
CTQ
8. o - 2- H
ft £.
o
e-. ft o
CL ~ C.
C 3
ft Po
3
2
8
O B’s-i
3 to £ 3 5' P ® ~ pK’
□ ■ ft
5
JQ § ft ft C.
»P orq
S 3
P H
(T>
ft
ft 5*
FiQ->
Q. 5
ft
S- 35 -g S’ OQ3“
o_ 22 2.
cn r*
ft
P
n> r-•
cr
3"
5to □’ ft
a.
B
ft
Q.
3
~ ^3
CTQ
3 5'
9
5
’
8.3
q
%
ft K ft « 5’2, &
3
ft
z
S S
0
1
C
H
5*
4-h-
ft
CL
p
9 " ■ c 9 9.
S 2... Y? S'
0
ft
3
ft
P
I
3
g
2
p
□
CL
o
o
3
no
ft
2
p
ft
p
Cd"
CT
to
CO
Ln
*O
P
oa3 x) ft3 oo8* Se’ fta> £p § < s.
5 °-<g B_ 3 S’
5’
b a S • p
ft_
to
<D
_EXISTINC CESERAL PURPOSE A SPECIALTY PLASTICS
12.
1.
MATERIAL &
-----------------
J
J' fe
I
£
£ fu
ACRYLICS
(*) optical clarity;
Windows; fiber optlcsi(
weather resistance
building panels;
'
lighting; tubing
| (l) abrision resist.
2’
3
J0)1 U)
o
•-<
ai
c
•>
«
x
c .
1
<—
<0
"x,. —
u
—
ri
c
S’. C’C
ttf! v.cd
~~.
u
L?
t -t
3 I 1
tn
«^4
■-
£
40 or 20 top1 ess
than
* of the panel Tore
40
?n _
'i rection
Improvement
>f expected
'hanqe
t>eqradation
fJ L
1
2
£|4
2
3
None
1
rnrw n 11
1
(Included in package No. 2).
CEL^JOSICS
; (A) tough; clear
Packaging; film; toysj
telephones; instru
ment glass
(L) abrasion resist.
2
3
CAST EPOXY
Printed circuits’.
potting compounds
2
2
(*) strong & flexible
2
2 52
3
2
1
3
3
2
3
•Wot competitive with low cost plastics,
e.g., vinyls.
1
2
0
2
1
(Included in package No. 2).
j (A)
10:::
transparent.tough
2
Folded housewares;
ch^^’<’0 1 stance
toys; extruded tub(L)
strength;
il’;?'rir!'h'’ctlng; Pack- I
temp, r-in/'e
MF!A?1:
f' lr?yAS
J-jTy
Di:-hes; wood laminate^
surfnc<- hardness
appliance cabinets;
fL) irpnet strength;
electrical devices
t<">p. range
PHENOLTCr.________
I (A) cost;
Appliance cabinets & I
strong.hard,rigid;
parts; bonding
abrasion resist.
rosjns; electrical
[ (L) chemical resist.
3
LOW DENSITY POLYETIfYLENEI A) fiexible; cost
Dishes; bottles;
( L) strength
* .ngth;
pipes; tubing; film
veathcrabil1ty;
packaging_________
flammability
^Degradations encircled indicate panel
3
□
. s
cn
«^4
tn
c
o
( A) Assets
(I) Liabilities
I
Property changes anticipated by the panel,
likely to affect the widespread use by 1985,
ire noted in Italics as follows:
------- «7
Ji 5
Proprietary qualities
(relative to typical
uses):
Typical uses
CODE
Property
3
3 1
2
2
1
(Included in package No. 2).
2
2
3
2
2
3
1
3
2
2
2
2
3
2
1
2
2
2.
(Included in package No. 2).
3
1
1
3
2
3
•Compounding can improve weather resistance.
di sagree.-aents with Lite original rating not a forecast
Outstanding in propertyindicated; a-mong the
best performers available.
7^
] Acceptable performance
I in this property; still
I--- 1 suitable in most cases.
□
□
Not acceptable If Indicated
property is important to
intended use.
r
1
2
Fig. 2. I ypical feedback of results of initial estimates of changes in existing materials.
ENGINEERING PLASTICS
a^I^n'i^ANT^NCr'eaSE HE
O
USEC& A985IPATES
3
KEY PROPERTIES IN WHICH IMPROVE
MENTS ARE CONSIDERED LIKELY BY
THE PANEL
1 .
CURRENT
PROPfHTY
VALUE
ESTIMATE
PROPERTY
VALUE
LIKELY BY
1985
COMMENTS BY THE PANEL
uj
. - Reduction in depolymerization will
improve flammability.
- Alloying and blending will improve
Maximum Service Temperature
temperature range and flammability
*(D 648) 'F
180-245
250
- Improved shaping of disperse par
ticles as well as processing tech
Flammability, ln./mln.*(D 265) 1.0-2.0 0.65-2.0
niques will orient particles.
Impact Strength
- Reduction in compound prices;
Notched Izod, ft. lb./in.
scale and competition lowering
*(D 256)
0 73°F
2.0-10.t
prices.
- Availability of composite forms as x
0 40°F
.8-3.5
.9- 5.0
sheet and ability to fabricate in
Deflection temperature, °F
inexpensive equipment will make
*(D 648)
Cd 264 psi 214-244 214-280
this material competitive with
glass-polyester.
0 66 psi 215-250 215-280
- Platability makes this material in- x
creaslngly attractive for automo
bile parts, this will be especially
important If low temperature
strength and crack resistance can
Price, ($/lb.)
.28-.44
MARKET BREAKDOWN
(MAJOR USES)
.20-.30
VOLUME- MILLION
LB/YR
CURRENT
ESTIMATE
1965
TOTAL VOLUME
508
1500
Automot1ve
80
250
60
J 50
60
200
o
ABS
rt>
APPLICATIONS
Z
kJ Ul
lU
za
Major
Appllancea
Pipe &
Fittinps________
Bus.Machines,
Phones____________
Recreational
Vehicles
90
p
cn
8
P
□
27
65
350
upon filler.
- AUS will include chemical and
cross-linked materials.
!'
150
196
be improved.
s’
45
Luggage
- Properties will be very dependent
i
40
Other:
Interior panels
& sheeting
3
2
CL
o
o
3
2
5*
OQ
2
p
q
3'
er
*<
<O
CO
<_n
*AST>! Test
to
o
Method
Fig. 3. Typical results of final assessment of important changes in existing plastics.
.
.
;;
-■
.. f
o
O
202
O
I he comments received from the panel arc presented in Column 2 of this
figure. Because these generally referred to the reasons why the material
O
Sclwyn Enzer
property changes were anticipated, the panel was asked to indicate whether or
not they agreed or disagreed with each statement. The results of this assessment
arc also shown in Column 2. Those items presented in italics in this column
weie added in round two and hence were not assessed by the entire panel.
Column 3 presents the current major markets and their annual volume
General Applications:
Plastics and Competing Materials by 1985
Table 3
Other Materials Suggested by the Panel
E ng i neeri n g Pl ast ics:
usage. Shown in italics arc
■■■ ■ new markets suggested by the panel and their
estimated 1985 usage.
In that portion of the investigation concerned with nonplastics, many new
Polybutadiene (High 1,2 Content)
Polyethyleneterephthalate
Polyphenylene Oxide Derivatives
New Thermoplastic
material developments were suggested, but only a few of these were regarded as
threats to the growth of plastics. This can be seen in the following general
New Tougher Plastics
comments received from the panel.
•
I he main competition between plastics and aluminum will occur in the
construction field, particularly in residential housing and light industrial
buildings. New developments in aluminum will hurt plastics in the applica
General Purpose and Specialty Plastics:
tions which are primarily structural. On balance, however, these develop
O
O
ments will affect the use of other metals more than plastics.
• In general, plastics will continue to replace iron and steel in some applica
tions. This will be significant to the plastics industry; however, it will be a
relatively small change to the steel industry. Any development which brings
steel closer to “one-step” finishing, with improved environmental resistance,
will be important in this regard, since it will blunt some of the basicadvantages that plastics have over steel, allowing the use of “conventional”
technology and existing capital equipment. Such developments will bring
steel and plastics cLloser to a straight-cost competition. However, these developments must be realized before potential markets have switched from steel
to plastics
j 1
to maintain the continuity of technology and equipment.
• Developments in concrete appear more likely to enhance the demand for
plastics than to replace or be replaced by them. Developments in wood and
plywood are more likely to be in combination with plastics and hence
: are apt
to increase the demand for such materials. However, unlike the cc
concretes,
wood will increasingly be replaced by plastics, particularly in furniture and
siding.
Other Materials Suggested bv the Panel as Likely to Become Important
by 1985
O
In addition to the changes suggested in the existing materials, other materials
(some already in existence) were suggested by the panel as prospects for
widespread use by 1985. These are presented in 'Fable 3.
These materials were submitted for consideration by the entire panel in the
final round of reestimation. Format and typical results
> are presented in Fig. 4.
I.
203
PVC—Polypropylene Copolymers
Ethylene—Polar Copolymer
Acrylic—PVC
New Polyolefins
New Thermosetting Resins
Completely Nonburning Organic
Semiorganic & Inorganic
Glass Fiber Reinforced Plastics:
PVC
Polyimides (or Amidemides)
Polysulfones
Polyurethane
Vinyl Ether
Thermoplastic Polyester
Thermoplastic Sheet
New Thermoplastic Resin
as Likely to Become Important by 1985
I
Other Fiber Reinforcements and
Reinforced Plastics:
!■ S
Boron Fibers
Graphite Fibers
Fiber Strengthened Oxides
Aluminum Oxide Fiber & Whisker
Composites
I1
Boron/Epoxy
Boron/ Polyimide
Graphite/Epoxy
Graphite/Polyimide
Foamed Plastics:
Phenolic Foams
Vinyl Foam
Polyolefins (Ethylene, Propylene, etc.)
Isocyanurate—Urethane
Silicone Foams
Special Hi-Temp Foams
Structural Foams
Foamed Thermoplastics
Injection Molded Urethane Foams
iI
H
t1
j
Miscellaneous:
Silicate Glasses & Polymers
Titanium Alloys
Cermets
As seen, this is similar to the format presented earlier. One notable difference
is in Columns 5 and 6, which contain estiamtes of the likelihood of these
materials being in widespread use by 1985 and the annual production esti
mated by that time. These estimates should be treated with even greater care
than those presented earlier, since they often represent the comments of as few
as two or three respondents who were familiar with the development
I
■
a
j
j
'A
ji
H
iI
•»/-
.? 1G
Nancy H. Goldstein
General Applications:
I here were two principal elements to handling the results of round one
Krst a determination was made, for each graph, as to the location of upper
,md lower hmns of the extensions which would include 50 percent of he
responses to that graph.
The second element involved the gathering and synthesizing of comments
presented under Questions 2 and 3 of the graph sheets. The comments of all the
respondents were collected, and each comment was then studied and deter
mined to be either a forecasting assumption, an economic and international
consideration, a key development, or a comment to be associated with that
pamcular graph page. The comments were grouped accordingly, and the final
product was retyped for inclusion on the second round. It Was quite apparent
(hat in many instances one respondent's assumption was another’s uncertaintv
1 he frequency with which a topic was brought up influenced the judgment on
its choice as a key development for round two.
The process of collecting and editing the large number of comments obtained
on the first round represented the largest single task in the exercise, in terms of
both clerical tune and professional judgment. Assumptions from all respondents
were initially xeroxed, cut out. and taped on large sheets. Each sheet represemed different topics or curves. On these large sheets duplications were crossed
out and edtttng of assumptions to produce shorter wordings took place. This
conglomeration was then retyped once and put through a final polishing
cdH.ng, and reordering before the final typing for the second-round
questionnaire. I he process of putting each set of assumptions through a
two-stage edtttng process allowed each professional to check the other's work It
is nottceable to a certam extent that the availability of the xerox machine is a
key feature in making large-scale I lelphis possible via paper-and-peneil
qiptoaihes. Ibis ,s parncularly true where one is handling
large volume of
u xiual comments on the pan of tin- respondents.
beetiun III in round one. ■'Suggested Additional Variables and Kev Devel
opments, produced few responses from the participants. Some of the response.,
m this section became key developments for Section 111 of round two some
became assumptions for round two. and the remainder were dropped from the
■ xeicise. Three additional curves suggested by the respondents were prepared
or inclusion in round two.
i he flow chart appended to Section I in
the first round also received scant
’esponse «(i.e., about ten respondents). The responses that
were supplied were
averaged for each <entry in the flow chart and standard
deviations were
tovided through the use-Ja
of simple computer program.
217 I
th.rty-ftve •Economic and International Considerations,” and all the graphs
untamed m round one (Section I) with their associated reasons. The rlspondent was asked to assoc.ate a validity score with each forecasting assumption
I
econ^^ and tnternanonal consideration presented. The scores were based
on the validity codes which are shown in Fig. 2.
l or all graphs the original trend line was presented and the 50 percent
confidence limits were indicated. The respondent was asked to reestimate his
previous extension, after viewing
iewing the 50 percent limits, and to identify his
estimate as reliable, as good1 as anyone’s, or risky. Each graph also contained '
associated reasons given by the respondents for increasing or decreasing the
graph extension. In Section I, a total of 116 reasons associated with the graphs
were presented for evaluation. The respondent was asked to assign a rank of 1
to 6 to each reason given according to the validity scale. He was also invited to
Show additional reasons if he wished. A sample of a typical round-two question
is presented in Tig. 3. Phis is exactly the same basic form as used in round one.
1969 Vi'otnf Sr,eCtl°? ‘d the flOW ChartS Were again Presented- Two charts, for
1969 and 980 showed the means and standard deviations for each chart entry
and also showed some additional boxes and pa(hs no(
jn round
X
hese modifications were suggested by the respondents and incorporated by the
senior professional. The respondent was asked to circle the estimate presented if
he agreed w.th it or to cross it out and provide a new figure on the blank
charts provided d he disagreed. The absence of either action was considered to
be a No .Judgment vote. It was a surprise to the designers that almost all the
tespondents to the flow chart chose to modify it, since this was not an action
suggested in the instructions.
Section II. ■■Alloys,” was similar to Section I in design. The category of
(O.< C.isimg assumptions included sixty-nine assumptions about individual alloys
sliKhed. as well as seventeen general assumptions. The respondents were again
asked to assign each statement a validity score of from 1 to 6. There was no
category of Economic and International Considerations, but all the graphs
from Section II of the first round were included with the 50 percent confidence
hmns and 128 associated forecasting reasons. The respondents were asked to
provide the same information requested in Section I.
Section II was entitled “Key Developments and Added Curves.” Under Key
evelopments, thirty-six items were presented for scoring by the respondents,
or each Hem, the respondent was asked to evaluate the likelihood of occur
rence by 197J on a scale of 1 to 6 and to indicate, on a scale of 1 to 4 the
impact on the steel industry if the development were to occur. Figure 1 provides
a sample of the form utilized for key developments.
In addition to scoring the developments, the respondents were asked to
desenbe the nature of impacts they had characterized as strong or moderate
i<ound Two
The questionnaire sent to the respondents in round
two was patterned
• Iter the results described in the handling of round one.
he new Section 1. ■ Steel,” contained thirty-six •■rorecasting /S^urnpiionM,”
Steel and Ferroalloy Industries
■
I
in hiXT"' CT>
rcsPondents
°ne, were also included
1" 11,15 sect,,on- 1 he respondents were asked to handle these new eurves In the
‘mne way ihay irwccl the original curves in round nnn
i ..... i m
I
i
1
1
1 ’
I
z=
to
Z
~ = ~
= i -
Fr
- 2 1
-
= "S
- r: Y- ±
5
y
--
4 E
{.
7
i
r
-
2 ’2
•x
’z
{.
T=Z- 'z
- ;•
JZ
2
2
-■
—
7T
_
7
~
~r
7
’z
iz
~
/
X
-
'z
iz
7
1
=
2
E
Z 7
i-
F
1. Shown are the boundaries within which
50 ot tiie torec.ists fell. After reading
column 2 please reest imate. Relialv]..
As good is .myone's_______ Riskv
Reasons suggested tor imreasing or deer
the forecast are presented Ix-low. A.ki other
if you wish.
Validitv
('lioiee
Stainless Steel:
-Imports in semi-finished form are bound to
increase sharplv because of lower cost pro
duction outside I’. S.
-Current export increase due to Canadian
semi-finished imports for re-export after
finishing.
-Re-exports from Canada should decrease
Canada becomes self-sufficient.
-Semi-finished shipments to Europe will fall
as melting begins there.
-Foreign production facilities for flat stainless
are increasing and will be competitive with
U.S. in 2 to 3 years.
- True exports will be confined to special grades
where some proprietary position held.
-U.S. expected to retain competitive position
in low nickel stainless.
-Chromium shortage will boost
imports.
-Improved quality by use of higher purity alloys
(e.g. . vacuum process) will decrease imports.
-Since we do not have Ni or Cr. rest of world
will supply, utlimately, their own SS.
-Imports of 300-series (Cr-Ni) steel
expected to grow.
Imports. Exports
.350
.300
c
250
o
z
'o
_w
200
Exports
150
100
?
I
f
r
I
O)
2p
□
CL
z
2>
o
p_
o
□
CL
«-♦
a'
C/3
Imports
50
1060
'65
'70
YEAR
'75
'80
I 1-7 of 21
ro
IO
Fig. 3. Form for Trend Extrapolations.
Nam v H. (Joldsicin
G e n e ra 1 Applications:
Handhno
/{t'sii/ls. Round two was sum to fiftv-two potential respondents;
'hilly-four replies were received. Several respondents, re presentint' the polymer
mdustry. were added during this round. 1 hey had not been represented in
ound one and were introduced for the purpose f addressing specific issues on
i he substitution of plastics which had been gener ated in round one.
Question
Number
Mean
St a nd a rd
Deviation
1
2
3
3.4
2.6
3.0
1.0
0.9
1.2
I he results of round two required three separate tvpcs of handling: ( 1 i new
4) percent confidence limits were supplied for the graphs; (2) verbal comments
issokiatcd with the assumptions, the graphs, and the Rev Development section
.ere collected and considered; and (3) the numerical results, i.c,. validity
hoiccs. Key Development scores, and flow chart inputs, were collected and
i abulatcd.
1 here were* several steps necessary in handling the- large amount of data that
\as gcmcuatc’d by the* results of round two. It was lust detcrminc’d that several
latistical calculations on the data would be desirable specifically, the mean
iiid standard deviation of the validity choices for each statement, a distribution
bowing the percentage of responses falling under- each of the scores from 1 to 6
!<>i each of the statements, and a matrix comparing the- distribution, by
lumbers, of the different occupation categories represented bv the respondents
Dili the* range* of scores from 1 to (>. 1 hese' occupation categories included:
piimarv steel producers, reseau h institutions, sic‘c-1 producers. ferroalloy pro
Steel and Ferroalloy Industries
X’alidity Choice
Question
Number
1
1
2
.25
3
.20
2
.33
.20
Examples of the statistical presentations are shown in Fig. 4.
3
4
.25
.33
.25
5
Number
Responding
6
.25
.33
.20
.20
25
16
19
.20
Question 1
Occupation
('atetjorv
Validity Choice
arc cus. research institutions, govcrnmc’iit. and the universities. A computer
'logiam was wr iitc-n to carry out these c ompui.moris. I his breakdown
i cakdi»\\ u allowc-d
i- lo observe if there were any chllcrein <s in |uclgm<in which
w lii< li mav have
•flvctcd differences in affiliation of the- respondents.
221
Steel
I <‘t roalloy
3 R&I)
4 Government
1
2
3
4
5
6
3
1
3
0
2
1
0
0
1
2
0
1
3
0
0
0
2
1
0
0
0
4
1
0
Flic flow chart included in Section I rece ived very little- additional informaa*H in round two. I he scant information received was averaged into previous
ilormation on the flow chart and presented in a summary for round three,
ic to the differences of opinion among the respondents on how actually to
nodcl the flow of steel-making materials, it was felt by the designers that this
ic c|ucstjon could have constituted the total Delphi exercise among a select
mailer group of respondents.
Aound Three
/)esi!>n. Round three again consisted of three major sections: Section 1: A
‘Utnmary of Round 1 wo. Section I. Steel; Section II: A Summary of Round
'wo. Section II. Alloys; and Section 111: Key 1 )c\clopments and .Added
' UIA CS.
Sections I and II were provided for summarv purposes and reejuited no
urther input by the respondents, l or eac h statement, the mean and standard
deviation as calculated from round two results were shown. Several new
Fig.
4.
Examples of statistical presentations.
statements were added in Sections I and II; a few new developments were
developments
added to Section III to be assigned a validity score by the respondents. The
new □0 percent confidence limits, taken from the results of round two, were also
presented in a final form for each graph.
Section 111. the only section to be returned by the respondents, contained all
the new ;assumptions
................ — and' all
11 previously
•
evaluated assumptions which exhibited
a large standard
(
....................deviation
(LC- disagreement!.
I, nkn
t„,called
a reevaluation
. disagreement).
It also
for
of all key developments.
The first portion of Section III indicated the
percentage distribution of the
scores from round two on the likelihood and iimpact of potential key developments. The estimated average score for each development was indicated and
J a
summary of all verbal comments associated with each development was g‘
given.
The respondent was asked ;again to give his preference on the likelihood and
impact of each potential development.
222
Nancy H. Goldstein
1 he second portion of Section 111 presented the three curves shown in this
section of round two and included a number of reasons given by the respon
dents of round two for their curve extensions. 1 he respondent was asked to
rccstimatc the curves after readmit the assoc iated past reasons and to rate the*
reliability <jf his estimation, lie was also asked to vote on the reasons given for
each curve using the validity scale from 1 to 6 described earlier.
1 he third portion of Section 111 contained all assumptions from Sections I
and II which exhibited a considerable degree of disagreement. Phis category
generally, although not exclusively, included assumptions with a standard
deviation of 1.3 or greater. 1 he respondent was asked to reevaluate his previous
validity choice and submit a new score. Several new assumptions were also
added and the respondent was rec|ucsted to provide a validitv choice* for these
new assumptions.
In the final portion of Section III a new chart was introduced showing
percentage breakdowns of inputs and ou tputs for three major steel processes,
The figures were supplied for 1969 and a blank sheet was provided for 1980.
The respondent was asked to fill in the sheet for 1980 and to change anv 1969
figures with which he disagreed. A space was provided for an explanation of
any disagreements with 1969 figures. The results of this chart were to be
considered a summary response, as the monitors did not plan to feed back the
responses for changes.
I he monitor also surveyed briefly the attitudes of the respondents toward the
Delphi approach by asking the rspondents a number of questions, e.g.. was the
time spent in participating in Delphi well used: what organizations should
sponsor an exercise of this type on a regular basis, etc.
Handling the Results. Round three was sent to thirty-eight respondents on
December 10. 1970. Thirty-three respondents actually replied to Round 3.
A computer program provided the* means, standard deviations, percentage
distributions, and industry category matrices for all key developments, assump
tions to be reevaluated, and new assumptions. The percentage distributions
were then examined by the senior professional. If
percent or more of the vote
fell into the “not pertinent" category (a validity score* of 6). the items were
dropped from the exercise. Bight items were* dropped for this reason. The
remaining items were regrouped so that every assumption was associated with a
curve. The assumptions and reasons for each, curve were then reordered
according to their mean validity score's.
The final report was then prepared for the National Materials Advisory
Board of the National Academies of Science and Engineering.
Comparison of Delphi and Panel Studies
A separate panel appointed by the National Materials Advisory Boa rd
approached the same problem considered by the Delphi exercise in a more
General Applications:
Steel and Ferroalloy Industries
223
conventional vein. I he conventional study was carried out simultaneously
__ 1_____^.w’z
with the Delphi, but the results were not compared until both exercises had
been completed.
In the panel approach, individual members of the Panel on Ferroalloys
reviewed portions of the problem with which they were most familiar. The
Panel Report, NMAB-276, “Trends in the Use of Ferroalloys by the Steel
Industry of the United States” consists of chapters, each of which is logical,
comprehensive, and definitive with respect to its topic. While appropriate
caveats exist, its forecasts are precise. The recommendations and conclusions
therein represent the unanimous agreement of the panel and no areas of
disagreement are spelled out. The result is typical of a competent panel (or
committee) activity. Based upon the expertise of the card idly selected partici
pants. the report is a reliable and comprehensive account of known information
and of projections based on this information and on current research and
development. In contrast, this Delphi was designed to complement the panel
report. I he planned approach was to provide an opportunity to indicate
uncertainties or disagreements about the subject and to evaluate quantitatively
the degree of uncertainty which exists within a large group of experts. The
Delphi product attempts to present an awareness of the areas which are subject
to differences of view and to highlight the topics which appear to concern the
respondent group. The Delphi provides a group evaluation of every’ statement
advanced by the respondents who, presumably, express their beliefs. Although
the results of this exercise include a number of statements which were rated
uncertain, risky, or unreliable by the whole group, this variation does not imply
that one dissenter from the group will be incorrect in retrospect. The group
view has a higher probability of being correct than the view of any one
individual. However, in the past, developments that significantly affected
industries were often unforeseen by most of the involved experts. Therefore, the
reader is cautioned not to extrapolate blindly from the group judgments
exhibited in a Delphi to assumed facts.
In this case, the Delphi exercise is a literal exploration of the minds of
experts in the steel and ferroalloy industries regarding their views on individual
items. I his exploration allowed a broader coverage of the subject area than was
possible in the panel report.The presentation of the Delphi results allows the
reader to compare easily his judgments with those of the group. No attempt is
made to arrive at conclusions or recommendations, or to present a definitive
view as was done in the panel activity.
Where the panel and Delphi activities overlap, there is considerable agree
ment in their forecasts. Figure 5 compares the consumption in steel of a number
of alloys, as predicted by the panel and the Delphi, and some of the qualitative
features of the two methodologies.
Other comparisons could be made between information presented in the
panel report and the Delphi predictions. For example, NMAB-276 projected
that carbon steel shipments would increase by 22 percent in the next ten years;
Nancy 11. (loklsicin
Genera) Applications:
Steel and Ferroalloy Industries
225
the increase projected on the Delphi graph for the next decade was 22 to 26
percent over the current figure. Also, the panel report stated that High
Strength Low .Alloy Steel (HSLA) is the fastest growing segment of the steel
industry; the Delphi results were that HSLA is one of the two fastest growing
Quantitative (Comparison
Predicted Ferroalloy Consumption in Steels
and Superalloys in 1980
(In Short I ons of Container Element)
Panel Report
Chromium
Cobalt
Columbium
Manganese
Molybdenum
Nickel
I ungslen
X’anadium
Delphi
validity of an assumption or the averaging of opposing judgments on the
validity of a given assumption. Of the 135 statements that fell into this
319,260
2.732
1,977
1.011,235
21.540
124.200
1.850
5.7*»(»
250,000-303,000
3,000-4,000
1,300-1.850
1,100,000-1,250,000
17.400-21,000
90,000-115,000
1,550 2,600
5,000 (i,200
classification, seventy-three reflected an actual disagreement among the respondents by exhibiting a high standard deviation.
'The following assumptions exemplify those that fell into the “not determin
able” category of the Delphi as a result of disagreement:
Delphi
form of Information
Weighting of Information
Provided__________________
I )isagreenient among Com
mittee Members or
Respondents______________
Presentation of Back
Ground Information_______
Recommendations
Range of Information
Provided
As was mentioned earlier, the panel report did not indicate the areas of
disagreement. In the Delphi category, “not determinable” with respect to
validity reflected either the inability of the entire group to determine the
NMAB-276
Qualitative Corn parison
'Type of Activity
segments of the industry.
Specific Comments
Most Rated on Reliability
Scale____________________
Indicated in Reliability
Score
Only as Randomly Generated
by Respondents
Not Specifically Stated
Broad. Reflecting Wide
Interest of Respondents
Committee
Ceneral Discussion
None
• Cobalt-iron base tool steels will be marketed.
• Continuing nickel shortages will establish permanent substitution.
• New techniques will allow significantly greater flexibility of substitution
among all<»ying elements based upon price changes.
• (Critical shortages of nickel will reoccur.
• No important new use for cobalt.
• Present and projected investment in ocean studies is too large to exclude
development of economical offshore mining except in short term (i.e., next five
years).
• Alloy steels will increase in nickel content.
• Tungsten content of carbides will decrease because of cost.
• Full alloy shipments will parallel automotive production.
Eliminated
• Low-cost method of preparing high-puritv iron powder will be developed
by 1975.
Thorough and Systematic
• More and cheaper scrap will result from urban waste recycling.
• Shortages of natural gas will be a primary limiting factor in the expansion
or modernization of the steel industry.
• Electron-beam refining will grow significantly.
(Consensus Recommenda
tions Indicated
Limited to Specific
Committee Subject Area
An additional fifty-seven assumptions were rejected by the Delphi respon
dents as either risky or unreliable. One must reflect that while each of the 667
assumptions were suggested by at least one expert in the Delphi respondent
Fig. 5.
Comparison of the panel and Delphi approaches.
group, approximately two hundred of these, or 30 percent, were considered less
than reliable by the group as a whole. Furthermore, there is pften considerable
value to decisionmakers in observing the nature of rejected assumptions. For
this reason the final Delphi report listed all the evaluated assumptions pertain
ing to any curve in the order of decreasing validity so one may observe the
complete span of the topics covered. Probably the most significant difference
between the Delphi and committee approaches is the itemization of what the
Delphi group could not agree on or what they rejected. Usually the psychologi-
226
Nancy H. Goldstein
■J
cal process in a committee of experts tends to eliminate these categories of
information from the final report.
In summary, the two reports did not always cover the same subject matter.
However, when they did touch upon the same subject matter, the results were
generally compatible.
1
Advice to Future Delphi Monitors and Designers
The monitor’s <experience with the Steel and Ferroalloy Delphi gives rise to a
number of observations
and advice
-------------------to those planning to monitor Delphis in the
future.
(1) When presenting statements for a vote, or synthesizing the respondents’
suggestions, be alert for ambivalent wording. Two separate statements may
appear as one, leading to confusion as to what should be voted upon. Vague
wording or easily misinterpreted wording may also lead to confusion.
(2) When editing respondents’ comments for clarity, try to preserve the
intent of the originator. When editing from round to round, avoid changing a
statement so that it has one meaning in round one and another in round two.
(3) Lay out the expected processing of the data throughout all the rounds of
, the Delphi before you finalize the design. You may, by circumstance, be forced
later to modify the procedure, but the process of planning ahead will usually
turn up any gross problems in your initial questionnaire design and its impact
on following rounds.
(4) Design the handling of your data so that
that each
each response
response can
can be
be processed
processed
(or punched for processing) as it comes in. Thus you will not have a frantic
rush to analyze all the responses at once when the last tardy return comes in.
(□) Keep track of how different subgroups in your respondent group vote on
specific items. I his can be very useful in analyzing the results and will
occasionally produce situations where you wish to let the respondent group
know that polarizations or differences based upon background exist.
(6) If you are covering a number of fields of expertise, make sure that each
field is adequately represented in your group.
(7) It should be mandatory that at least two professionals work on monitor
ing any one Delphi exercise, particularly when the abstracting of comments is a
notable portion of the exercise. With two individuals one can always review
what the other has done.
(8) 1 retest your questionnaire on any willing guinea pigs you can find
outside your respondent or monitor group. If you have a sponsor, it is useful to
go over the design of each round with some of his people before finalizing it.
IV. Evaluation
Pi
I
■y
I
I
-•
fl
I
'■
I
i
II
»
fl
:j|
-.4 H
’I
w
i
IV. A. Introduction
HAROLD A. LINSTONE and MURRAY TUROFF
Skeptics from the allegedly “hard” sciences have at times considered Delphi an
unscientific method of inquiry. Of course, the same attitude is often encoun
tered in the use of subjective probability (even in the face of considerable
mathematical theory developed to support the concept). The basic reason in
each case is the subjective, intuitive nature of the input.
Vet Delphi is by no means unordered and unsystematic. Even in the
Gordon-Helmer landmark Rand study of 1964, an analysis of certain aspects of
the process itself was included.1 The authors observed two trends: (1) For most
event statements the final-round interquartile range is smaller than the initial
round range. In other words, convergence of responses is more common than
divergence over a number of rounds. (2) Uncertainty increases as the median
forecast date of the event moves further into the future. Near-term forecasts
have a smaller interquartile range than distant forecasts.
It was also observed in all early forecasting Delphis that a point of diminish
ing returns is reached after a few rounds. Most commonly, three rounds proved
sufficient to attain stability in the responses; further rounds tended to show very
Tittle change and excessive repetition was unacceptable to participants.
(Obviously this tendency should not unduly constrain the design of Policy
Delphis or computerized conferencing which have objectives other than forecasting.)
We shall briefly review here some of the systematic evaluations made in
recent years.
Dispersion as a Function of Remoteness of Estimate
J
f
jij
fc
Martino has analyzed over forty published and unpublished Delphi forecasts.2
For every event the panel’s median forecast dates (measured from the year of
the exercise) and the dispersion were determined. A regression analysis was
performed and the statistical significance presented in terms of the probability
that the regression coefficient would be smaller than the value actually
obtained if there were no trend in the data.
The results are quite clear-cut. The remoteness of the precast date and the
*T. J. Gordon and O. Helmer, “Report on a Long Range Forecasting Study,” Rand Paper
P-2982, Santa Monica, California, Rand Corporation, September 1964.
2J. P. Martino, “The Precision of Delphi Estimates,” Technological Forecasting 1, No. 3 (1970), pp.
293-99.
The Delphi Method: Techniques and Applications, Harold A. Li nstone and Murray Turoff (eds.)
ISBN 0-201-04294-0; 0-201-04293-2
Copyright © 1975 by Addison-Wesley Publishing Company, Inc., Advanced Book Program.
All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or
transmitted, in any form or by any means, electronic, mechanical photocopying, recording, or
otherwise, without the prior permission of the publisher.
229
I
|
_'.iu
-il
i 4|
Harold A. Linstone and Murray Turoff
degree of dispersion are definitely related. 'The regression coefficient is in nearly
all cases highly significant for a single panel addressing a related set of events.
However, there is no consistent relation among different panels or within a
panel when addressing unrelated events.
Martino also finds that the dispersion is not sensitive to the procedure used:
in cases where only a single best estimate year is requested the result is similar
to that where 10 percent, 50 percent, and 90 percent likelihood dates are
stipulated.3
Distribution of Responses
Dalkey has analyzed the first-round responses by panels asked to respond to
almanac-type questions, i.e., those with known numerical answers.4 All re
spouses to each question are sstandardized
...........................
by subtracting the mean value and
dividing by the standard deviation for that question. The resulting distribution
of “standardized deviates” shows an excellent fit to a lognormal distribution.
Martino has applied the same techniques to the TRW Probe II Delphi using
10 percent, 50 percent, and 90 percent likelihood dates for 1500 events.5 Again
there is a very good fit to a lognormal distribution.
Optimism-Pessimism Consistency
Another interesting analysis on the TRW Probe II data was undertaken by
■Martino to ascertain whether a panelist tends to have a consistently optimistic
oi pessimistic bias to his responses,6 With each respondent providing 10
percent, 50 percent, and 90 percent likelihood dates, three standardized de
viates can be computed for each individual and a £given event. Taking the
means over all events of the standardized deviates for: a given individual and
likelihood, we find an interesting pattern. Most panelists are consistently
optimistic or pessimistic with respect to the three likelihoods, i.e., there are
relatively few cases where, say, the 10 percent likelihood is optimistic while the
50 percent and 90 percent likelihoods are pessimistic. Considering the totality
of events the individual panelist tends to be biased optimistically or pessimisti
cally with moderate consistency. However, the amount of the bias is not very
gieat, an optimistic panelist is pessimistic in some of his responses and vice
versa. In other words, each participant exhibits a standard deviation which is
comparable to, or greater than, his mean.
3In the first case the interquartile range of best estimates was used, in the second case the 10
percent to 90 percent span was taken.
*N. C. Dalkey, "An Experimental Study of Group Opinion,” Rand RM-5B«H-PR, Rand
Corporation, Santa Monica, California, Matc h 1969.
J. I’. Martino, “ The Logouttnality of Delphi Estimates," Teihnologual Foretasting I, No. 4 (1970)
pp. 355-58.
bJ. P. Martino, “The Optimism/Pessimism Consistency of Delphi Panelists,” Technological
Forecasting and Social Change 2, No. 2 (1970), pp. 221-24.
Evaluation:
Introduction
231
Accuracy of Forecasts
An apparent indicator of the value of Delphi as a forecasting tool is its
accuracy. Since the method was widely publicized only ten years ago, it is
difficult to have sufficient hindsight perspective to evaluate its success by this
measure. In any event caution is in order. The most accurate forecast is not
necessarily the most useful one. Forecasts are at times most effective if they are
self-fulfilling or self-defeating. The Forrester-Meadows World Dynamics model
has been sponsored by the Club of Rome in the hope that it will act as an early
warning system and prove to be a poor forecast. Delphi may be viewed
sTrnttUHy in terms of effectiveness.
\\e should also observe that long-range forecasts tend to be pessimistic and
short-range forecasts optimistic. In the long term no solution is apparent; in the
near term the solution is obvious but the difficulties of lystem synthesis and
implementation are underestimated.7 Thus in 1920 commercial use of nuclear
energy’ seemed far away. By 1949 the achievement appeared reasonable and in
1964 General Electric estimated that fast breeder reactors should be available
in 1970. I oday the estimate has moved out to the 1980s. The same pattern has
been followed by the supersonic transport aircraft. Buschmann has formulated
this behavior as a hypothesis and proposed an investigation in greater depth.9 If
this pattern is normal, forecasts should be adjusted accordingly, e.g., forecasts
more than, say, ten years in the future brought closer in time and forecasts
nearer than ten years moved out. Subsequently Robert Ament made a com
parison between a 1969 Delphi study on scientific and technological develop
ments and the 1964 Gordon-Helmer Rand study.*9 Focusing on those items
forecast in both studies, he found that all items originally predicted to occur in
years before 1980 were later shifted further into the future, i.e., the original
year seemed optimistic by 1969. On the other hand, two-thirds of the items
originally forecast to occur after 1980 were placed in 1969 at a date earlier
than that estimated in the 1964 study. Thus we find evidence here, too, of
Buschmann’s suggested bias.
Grabbe and Pyke have undertaken an analysis of Delphi forecasts of infor
mation-processing technology and applications.12 Forecast events whose occur
rence could be verified cover the time period 1968 to 1972. Although six
I he large cost overruns on advanced technology aerospace and electronics projects are evidence
of this trend (see Chapter III, A).
8E. Jantsch, "Technological Forecasting in Perspective,” OECD, Paris, 1967, p. 106.
9R. Buschmann, "Balanced Grand-Scale Forecasting,” Technological Forecasting 1 (1969), p. 221.
IOR. 11. Amrnt, "Comparison of Delphi Forecasting Studies in 1964 and 1969,” FUTURES,
March 1970, p. 43.
" I .J. Gordon and H. R. Ament, “Forecasts of Some Technological and Scientific Developments
and Their S<x ietal Consequences,” IFF Report R-6, September 1969.
'"E. M. Grablie and D. L. Pyke, “An Evaluation of the Forecasting of Information Processing
Technology and Applications,” Technological Forecasting and Social Change 4, No. 2 (1972), p. 143.
f
J
j
I
■ I
1
J
liaiold A. Linstonc and Murray l uroff
I'lvaluation:
degree of dispersion are <definitely related. The regression coefficient is in nearly
all cases highly significant for a <
’ panel addressing a related set of events.
single
However, there is no consistent relation
I
I
among different panels or within a
panel when addressing unrelated events.
Martino also finds that the dispersion is not sensitive to the procedure used:
measure In any event caution is in order. The most accurate forecast is not
"‘irTmir lhc "“>SI useful °"e' I'’orecasts are al
'"ost effective if they are
self, ulfdlmg <„ self-defeating. The Forrester-Meadows World Dynamics model
has been sponsored by the Club of Rome in the hope that it will act as an early
warning system and prove to be a poor forecast. Delphi may be viewed
similarly in terms of effectiveness.
We should also observe that long-range forecasts tend to be pessimistic and
short-range forecasts optimistic. In the long term no solution is apparent; in the
Dalkey has analyzed the first-round responses by panels asked to respond to
almanac-type questions, i.e., those with known numerical answers.4 All re
sponses to each question are standardized by subtracting the mean value and
dividing by the standard deviation for that question. 'The resulting distribution
of "standardized deviates" shows an excellent fit to a lognormal distribution.
Martino has applied the same techniques to the T’RVV Probe II Delphi using
10 percent, 50 percent, and 90 percent likelihood dates for 1500 events.5 Again
Another interesting analysis on the TRW Probe II data
was undertaken by
Martino to ascertain whether a panelist tends to have a <
■
- - consistently optimistic
or pessimistic bias to his responses.
6 With each respondent providing ”10
i
|w
‘rr/»n» 50. r-<Ar-/->An»
.
im
‘
percent.
percent, and nn
90 _percent
likelihood dates’ ihree’s'tandar
’drzed de”
viates can be computed for each individual and
--------------------- J a given event. 'Taking the
means over all events of the standardized deviates for
-------—
—
' a given individual and
likelihood, we find an interesting pattern. Most panelists are consistently
optimistic or jpessimistic with respect to the three likelihoods, i.e., there are
relatively few cases where, say. the 10 percent likelihood i.
is optimistic while the
50 percent and 90 percent likelihoods are
pessimistic. Considering the totality
of events the individual panelist tends to Ibe biased optimistically
or pessimisiieally with moderate consistency. However, the amount of the bias i.
— is not very
great; an optimistic panelist is pessimistic
j
’ ’ in
' some of his responses and vic:
vice
versa. In other words, each participant exhibits a
standard deviation which i:is
comparable to, or greater than, his mean.
i
A,, apparent indicator of the value of Delphi as a forecasting tool is its
Distribution of Responses
Optimism-Pessimism Consistency
Accuracy of Forecasts
231
^curacy. Since the method was widely publicized only ten years ago, it is
c i ficult lo have sufficient hindsight perspective to evaluate its success by this
m cases where only a single best estimate year is requested the result is similar
to that where 10 percent, 50 percent, and 90 percent likelihood dates are
si ipulated.
there is a very good fit to a lognormal distribution.
Introduction
near term the solution is obvious but the difficulties of system synthesis and
implementation are underestimated.’ Thus in 1920 commercial use of nuclear
J
energy seemed far away. By 1949 the achievement appeared reasonable and in
1964 General Electric estimated that fast breeder reactors should be available
in 1970. I Oday the estimate has moved out to the 1980s. The same pattern has
been followed by the supersonic transport aircraft. Buschmann has formulated
this behavior as a hypothesis and proposed an investigation in greater depth 9 If
this pattern is normal, forecasts should be adjusted accordingly, e.g., forecasts
more than, say, ten years in the future brought closer in time and forecasts
nearer than ten years moved out. Subsequently Robert Ament made a com
parison between a 1969 Delphi study on scientific and technological develop
6J. P. Martino, “The Optimism/Pessimism Consistency of Delphi Panelists,” Technoloeical
hot feasting and Social Change 2. No. 2 J 970). pp. 221-24.
j
ments and the 1964 Gordon-Helmer Rand study.16 Focusing on those items
forecast in both studies, he found that all items originally predicted to occur in
years before 1980" were later shifted further into the future, i.e., the original
1
i
year seemed optimistic by 1969. On the other hand, two-th.-. ds of the items
originally forecast to occur after 1980 were placed in 1969 at a date earlier
than that estimated in the 1964 study. Thus we find evidence here, too, of
Buschmann’s suggested bias.
Grabbe and Pyke have undertaken an analysis of Delphi forecasts of infor
mation-processing technology and applications.12 Forecast events whose occur
rence could be verified cover the time period 1968 to 1972. Although six
of JhBCha^IH, Ad)VanCed tCChnOlOgy aerospace and elCCtrOniCS projects
In the first case the interquartile range of best estimates was used, in the second
case the 10
percent to 90 percent span was taken.
‘N. C. Dalkey, “An Experimental Study of Group Opinion,”
Rand RM-5888-PR, Rand
Corporation, Santa Monica, California, March 1969.
^35^5 Mart*no’
FogHormality of Delphi Estimates," Technological Forecasting 1, No. 4 (1970),
I
I1'
II
eviden“
®E. Jantsch, “Technological Forecasting in Perspective,” OECD, Paris, 1967, p. 106.
R. Buschmann, “Balanced Grand-Scale Forecasting,” Technological Forecasting 1 (1969), p. 221.
K. H. Ament, “Comparison of Delphi Forecasting Studies in 1964 and 1969” FUTURES
March 1970, p. 43.
’
’
'' ]r,J-.G°rdon and H> R' Amcnb “Forecasts of Some Technological and Scientific Developments
and I heir Societal Consequences,” IFF Report R-6, September 1969.
'F k*’
anr D- L’ ,fyke’ ,An. Evaluation of the Forecasting of Information Processing
echnology and Applications, Technological Forecasting and Social Change 4, No. 2 (1972), p. 143.
■
■
232
Harold A. Linstone and Murray Turoff
different Delphi studies were used, eighty-two out of ninety forecasts covering
this period were taken from one study: the U.S. Navy Technological Forecast
Project. 1 he results appear to contradict the hypothesis that near-term forecasts
tend to be optimistic. In this case information-processing advances forecast four
to five years in the future occur sooner than expected by the panelists who were
drawn largely from government laboratories. There is, of course, the possibility
that these laboratories are not as close to the leading edge of technology in this
field as industrial and university research and development groups. Alterna
tively, the meaning of “availability of a technological application may be
interpreted differently by the laboratory forecasters and by the authors of this
a rticle.
Evaluation:
Introduction
233
to the occurrence of a familiar event. On the other hand, a longer-word
description raises the consensus level for unfamiliar events.
A corresponding pattern is found when expert respondents are compared to
nonexperts. The latter develop increasing consensus with longer-event descrip
tions. The experts, however, come to very high consensus with moderate
statement lengths (higher than the greatest nonexpert consensus) but fall to a
very low level of agreement with long statements. Apparently the addition of
words brings on an effect somewhat similar to that of disputations by Talmudic
scholars about minutiae.
Basis for Respondents’ Intuitive Forecast
Delphi Statements
The statements which comprise the elements of a Delphi exercise inevitably
reflect the cultural attitudes, subjective bias, and knowledge of those who
formulate them. This was recognized by Cordon and Helmer a decade ago and
led them to commence the first round with "blank’’ questionnaires. Every
student knows that multiple-choice examinations require insight into the in
structor’s mode of thought as well as the substance of the questions. Misin
terpretations of the given statements can arise in both superior and inferior
students. Grabbe and Pyke present examples of good and poor Delphi state
ments,11 Statements may be too concise, leading to excessive variations in
interpretation, or too lengthy, requiring the assimilation of too many elements.
Consequently, we would expect a constraint on the number of words leading to
the widest agreement in interpretation. Salancik. Wenger, and Helfer have
probed this question more deeply.14 They use an information theory measure
(bits) of the amount of information derivable from a distribution of responses to
a Delphi statement to measure consensus and the number of words needed to
describe an event as a measure of its complexity. The study uses a computer
development and application Delphi study as a test case. The authors find a
distinct relation between number of words used and amount of information
obtained, i.e., agreement in forecast dates. Low and high numbers of words
yield low consensus with medium-statement lengths producing the highest
consensus. In the particular case considered, twenty to twenty-five words To fin
the peak in the distribution. I his study also finds that the more familiar
respondents are with a specific computer application, the fewer words are
needed to attain agreement. If many words arc used, less information results as
x3/bld.
J- R- Salancik,
. Wenger, and E. Helfer, "rhe Construction of Delphi Event Statements,”
Technological Forecasting and Social Change 3. No. I (1971), pp. 65-73.
Salancik has examined the hypothesis that the panelists in a forecasting Delphi
assimilate input on feasibility, benefits, and potential costs of an event in aii additive
fashion to estimate its probable date of occurrence.15 The subject of the test is
again a panel forecast of computer applications. Separate coding of partici
pants’ reasons for their chosen dates in the three categories enables the author
to make a regression analysis. The second-round median date is made a linear
function of the number of positive and negative statements in each of the three
categories. He finds that the multiple regression strongly supports the hypothe
sis. The more feasible, beneficial, or economically viable a concept is judged,
the earlier it is forecast to occur. The three categories contribute about equally
to the regression.
In a second study independent assessments of feasibility and benefits are
rated for twenty computer applications and then combined to form the basisifor
a rank ordering. This ordering is then compared to the Delphi panelists’
responses. /Xgain the correlation supports the suggested model of Delphi input
assimilation. This paper adds another beam of support to the idea that Delphi
is a systematic and meaningful process of judgment synthesis.
Self-Rating of Experts
Dalkey, Brown, and Cochran tackle another aspect of Delphi: the expertise of
the respondents.16 With a given group we might consider two ways of improv
ing its accuracy: iterating the responses and selecting a more expert subgroup.
The latter process implies an ability to identify such a subgroup (e.g., by
self-rating) and a potential degradation in accuracy due to the reduced group
15J. R. Salancik, “Assimilation of Aggregated Inputs into Delphi Forecasts: A Regression
Analysis," Technological Forecasting and Social Change 5, No. 3 (1973), pp. 243-48.
,6N. Dalkey, B. Brown, and S. Cochran, “Use of Self-Ratings to Improve Group Estimates/’
Tpthflblogicftl ForreaslihH b
pp.-
5
I
II
I
I
234
Harold A. Linstone and Murray Turoff
size. The authors stipulate a minimum subgroup size to counteract this degra<1.Ilion .tnd (hey (on r a ( Icai sc p.naiion in sclf-ialings of low- and highex pci I IM- Ml I >g i OU ph. The cxpci i n Id i l.h wcic ( an led on I by I he an I hois using 282
univdsily sludcnls and \'d i liable ahnan.K -lvp<* (pics I ions. The conclusions: ( I )
self-rating is a meaningful basis for identification of expertise, and (2) selection
of expert subgroups improves the accuracy to a somewhat greater degree than
docs feedback or iteration.
One must raise the question whether an experiment based on almanac type
questions serves as an adequate basis for a conclusion about the validity of
self-ratings of expertise for forecasting Delphis. While the lognormality be
havior exhibited a similar pattern for factual (almanac-type) and forecasting
cases, this similarity might not carry over for self-ratings.
/\nd there are other fascinating unanswered questions. Why do women rate
themselves consistently lower than men? Should only the expert subgroup
results be fed back to the larger group in the iteration process? How do age,
education, and cultural background condition the response of individuals?
The four articles in this chapter provide us with further evaluations of the
process. When we use Delphi to draw forth collective expert judgments, we are
actually making two substitutions: (1) expert judgment for direct knowledge,
and (2) a group for an individual. In the first article, Dalkey strives to develop
some mathematically rigorous underpinnings, i.e., a start toward a theory of
group estimation. It quickly becomes evident that we still have much to learn
about this process. Dalkey emphasizes the concept of “realism,” or “track
record," to describe the expert’s estimation skill and the theory of errors for the
group. But the final verdict on their applicability is by no means in.
Scheibe, Skutsch, and Schofer report on several highly instructive findings
based on research in the application of Delphi to the derivation of explicit goals
and objectives. Analysis of a Delphi goal-formulation experiment for urban
systems planning yielded the following important results:
(1) The three-interval scaling methods used—simple ranking, a rating scale,
and pair comparisons—give essentially equivalent scales. The rating scale is
found to be most comfortable to use by the participants.
(2) Respondents are sensitive to feedback of the scores from the whole group
and lend to move (at least temporarily) toward the perceived consensus.
(3) 'There is only a modest tendency for the degree of confidence of an
individual with respect to a single answer to be reflected in movement toward
the center of opinion, i.e.. less confident members exhibit a somewhat larger
movement in the second round.
(4) Stability of the distribution of the group’s response along the interval
scale over successive rounds is a more significant measure for developing a
stopping criterion than degree of convergence. The authors propose a specific
stability measure.
Evaluation:
Introduction
235'
Next, Mulgrave and Ducanis discuss an experiment which focuses on the
behavior of the dogmatic individual in sue < cssivc Delphi rounds. Surprisingly,
di< high (logmalism gioup exhibits significantly more changes than the lowdoginatisin group. It is the authors’ belief that the dogmatic individual looks to
authority for support of his view. In the absence of a clearly defined authority,
he views the median of the group response as a surrogate.
There clearly exists the possibility of an unnatural overconsensus. Confor
mists may “capitulate" to group pressures temporarily, on paper. It would be
interesting to compare the behavior of such psychological types in a Delphi
with that in a conventional committee.
Finally, Brockhoff examines a series of hypotheses on the performance of
forecasting groups using the Delphi technique and face-to-face discussions in a
Lockcan context. He focuses on short-range forecasting and small homogeneous
groups. Staff members of local banks trained in economics are queried about
data concerning financial questions, banking, stock quotations, and foreign
trade. Croups vary in size from eleven to four participants (the latter below the
size considered minimal by Dalkey, Brown, and Cochran17). The Delphi
process uses an interactive computer program for structuring the dialogue as
well as computing intermediate and final results. The correlation of self-rating
of expertise with individual or group performance, the relation between infor
mation exchange and group performance, and the relevance of almanac-type
fact-finding questions for short-term forecasting analysis are among the ques
tions examined. One may speculate whether the dogm.: ism aspect raised by
Mulgrave and Ducanis plays a significant role in groups of the type used in
Brockhoff’s experiments.
For the reader the thrust of this chapter is that, to develop proper guidelines
for its use, we can and should subject Delphi to systematic study and evaluation
in the same way as has been the case with other techniques of analysis and
communication. Much still needs to be learned!
1
■
. . B. 1 owarcl a 1 heory of Group Estimation *
NORMAN C. DALKEY
Introduction
1 he term "Delphi has been extended in recent years to cover a wide variety of
types of group interaction. Many of these are exemplified in the present
volume. It is difficult to find clear common features for this rather fuzzy set.
Some characteristics that appear to be more or less general are: (1) the exericse
involves a group; (2) the goal of the exercise is information; i.e., the exercise is
an inquiry; (3) the information being sought is uncertain in the minds of the
group; (4) some preformulated systematic procedure is followed in obtaining
the group output.
This vague characterization at least rules out group therapy sessions (not
inquiries), team design of state-of-the-art equipment (subject matter not un
certain), brainstorming (procedure not systematic), and opinion polls (responses
arc not treated as judgments, but as self-reports). However, the characterization
is not sufficiently sharp to permit general conclusions, e.g.. concerning the
effectiveness of types of aggregation procedures.
Rather than trying to deal with this wide range of activities, the present essay
is restricted to a narrow subset. The subject to be examined is group estimation—
the use of a group of knowledgeable individuals to arrive at an estimate of an
uncertain quantity. The quantity will be assumed to be a physical entity—a
date, a cost, a probability of an event, a performance level of an untested piece
of equipment, and the like.
Another kind of estimation, namely, the identification and
assessment of
value structures (goals, objectives, etc.) has been studied
to some extent, and a
relevant exercise is described in Chapter VI. Owing to the difficulty of
specifying objective < rilcria foi th< ■ pt rfonnaiKc <>l a group on this task, it is not
considered in the present paper.
Io specify the group estimation
piocess a little more sharply , we consider a
{ /,} of individuals, ian eveni space 0={El'l where E can be either
Hioup /={/,)
discrete or continuous, and a response space /? = {/?} which consists of an
estimate for each event by each member of the group. In addition, there is an
external process P= { P( E})). which determines the alternatives in E which will
occur. Depending on the problem. P can either be a 3-function on E—i.e., a
specification of which event will occur—or a probability distribution P on the
•Rj-search reported herein was conducted under Contract Number F30602-72-C-0429 with the
Advanced Research Projects Agency. Department of Defense.
The Delphi Method: Techniq
ISBN 0-201.04294-0; 0-201 .oTc."'1 AppliCaliOnS- Harold A l-ins,one a"d
Turoff (eds.)
Gopynglil © 197a by Addison-Wesley Publishini; Company. Inc.. Advanced Book Program
All nghts reserved. No pari of ibis publication may lx- reproduced, stored in a retrieval system, or
transmitted, in any form or by any means. »•!<•< tn»ni< . mechanic.J photocopying, n-cordiiu1
otherwise, without the
In:..„ ,4 0 ... j,;
(
> •h»
Evaluation:
I heory of Group Estimation
237 i .■>
event space. In general Pj is unknown. For some formulations of the
group
< SU,nanon process, tt is necessary to refer to the a priori probability of
an event.
Ih.s ts not the same as the external process, but rather, is (in the
present
context) the probability that is ascribed to an event without 1
knowing the
md.vnduaI or group estimates. This a priori probability will be designated by
In many cases the
are simply selections from E. The weatherman savs “It
Wtll ram tomorrow -a selection from the two-event space ra.n tomorrow ..nd no
ram tomorrou-. I he long-range technological forecaster says, “Controlled nuclear
fusion will be demonstrated as feasible by 1983"-selection of a single date out
of a continuum. In these cases the
can be considered as 0’s and Ts, 1 for the
selected event and 0 for the others. Usually, the 0’s are left implicit. More
complex se ecttons can be dealt with-“It will either rain or snow tomorrow.”
controlled nuclear fusion will be demonstrated in the interval 1980-1985”—
by allowing several 1’s and interpreting these as an or-combination. Selections
can also be considered as spec.al cases of probability distributions over the
event space. In the case of probability estimates, the
can be probability
qu^mhies tS
ete alternatives’ Or continuous distributions for continuous
nor\k'ndrf eStimatywhich is “retimes used in applied exercises, but which is
not dmectly expressible in terms of elementary event spaces, is the estimation of
he functional relat.onsh.p between two or more variables (e.g., the extrapolaon of a trend). Such an estimate can be included in the present formalism if
the relationsh.p ls suff.ctently well known beforehand so that all that is
required is specification of some parameters (e.g., estimating the slope of a
■ near trend . Although of major practical importance, estimates of complex
uncuonal relat.onships have received little laboratory or theoretical treatment
In part.cular, there has been no attempt to develop a scoring technique for
mcasuimg (he excellence of such estimates.
1
to the group /, event space E, and response space /?, a Delphi
exercise mvolves a process G=G[/,EJ<] which produces a group response G
for each event
m the event space. Square brackets are used rather than
parentheses in the express.on for G to emphasize the fact that generally the
group esumanon process cannot be expressed as a simple functional re
lationship. 1 he process may involve, for example, discussion among members of
the group, other kinds of communication, iteration of judgments with complex
selection rules on what is to be iterated, and so on.
P
One other piece of conceptual apparatus is needed, namely, the notion of
s ow in'n
PerfOr'rl;lnuCe' Development of scoring techniques has been
s ou n DtIphl pract
probabIy because ,n mos(
data for measuring performance either is unavailable, or would require waiting
ccace or so. But in addition, the variety of suhlifeCt fiWteFSt th?
motivations for applied Mutliw, np.d i|k‘ abjuring effect of the i^dk-ttl u..’ria'Dty
am
with B-ri.
liky i
forvcji-ihig bf •okt! A'Ul tseh-
i
-1
h
A
I
1
§
)
Norman
Dalkey
nological events have inhibited the attempt to find precise measures of perfor
mance.
1 n t he present paper, emphasis will be put on measures related to the
accuracy of estimates. 'There is a Ilarge family
'
of such measures, depending on
the form of the estimate, and depending on the interests
— of the
...j user of the
estimate. Tor this essay, measures will be restricted to what might be called
scientific criteria, i.e., criteria which do not include potential economic benefits
to the user ((or potential costs in terms of experts’ fees, etc.) or potential benefits
in facilitating group action.
Tor simple selections out of discrete event spaces a right/wrong measure is
usually sufficient, for example, crediting the estimate with a 1 or 0 depending
on whether it is correct or incorrect. However, as in the related area of
performance testing in psychology, the right/wrong measure is usually augmerited by computing a score—total number right, or proportion right, or
right-minus-wrong, etc.—over a set of estimates.
Tor simple selections out of continuous spaces (point estimates), a distance
measure is commonly employed, for example, difference between the estimate
and the true answer. However, if such measures are to be combined into a score
over a set of estimates, some normalizing procedure must be employed to effect
comparability among the responses. One normalizing procedure for always
positive quantities such as dates, size of objects, probabilities, and the like, is the
log error, defined as
/?,
Error = log|y|,
where 7’is the true answer and /?( is the individual response. 'The vertical bars
denote the absolute value (neglecting sign). Dividing by T equates proportional
errors, and taking the logarithm uniformizes under- and over-estimates. Com
parable scoring techniques have not been worked out for quantities with an
inherent zero, i.e,, quantities admitting both positive and negative answers.
Such quantities are rare in applied exercises. Whether this is because that type
of quantity is inessential to the subject matter or whether it is due to avoidance
by practitioners is hard to say.
Tor probability estimates, some form of probabilistic scoring system appears
to be the best measure available. I he theory of probabilistic scoring systems is
under rapid development. It is usually pursued within the ambit of subjective
probability theories, where the primary property sought is a reward system
which motivates the estimator to be honest, i.e., to report his “true” belief.
This requirement can be expressed as the condition that the expected score of
the estimator should be a maximum when he reports his true belief. If q—
is the set of probabilities representing the actual beliefs of the estimator on
event space {^),
is his set of reported probabilities, and ^(/?) is the
reward he receives if event /y occurs, then the honesty condition can bp written
Evaluation:
I heory of Group Estimation
239
in the form
J
J
(1)
The expression on (he left of the inequality is the individual’s subjective
expectation if he reports his actual belief; the expression on the right is his
expectation if he reports something else.
Tormula (1) defines a family of scoring (reward) systems often referred to as
"reproduemg scoring systems” to indicate that they motivate the estimator to
reproduce his actual belief.
It is not difficult to show that the theory of such scoring systems does not
depend on the mterpretation of q as subjective belief; it is equally meaningful if
q is interpreted as the objective probability distribution P on E. With this
interpretation the estimator is being rewarded for being as accurate as possible
-his objective expectation is maximized when he reports the correct probability distribution.
I his is not the place to elaborate on such scoring syst- is (see 111 [21 [31).
Although (1) leads to a family of reward functions, it is sufficient for the
purposes of this essay to select one. The logarithmic scoring system
S;(/?) = /llog/?/ + 5
y
I
i
(2)
has a number of desirable features. It is the only scoring system that depends
solely on the estimate for the event which occurs. The expected score of the
estimator is precisely the negative entropy, in the Shannon sense [4], of his
forecast. It has the small practical difficulty that if the estimator is unfortunate
enough to ascribe 0 probability to the alternative that occurs, his score is
negauvely infinite. This can usually be handled by a suitable truncation for
very small probabilities.
Within this restricted framework, the Delphi design “problem” can be
exptessed as fmdmg processes G which maximize the expected score of the
group response. 1 his is not a well-defined problem in this form, since the
expectation may be dependent on the physical process being estimated, as well
as on the group-judgment process. There are two ways to skirt this issue. One is
to attempt to find G’s which have some optimality property independent of the
physical process. 1 he other route is to assume that knowledge of the physical
process can be replaced by knowledge about the estimators, i.e., knowledge
concerning their estimation skill. The next section will deal with the second
possibility.
I here are two basic assumptions which underlie Delphi inquiries: (a) In
situations of uncertainty (incomplete information or inadequate theories) ex
pertjudgment can be used as a surrogate for direct knowledge; I sometimes rail
this the “one head H hPHer Ihrtfl
FhIf- (h) In 4 n'W? YQllpty
Mtuafions
tine■•■rtiimty, ;• group judgment (amalgamating the judgments of h group of
I
1
! |
I
I
I
a
I
u
Xonnan C. B.dkcx
experts; is preferable to the judgment of a typtca! member of the group the
heads are better than one” rule.
1 he second assumption is more closely associated with Delphi than the first
Which has more general application in decision analysis. These two assumptions
do mot. of course, exhaust all the factors that enter into the use of Delphi
techniques. I hey do appear to be fundamental, however, and most of the
remaining discussion in this paper will be concerned with one or the other of
the two.
Individual Estimation
Using the expert as a surrogate for direct knowledge poses no problems as long
as the expert can furnish a high-confidence estimate based on firm knowledge
of Ins own. Issues arise when existing data or theories are insufficient to support
a lugh-confidencc estimate. Under these circumstances, for example different
experts are likely to give different answers to the same questions
Extensive “everyday experience" and what limited experimental data exist
on the subject strongly support the assumption that knowledgeable individuals
can make useful estimates based on incomplete information. This general
assumption, then, is hardly in doubt. What is in doubt is the degree of accuracy
of specific estimates. What is needed is a theory of estimation that would enable
the assignment of a figure of merit to individual estimates on the basis of
readily available indices.
An interesting attempt to sidestep this desideratum is to devise methods of
rewarding experts so that they will be motivated to follow certain rules of
rational estimation. One approach to the thcorv of probabilistic scoring sys
tems described in the introduction is based on this sirategem |5].
1 he outlines of such a theory of estimation have been delineated in
the
literature of decision analysis; but it is difficult to disentangle from an atten
dant conceptualization of a prescriptive theory of decisionmaking, or as some
times characterized, the theory of rational decisionmaking. In the following I
will try to do some disentangling, but the subject is complex and is and ought
may still intermingle more than one might wish.
In looking over the literature on decision analysis, there appear to be about
•six desirable features of estimation that have been identified. The number is not
sharp, since there are overlaps between the notions and some semantic difficul
ties plague the classification. The six desiderata are honesty, accuracy, definiteness,
realism, certainty, and freedom from bias.
Honesty is a clear enough notion. In most cases of estimation, the individual
has a fairly clisrincl percepliun of his "actual belief," or put another way, he has
.1 telalively clear perception whether his reported estimate matches his actual
belief. I his is not always the case. In situations with ambiguous contexts such
as the group-pressure situations created by .Asch [6], some individuals appear to
lose the distinction. The reason for wanting honest reports from estimators is
Evaluation:
I heoiy of Group Estimation
241
also clear. Theoretically,
any report, honest or not, is valuable if the user is
aware of potential distortions and
J can adjust for them. But normally such
information is lacking.
■ Iccuracy is also a fairly straightforward i J...,
notton, and is measured by the score
m most cases. It becomes somewhat cloudy in the case
• _
------- 5 °f probability estimates
for single events, where an individual can make a good score by chancei In IhZ
I
jj
cast, t te average score over a sequence of events is more diagnostic But the
notion of accuracy then becomes mixed with the notion of reahsm Ci
meaningfulness of the term, the
the desirability
is clear
'
desirability of
of accuracy
accuracy is
clear.
□robin""'' mea,sures the dcgrec of sharpness of the estimate. In the case of
casethe
of
probabthues on discrete event spaces, it refers to the degree to which
pto abilities approach 0 or 1 and can Be measured by S’" /?2 In the
t
!
».r„„ee or lhe dr.ponlon, |„ the c„ „( Kteioni ft. oomp.r.blo JoUon k
prcfeicnce
has also been postulated by some analysts [7], Iiihe case of
I
-
"—I”Xr-yi|i
n
biieLivenerkal feeding that
i
;-<le. than “wishy-washy” estimates in .I.e^ighliid'iiS^siso
he eel,ng that an individual who makes a prediction with a probabditv of 8
and it turns out correct) knows more about the phenomenon being predicted
than someone who predicts a similar event with probability 6
Al of this is a little difficult to pin down. In the experiments of Girshick el
al. |8J, there was almost no correlation between a measure nf
.
•■4
overlap between the notion of definiteness and uncertainty, whkh is discussed
considered a'virtuT15’
that definiteness is
Reahsrn refers to the extent that an individual’s estimates are confirmed bv
events. It ,s thus closely related to accuracy. However, accuracy refers to I
mgle estimate whereas realism refers to a set of estimates generated by an
md.v.dual. Other terms used for this notion are cation [9],^^ [10],L*
Because the notion of realism is central to the first principle of Delphi stated
-r
"its
t
Norman C. Dalkey
242
probability, and count the number of times the estimate was confirmed.
Presumably, if the estimator is using the notion of probability correctly, the
relative frequency of successes in that sequence should be approximately equal
to the estimated probability. Given enough data of this sort for a wide range of
different estimates, it is possible in theory to generate a realism curve for each
individual, as illustrated in Fig. 1.
1
Evaluation:
Theory of Group Estimation
243
the individual were fully realistic, the desired probability would be 7?-. At first
sight, it might appear that one individual, given his realism curve, is all that is
needed to obtain a desired estimate, since the curve furnishes an “objective”
translation of his reports into probabilities. However, for any one specific
estimate, the reports of several individuals typically differ, and in any case the
realism curxe is not, by itself, a measure of the expertness or knowledgeability
of the individual. In particular, the frequency with which the individual reports
relatively high probabilities has to be taken into account.
/\s a first approximation, the knowledgeability 2C,- of individual i can be
measured by
■
j
K,= f's(R,)D(R,\
1
I
i!
•'O’
RFfCiR,)
RELATIVE
FREQUENCY
OF CORRECT
RESPONSES
0 E
0
R
Fig. 1. Typical realism curve.
In Fig. 1 the relative frequency with which an estimate of probability Rt is
verified, RF(C\RI>
) (“C” for “correct”), is plotted against the estimate. Realism
can be defined as the degree to which the /?/•’(Cj/?,) curve approximates the
theoretically fully realistic curve, namely the dashed line in Fig. 1, where
/?/’’(C|/?,)= R,. Figure 1 illustrates a typical realism curve where probabilities
greater than
are “overestimated” and probabilities less than | are underestimated [11].
Various quantities can be used to measure the overall realism of an
estimator. !q(RF( C|/?,) - R,)2
where
is the distribution of the
estimator’s reports Rt—roughly the relative frequency with which he uses the
various reports Rt—is a reasonable measure. However, for most applications of
the concept, it is the realism curve itself which is of interest.
If such a curx’e were available for a given inthvirluaL H .could be used duecdy
m obtain the
;
.. L. particular, if
where
is the probabilistic score awarded to each report Ri and 2)(/?,•) is, as
before, the distribution of the reports Rt.
It is easy to verify two properties of Kt: (a) Kl is heavily influenced by the
degree of realism of the estimator. For a given distribution of estimates, Z)(/?,),
K' is a maximum when the individual is fully realistic, (b)
is also influenced
by the average definiteness of the estimator. The higher the definiteness (e.g.,
measured by J A,2 £)(/?,)), the higher the expected score.
Theoretically, one miight pick the individual with the highest K rating and
use him exclusively. There are two caveats against this procedure. On a given
question, the individual with the highest average K may not furnish the best
response; and, more in the spirit of Delphi, if realism curves are available for a
set of individuals, then it is sometimes feasible to derive a group report which
will have a larger average score than the average score of any individual—in
short, the K measure for the group can be higher than the K measure for any
individual.
As far as the first principle—substitution of expert judgment for knowledge—
is concerned, the question whether realism curves exist for each individual is a
crucial one. Detailed realism curves have not been derived for . .e types of
subject matter and the type of expert desired for applied studies. In fact,
detailed track records for any type of subject matter are hard to come by. Basic
questions are: Is there a stable realism curve for the individual for relevant
subject matters? How general is the curve—i.e., is it applicable to a wide range
of subject matters? How subject is the curve to training, to use of reward
systems like the probabilistic score, to contextual effects such as the group
pressure effect in the Asch experiments?
Certainly is a notion that is well known in the theory of economic decision
making. It has not played a role in the study of estimation to the same extent.
In the case of economic decisionmaking, the distinction has been made between
risk (situations that are probabilistic, but the probabilities are known) and
WMlalnly
wljcrp |h?
W
town)
I
1
I
flT-
Maw
h.• • ■ appear to believe that in the area of estimation this distinction breaks
$
244
Norman (J. 1 Jal key
down—uncertainty is sufficiently coded by reported probabilities. However, the
distinction appears to be just as applicable to estimation as to any other area
where probabilities arc relevant. Consider, for example, the situation of two
coins, where an individual is asked to estimate the probability of heads. Coin .A
is a common kind of coin where the individual has flipped it several times. In
this case, he might say that the probability of heads is J with a high degree of
confidence. Coin B, let’s say, is an exotic object with an unconventional shape,
and the individual has not flipped it at all. In the case of coin B he might also
estimate a probability of J for heads, but he would be highly uncertain whether
that is the actual probability. Probability J, then, cannot express the un
certainty attached to the estimate for the second coin.
A closer approximation to the notion of uncertainty can be obtained by
considering a distribution on the probabilities. For example, the individual
might estimate that the probability of the familiar coin has a tight distribution
around
whereas the distribution for the unfamiliar coin is flat, as in Fig. 2.
I he independent variable is labeled q to indicate that it is the individual’s
belief, and not necessarily his report, which is being graphed.
UNFAMILIAR
I
I
✓X
/
/ >
0
245
The problem of representing uncertainty in precise terms is closely related to
past attempts to translate lack of information into probabilities by means of
pnnctples such as the “law of insufficient reason,” or the rule of equal
ignorance. These have invariably lead to paradoxes [13].
4
Using the idea of the dispersion of a second-level distribution as an
approximate measure of uncertainty, there is some interaction between the
notions of realism, definiteness, and certainty. It is not possible for a set of
to be simultaneously realistic, definite, and uncertain. Assuming that j
the mdmdual will g.ve as his first level report Rt the mean of his second-level I
dtstribution then as R, approaches 1 or as R, approaches 0, the standard |
detiation of the distribution Z)(9) approaches 0. Figure t
*
3 illustrates this fi
coupling for /?, = .9. If the individual is realistic and estimates
s a probability of
.9 for a given event, then
hk hJah^r-i
then the
the standard
standard deviation
deviation of
of his
higher-level distribution J
for that estimate must be small.
-t
s
1
FAMILIAR
0
I
-I.
I
I
I
I
I
I
I
I
.5
1
CT
I
I
Q
Theory of Group Estimation
I
I
I
^cr
Evaluation:
l_L
0
1
q
.9
1
I
-•'I
q
Fig. 3. Illustration of coupling between certainty and definiteness.
Fig. 2. Uncertainty represented as a higher-level distribution.
The use of a higher-level distribution is only an approximation to the notion
of uncertainty, since the distribution itself might be uncertain, or, in more
familiar language, the distribution may be “unknown.” The use of additional
levels has been suggested, but for practical reasons seems highly unappealing.
Unfortunately, the coupling applies only to the extremes of the
/ to the extremes of the 0 to 1
interval. At q- J, D can be
be about
about anything,
anything, and
g
the estimator still be realistic.
If the average probabilistic score for an estimate with
second-level distribu
tion D is computed, the average score is influenced only by the mean and
independenl „( D. Tbu. p^.biU„fc
„„
i
f.
246
Evaluation:
Norman C. Dalkey
I
247
good reason why what holds for achievement and intelligence tests should not
also hold for “real life” estimates. Almost by definition, the area of most interest
in applications is the area of difficult questions. If so, assuming that the set of
counterpredictive questions can be identified before the fact, then a good fair
coin would be better than an expert. It is common in experimental design to
use randomization techniques to rule out potential biases. There is no logical
reason why randomization should not be equally potent in ruling out bias in
the case of estimation.
At the present, the only “visible” index of certainty is the self-rating—i.e., a
judgment by the individual of his competence or knowledgeability concerning
(he estimate. This has turned out to be a significant index for rating group
estimates [14]; it is not so effective for individual estimates. Due to the lack of a
theoretical definition of the self-rating, it has not been possible to include it in a
formal theory of aggregation. However, the self-rating has proved to be
valuable for selecting more accurate subgroups [15].
liias is a term that has many shades of meaning in statistics and probability.
1 am using the term to refer to the fact that there may be subclasses of events
for which RF{C\Ri) may be quite different from the average relative frequency
expressed by the realism curve. Of course, for this to be of interest, the
subclasses involved must be identifiable by some means other than the relative
frequency. It is always possible after the fact to select a subset of events for
which an individual has estimated the probability /?( which has any RF{C\Rl).
In the theory of test construction, e.g., for achievement tests or intelligence
tests, it is common to assume an underlying scale of difficulty for the questions,
where difficulty is defined as the probability that a random member of the
target population can answer the question correctly [16]. This probability will
lunge from 1 for very easy questions to 0 for very hard questions, as illustrated
bv the solid curve in Fig. 4. From the standpoint of the present discussion, the
significant fact is that when a class of questions is identified as belonging to the
\cry difficult group in a sample of the population, that property carries over to
other members of the population—in short the property of being very difficult
■
■
I
i
1
PROBABILISTIC
PROCESS
1
PROBABILITY
OF CORRECT .5
ANSWER
s
B.
I
I
I
I
1
j
OBSERVED
PROCESS
I
I
I
0
d
DIFFICULTY
is relatively well defined.
At some point in the scale of difficulty, labeled d in Fig. 4, a typical member
of the population could increase his score by abandoning the attempt to
“answer” the question and simply flipping a coin (assuming that it is a
tiue/false or yes/no type of question). Put another way, from point d on, the
individual becomes a counterpredictor—you would be better off to disbelieve
his answers.
Contrasted with this notion of difficulty is the notion that underlies theories
of subjective probability that, as the individual's amount of information or skill
declines, the probability of a correct estimate declines to 50 percent as il
lustrated by the dashed curve in Fig. 4. Ironically, it is the probabilistic notion
(hat influences most scoring schemes, which assume that the lestee can achieve
50 percent correct by “guessing,” and hence the score is computed by subtract
ing the number of wrong answers from the number right. By definition, for the
more difficult items, the lestee cannot score 50 percent by “guessing” unless
that means literally tossing a coin and not trusting his “best guess.
If it turns out that “difficult” questions in the applied area have this
property, even for experts, then the first principle does not hold for this class.
/Mthough there are no good data on this subject, there does not appear to be a
Theory of Group Estimation
Fig. 4. Scale of difficulty in test construction.
The four notions, honesty, accuracy, definiteness, and precision, are all tied
together by probabilistic scoring systems. In fact, a reproducing scoring system
rewards the estimator for all four. As pointed out in the introduction, condition
(1) defines the same family of scoring systems whether q is interpreted as
subjective belief, or as objective probability. Thus, the scoring system rewards
the estimator for both honesty and accuracy. In addition, the condition leads to
the result that ^JqJSj{q') is convex in q. This convex function of q can be
considered as a measure of the dispersion of q\ and in fact, three of the
better-known scoring systems define three of the better-known measures of
dispersion. Thus, if
is the quadratic scoring system
I
1
i
J
then 'ZJq)Si(q') = ^jq^ which is a measure of variance. If Sj is the spherical
■
*
2-18
Norman C. Dalkey
Evaluation:
Theory of Group Estimation
249
scoring system
I
mathematical rule for deriving a group
i—
response, from a set of individual 5
responses; thus, an elementary group estimation
process can be defined as a
function,
G(E,/,/?).
as
I
then 2,?//?) = ^^ , a measure similar to the standard deviation. Finally,
Theory of Errors
for the logarithmic scoring system.
which is the negative of
the Shannon entropy, another measure of definiteness.
Realism enters in a more diffuse fashion. In general, the probabilistic score
for a single event is not very diagnostic, since the individual may have obtained
a high (or low) score by chance. 'I hus. as for most scoring systems, an average
(or total) score over a large set of questions is the usual basis for evaluation. But
over a large set of (juestions, the average score is determined by the realism
curve of the individual, in conjunction with the relative frequency with which
he makes reports of a given probability. In general, if the estimator is not
realistic, he will lose
f (RF(CI/<)5(/;/■•(CI/<))-«/•(CIR,)S(/?,))D(/?,).
•'0
As pointed out above, the probabilistic score does not include a penalty for
tin. ertainty, nor does it include a penalty for bias, except where bias shows up
ill the realism curve. 1 he latter case is simply the one where, for whatever
reason, the individual is faced with a stream of questions in which the number
of questions biased in a given direction is greater than the number biased in the
opposite direction.
Io sum up this rather lengthy section: The postulate that, in situations of
uncertainty, it is feasible to substitute expert judgment for direct knowledge is
grounded in a number of empirical hypotheses concerning the estimation
process. I hese assumptions are, primarily, that experts are approximately
realistic in the sense defined above, that the realism curve is stable over a
relatively wide range of questions (freedom from bias), and that knowledgeabil
ity is a stable property of the expert. At the moment, these are hypotheses, not
well-demonstrated generalizations.
%
1'-1
I his approach interprets the set of judgments of
a group of experts as being
1
similar to the set of readings taken with an i
instrument subject to random error.
It seems most appropriate when applied to point estimates of
.
-----a continuous i
quantity, but formally at least, can be applied to any type of estimate. In
analogy with the theory of errors for physical i------ -- ■
measurements, a statistical
measure of central tendency is considered to be the best estimatrof^
family. Some measure of dispersion is taken to represent a confidence interval
about the central value.
Relevant aspects of the individual estimation
process such as skill or amount
of information of the expert, are interpreted as
features of the “theory of the 1"
instrument.”
This point of view appears to be most popular in the Soviet Union [17]; |
however, a rough though unexpressed version of this approach underlies ... _
much
of the statistical analysis accompanying many applied Delphi studies. To r—
- - - _ j my
knowledge, this approach has not been
been developed
in a
a mhprpnt
coherent theory, u..*
but >
developed in
rather, has been employed as an informal
informal “
“interpretation
”
—
i.e„
as
a
useful
interpretation”—i.e., as a useful :1
analogy.
The theory-of-errors approach has the advantages of simplicity, and similar
ity with well-known procedures in physical measurement theory. Much of the
empirical data which have been collected with almanac and short-range
prediction studies is compatible with the analogy. Thus, the distribution of
estimates tends to follow a common form, namely the lognormal [181. If thd
random errors postulated in the analogy are assumed to combine multiplica
tively (rather than additively as in the more common Gaussian theory), then a
lognormal distribution would be expected.
The geometric mean of the responses is more accurate than the
average
response; or more precisely, the error of the geometric mean is smaller than the 1!
average error. Since the median is equal to the geometric mean for
a lognormal
distribution [19], the median is a reasonable surrogate, and has been the
most
widely used statistic in applied studies for the representative group response. I
I
he error of the median is, on the average, a linear function of the standard
eviation [20], which would be predicted by the theory of errors. The large bias
observed experimentally (bias = error/standard deviation) is on the average a
constant which again would be compatible with the assumption that experts
perform like biased instruments.
J
Theoretical Approaches to Aggregation
Assuming that, for a given set of questions, we can accept the postulate that
expert judgment is the “best information obtainable,” there remains the
question how the judgments of a group of experts should be amalgamated. In
the present section, three approaches to this issue are discussed. The discussion
is limited to elementary forms of aggregation, where the theory consists of a
k
■k
f
■i
ii
/.A)
Norman (Dal key
Although the analogy looks fairly good, there are several open questions that
prevent the approach from being a well-defined theory. 'I’here does not exist at
present a theory of the instrument” which accounts for either the observed
degree of accuracy of individual estimates or for the large biases observed in
experimental data. Perhaps more serious, there is no theory of errors which
accounts for the presumed multiplicative combination of errors—especially
since the “errors” arc exemplified by judgments from different respondents.
Despite this lack of firm theoretical underpinnings, the theory-of-errors
approach appears to fit the accumulated data for point estimates more fully
than any other approach.
In addition, the measures of
of central
central tendency
tendency “
“recommended
recommended by
by”” the
the theory
theory
of errors have the desirable feature that the advantage of the group
group response
response
over. the
individual
t
------------1 response
can be demonstrated irrespective of the nature of
the physical process being estimated. So far as I know, this is the only
theoretical approach that has this property.
rI o make the demonstration useful in later sections, a somewhat more
sophisticated version of the theory will be dealt with than i:is necessary just to
display the “group effect.”
Consider a set of individual estimates Ri on an <event space
where the Rtj
are probabilities, i.e..
1. We assume there is a |physical process that
determines objective probabilities P={PI] for the eventl space, but P is un
known. Consider atgroup process G
-. which
. . . takes the geometric mean of the
individual estimates as
the
best
estimate
< the probability for each event.
— — ---- - of
However, (he geometric means will not be-• a probability, and must be normalized. This is accomplished by setting
Evaluation:
Iheory of Group Estimation
251
have
J
6;(G) = lOg^c( n
(3).
1
n
= T S log
+ log C.
I" 1
1 a king the expected score,
(fl)
1
m
(?)
f <C> d“"“ *•
I
5 (C) " " ,?!
Pj 1Og + log C’
(6)
log C appears outside the summation, because, ;
—
as a constant,
Pj log C =
‘og C. The expression Y™., P. log
is just the (
-• expected score S (R.) of
'"d'vidua ,, and the expression on the right of (6) excluding
is th
■
•*
_ log C '1/
ie average
m ua expected score, which we can abbreviate as S (R). Thus
I
(7)
nd"
(2)
J= 1 ' I= I
sxs
>
We can now ask how the expected probabilistic score of the group will
compare with the average expected score of the individual members of the
group. It is convenient to use the abbreviation C for the reciprocal of the
normalizing term
1
C
m
z
I M
\n
'
Using the logarithmic scoring system and setting the constants A
g
erent statistics as the aggregation rule and different scoring rules.1
I,
« 0, we
f
i
!•
I
Probabilistic Approach
I
Theoretically, joint realism
curves similar to the individual realism curve of
Eig. 1 can I e generated, g.ven enough data. In this case, the relative frequency
PP(C\P) of
correct estimates would be tabulated for the joint space of
ii d
) = 1 '' = I
-*-•s. - » A-
bv'usine il f^ CXCn’Plifi“ a ,arge variety of similar results that can be obtained
I
1
.h. W
— S (/?) + logC.
/
j
i
i
1 = 1
II
I
'B-"n
Mild lliP HUHn
MSnu9M etei,H),!8n”
I
•I
Norman C. Dalkey
espouses R for a group. Such a joint realism curve would be an empirical
iggregation procedure. Rb\C\R] would define the group probability judgment
as a function of R.
Although possible in theory (keeping in mind all the caveats that were raised
with respect to individual realism curvesj. in practice generating joint realism
urves for even a small group would be an enormous enterprise. It is conceivble that a small group of meteorologists, predicting the probability of rain for
i given lotality many thousands of limes, mighi ( over a wide enough region of
lie /i space lo furnish stable statistics. However, foi the vast majority of types of
[uestion where group estimation is desired, individu.il realism curves arc
lifficull to come by; group realism curves appeal to be out of the question for
he present.
One possible simplification at this point could be made if general rules
oncerning the interdependence of individual estimates on various types of
stimation tasks could be ascertained. In such a case, the joint realism curves
mid be calculated from individual realism curves. Although very iffy at this
point, it is conceivable that a much smaller body of data could enable the
testing of various hypotheses concerning dependence. In any case, by developng the mathematical relationships involved, it is possible to pursue some
.coretical comparisons of probabilistic aggregation with other types of aggre
gation.
In the following, the convention will be used that whenever the name of a set
of events occurs in a probability expression, it denotes the assertion of the joint
iccurrence of the members of the set. For example, if ,V is a set of events,
K—
then T( A') = P(A’1 -X?... A'n), where the period indicates “and.” In
addition, to reduce the number of subscripts, when a particular event out of a
et of events is referred to, the capital letter of the index of that event will be
used to refer to the occurrence of the event. Thus P(X) will be written P(J).
The degree of dependence among a set of events A' is measured by the
leparture of the joint probability of the set from the product of the separate
probabilities of the events. Letting Z)v denote the degree of dependence within
die set A', we have the definition
^(■V)
n /j(.v,)
(B)
Evaluation:
I heory of Group Estimation
)
253
A useful extension of the notion of dependence is that
of dependence with
respect to a particular event, say E - J.
DJX =
p<x\r>
n pwj)
i- 1
n
11
(9)
I
From the rule of the product, we have
nV- PU-VPU)’
PU^U-x.')
|
'
I
'1
■ i Mi
!
(10)
The probability we want to compute is P(J\Ry, that is, we want to know the
probability of an event
given that the group reports R. Again, from the rule
of the product, we have
j
I
I
PW '
Substituting R for X in (10) and multiplying the top and bottom of the
right-hand side by P(*)/II7_ ,/’(*,), and rearranging, gives
djr II /VI*,)
^C7I^) =
»-1
(ii)
Formula (11) presents the computation of the joint probability in terms of
the individual reports, the dependency terms, and the “a prior.” probability
C (/). lhe ^(71^) can be derived from individual realism curves. In case the
estimators are all fully realistic, then
is the probability of
the evenly based on whatever information is available without knowing R.2
The ratio
measures the extent to which the event J influences the
dependence among the estimates. If the estimates are independent “a priori,”
~ i • However, the fact that estimators do not interact (anonymity) or make
separate estimates, does not guarantee that their estimates are independent.
11I
1
j
J
j
I= 1
This notion is usually introduced by taking into account dependence among
subsets of X as well as the more global notion defined by (8). However, for
generating a probabilistic aggregation function, interactions among subsets can
be ignored, proving we maintain a set fixed throughout any given computation.
J
(JI"/*)- However, since W would be constant throughout, and ubiquitous in each probability
expression, it is omitted for notational simplicity.
i‘
Norman C. Dalkey
254
They could have read the same book the day before. The event related
dependence
is even more difficult to derive from readily available informa
tion concerning the group.
If there is reason to believe that a particular group is completely independent
in their estimates, and in addition each member is completely realistic, (11)
reduces to
Evaluation:
!
I
i
'Theory' of Group Estimation
the individual estimates for each alternative. (13) generalizes to
1= I
(12)
'The simplicity of (12) is rather misleading; it depends on several strong
assumptions. (11) on the other hand, is exact, but contains terms which are
difficult to evaluate.
An exact expression for P{J\R) can be obtained which does not involve DR
by noting that
/3UI«) =
Substituting for P (71R) on the right-hand side from (11) and the correspond
ing expression for P(J\R) (7 denotes the complement of 7 or ““not-y
J”') and
dividing top and bottom by Dj/
}DH we obtain
II /’(JIA,)
__________________ I = 1_____________ ___________________
11 A(£4|Z?,)
»»I______________
P(£tl/f)=
S D]t II P^R.)
(13)
where
I U(Ek)\n
-1
I
(14) is similar to a formula that can be derived using the theorem of Bayes
122]. Perhaps the major difference is that (14) makes the “working” set of
estimates the P(£^|/?l) which can be obtained directly from realism curves,
whereas the corresponding formula derived from the theorem of Bayes involves
as working estimates P^R^Ej) which are not directly obtainable from realism
curves. Of course, in the strict sense, the two formulae have to be equivalent,
and the P^R^Ej) are contained implicitly in the dependency terms. Without
some technique for estimating the dependency terms separately from the
estimates themselves, not much is gained by computing the group estimate with
(H).
Historically, the “a priori” probabilities U(J) have posed a number of
conceptual and data problems to the extent that several analysts, e.g., R. A.
Fisher [23], have preferred to eliminate them entirely and work only with the
likelihood ratios—in the case of (14), the ratios
4
II PU\R,)+l) H (1-A(J|A,))
n r^r,)
I=1
where
i= 1
n r^r.)
b.4(W'
1= 1
If the estimators are all fully realistic and fully independent, and the a priori
probability = J, (13) reduces to
II «,
p(yi^)= —
l = 1
(14)
II /<+ II (i-^)
i=i
!
J-l
II R,
/)(7I«) = A'C/)"-' '
1=1
4
n
n
^C/l^) =
255
i=i
To complete this set of estimation formulae, if there are several alteinatives
in A’, and it is desired m eum|nili- the group c’MinuiU* for enrh ftllvriifliive from
This approach appears to be less defensible in the present case, where the a
priori probabilities enter in a strong fashion, namely with the n — 1 power.
1'or a rather restricted set of situations, a priori probabilities are fairly well
defined, and data exist for specifying them. A good example is the case of
weather forecasting, where climatological data form a good base for a priori
probabilities. Similar data exist for trend forecasting, where simple extrapola
tion models are a reasonable source for a priori probabilities. However, in
many situations where expert judgment is desired, whatever prior information
exists is in a miscellaneous form unsuited for computing probabilities. In fact, it
is in part for precisely this reason that experts are needed to “integrate” the
ioIm rllaiiHHL*} Infornuulprh
I
I.1'
fl
II
I
■ s4
J
I
-
Norman C. Dalkev
Some additional Hght can be thrown on the role of a priori probabilities as
well as the dependency terms by looking at the expected probabilistic score. In
the case of the theory-of-crrors approach, it was possible to derive the result
dial, independent of the objective probability distribution l\ the expected
probabilistic score of the group estimate is higher than the average expected
score of individual members of the group. This result is not generally true for
probabilistic aggregation.
Since probabilistic aggregation depends upon knowing the a priori prob
abilities, a useful way to proceed is to define a net score obtained by subtracting
the score that would be obtained by simply announcing the a priori probability.
Letting 5*(G) denote the expected net score of the group and .S'♦(/?,) the
expected net score of individual /, and S(It') the score that would be obtained if
(
were the report, S*(G) = 5(G)-.S’f/<) and 5*(/?() = 5(/?,)- 5(£). The
net score measures the extent to which the group estimate is belter (or worse)
han the a priori estimate. I his appears to be a reasonable formulation, since
presumably the group has added nothing if its score is no better (or is worse)
than what could be obtained without it.
Many formulations of probabilistic scores include a similar consideration
when they are normalized. 1 his is equivalent to subtracting a score for the
case of equally distributed probabilities over the alternatives. Thus the score for
an individual is normalized by setting 5*(/?,) = 5(7?,) - 5(Q) where Q = 1 /m
and m is the number of alternatives. In effect
effect this is
is assuming
assuming that the
the a priori
probabilities are equal.
Computing the expected group net score from (11) we have
n
m
,n
n /'(/i/?,)
I
(16)
7=1
rn
m
n
)= 1
1=1
-(« - 1) y /(In E(J) +
7=1
rn
+ S /(ln/)/-ln/Js-S(£)
(17)
j-1
Evaluation:
I heory of Group Estimation
257
positive.
On the
other nana,
hand, it
if the average net score of the individual
°r r
—
tuc uuicr
members ts negative,
negative, then
then the
the group
group will
be « times as bad, still assuming th^
will be
(1B1 ”l nCy,‘Crm,S SmalL S,nce thc logarithm of DR will be negative if DR< 1
) shtms.that the most favorable situation is not independence where
less liki thaLth ’
nega'iVe dePendence- ie-case where it is
1
• “',7';"’
proh.MU.y f, i, high. Thi. l.vor.U.
ould be expected if the mdividuals are skilled estimators, but cannot be
guaranteed on logical grounds alone.
One of the more significant features of the probabilistic approach is that
nder favorable conditions the group response can be more accurate than any
group.
comnl'T I
Sr°UP' For ex;
eXample’ if the exPerts are folly realistic, agree
completely on a given estimate, are independent, and finally, if it is assumed
that the a pnon
priori probabilities ;are equal (the classic case of complete prior
ignorance), then formula (14) becomes
H
H
Pn
(19)
P^^-P^
where
pP is the common estimate, and
ir
J n is the number of members of the eroun
ToL'?’ T"
rapidly aPProaches 1 as n increa-.s. For example
A = 2/3 andU -s 5 then /^|/f) = 32/33. If the theory-of-errors approach were
being cm ployed, the group estimate would be 2/3 for any size group
In this respect, tt seems fair to label the probabilistic approach “risky” as
compared with the theory-of-<•errors approach. Under favorable conditions the
former can produce
]
group estimates that are much more accurate than the
individual members of the i
— ...c group; under less favorable conditions, it cai^
produce answers which are much
i worse than any member of the group.
Axiomatic Approach
■
•
— — nS(A ) + nS ( /?) 4- V
In /J/ - In
(18)
;= i
whence 5*(G’) = n5*(7?) + Expectation of dependenev terms.
If the average net score of the individual members is positive (i.e., the
average member of the group docs better than the a priori estimate), then the
group score will be n times as yood. providing the drpendrm v terms arc small
J
A somewhat different way to develop a theory of gtbup estimation is to
postulate a set of desired characteristics for an aggregation method and
determine the process or family of processes delimited by the postulates This
approach has not been exploited up to now in Delphi research. The major
applied D I h "
large "umber of nonformal procedures associated with an
apphed Delphi exercise-formulation of a questionnaire, selection of a panel of
experts, interpretation of results,” and the like. HowevW, if the agin tallan
process is (|Pf|nWl fortpaljy
,he ,^o prceedjpg sub^tlops, wj'^"
1
1
J Hl
Normal) C’. Dalkey
questionnaire design is interpreted as defining the event Jspace E, and panel
selection is reduced tto defining
' "
the response space R, then thei axiomatic
approach becomes feasible.
C’onsidering the group estimation process as a ffunction G= G^EJ.R),
various properties of this function appear “reasonable”’ at first glance. Some of
the more evident of these are:
(..•1) Unanimity. If the group is in complete agreement, then the group estimate
is equal to the common individual estimate; i.e., if R = R. for all i and k
then G’(/?)=/?.
9
J
(/?) Monotony. If R and R' arc such that Rtj > R'tj for all z, then G(/?) > G (/?').
If R and G are defined as real numbers then they fulfill the usual ordering
axioms, and condition B implies condition .4.
(G) Nonconventionality. G is not independent of the individual estimates; i.e.,
G(R)^ G(S) for every' possible R and .S’.
(/-^) Responsiveness. G is responsive to each of the individual estimates; i.e.,
G^/?)^ G’'(7’), where T is a proper subvector of R.
Preservation of Probability Rules. If G is an aggregation function which maps a
set of individual probability estimates onto a probability, then G preserves
the rules of probability. For example, if Ttj = R^ for all i and j (as would
be the case if Rtj is the estimated probability of Et and Stj is the estimated
relative probability of an event k'j given that G occurs) then
g(7;) = g’(/?7)g(.s;).
'This set of conditions will be displayed more fully below.
.All of these conditions have a fairly strong intuitive appeal. However,
intuition appears to be a poor guide here. I he first four postulates are fulfilled
by any of the usual averaging techniques. But .4, which is perhaps the most
apparently reasonable of them all, is not fulfilled by the probabilistic aggrega
tion techniques discussed in the previous subsection. It was pointed out there
that one of the more intriguing possibilities with probabilistic aggregation is
that the group estimate may be higher (or lower, depending on the interaction
terms) than any individual estimate.
Il can be shown that there is no function that fulfills all five of the postulates;
in fact, there is no function that fulfills I) and E. 'The proof of this impossibility
theorem is given elsewhere [24]; it will only be sketched here.
Three basic properties of probabilities are («) normalization, if p is a
probability, 0< p < 1; (6) complementation. P( J)+ /,(.7)= 1; and (r) multi
plicative conjunction, i.e., P(JCJ,)= p(JjP(JJJj. The last is sometimes
taken as a postulate, sometimes is derived from other assumptions.
If the individual members of a group are consistent, their probability
judgments will fulfill these three conditions. It would appear reasonable to
require that a group estimate also fulfill the conditions, consistently with the
Evaluation:
Theory of Group Estimation
259
individual judgments. In addition, condition
D, above, appears reasonable.
This leads to the four postulates:
PI.
P2.
P3.
P4.
0<
1.
G(1 - /?)=* 1 G(R-S^G\R)G(S).
GtR^G'tS), where S’ is a subvector of R (condition D).
Here, /? S is the inner product of the two vectors R and S', i.e.,
Pl-1’3 have the consequence that G is both multiplicative and additive The
tions of a single variable, there is only one which is both multiplicative and
additive, namely the identity function /(x) = x. There is no corresponding
identity function for functions of several variables except the degenerative
function, G'(7?)— GZ(7?J)=
which violates P4.
This result may seem a little upsetting at first glance. It states that probabil
ity estimates arrived at by aggregating a set of individual probability estimates
cannot be manipulated as if they were direct estimates of a probability.
However, there are many ways to react to an impossibility theorem. One is
pamc I here is the story that the logician Frege died of a heart attack shortly
after he was notified by Bertrand Russell of the antinomy of the class of all
classes that do not contain themselves. There was some such reaction after the
more recent d.scovery of an impossibility theorem in the area of grouo
preferences by Kenneth Arrow [25], However, a quite different, and more
pragmatic reaction is represented by the final disposition of the case of 0. In the
1 th century, there was long controversy on the issue whether 0 could be
treated as a number. Strictly speaking there is an impossibility theorem to the
effect that 0 cannot be a number. As everyone knows, division by 0 can lead to
contradictions. 1 he resolution was a calm admonition, “Treat 0 as a number
but don’t divide by it.”
’
In this spirit, formulation of group probability estimates has many desirable
properties It would be a pity to forbid them because of a mere impossibility
theorem. Rather, the reasonable attitude would appear to be to use group
probabthty estimates, but at the same time not to perform manipulations with
the group aggregation function which can lead to inconsistencies
Coda
I he preceding has taken a rather narrow look at some of the basic aspects of
gioup esimiation. Many signlfteant fotHurw, such 4*
of .orm i. frrdbackj the role of additional Infoimadmi “fetMii” lu
g;uup( the
I
■
■
I
1
i
4
260
Norman (.!. I )alkcy
differences between open-ended and prescribed questions, and the like, have
not been considered. In addition, the role of a Delphi exercise within a broader
decisionmaking process has not been assessed. What has been attempted, albeit
not quite with the full neatness of a well-rounded formal theory, is the analysis
of some of the basic building blocks of group estimation.
1 o summarize briefly: I he outlines of a theory of estimation have been
sketi hed, based on an objective definition of estimation skill—the realism curve
ot track record of an expert. Several approaches to methods of aggregation of
individual reports into a group report have been discussed. At the moment,
insufficient empirical data exist to answer several crucial questions concerning
both individual and group estimation. Eor individual estimation, the question is
open whether the realism curve is well defined and sufficiently stable so that it
can be used to generate probabilities. For groups, the degree of dependency of
expert estimates, and the efficacy of various techniques such as anonymity and
random selection of experts in reducing dependency have not been studied.
By and large it appears that two broad attitudes can be taken toward the
aggregation process. One attitude, which can be labeled conservative, assumes
that expert judgment is relatively erratic and plagued with random error.
Under this assumption, the theory-of-errors approach looks most appealing. At
least, it offers the comfort of the theorem that the error of the group will be less
than the average error of the individuals. 1 he other attitude is that experts can
be calibrated and, via training and computational assists, can attain a reason
able degree of realism. In this case it would be worthwhile to look for ways to
obtain a priori probabilities and estimate the degree of dependency so that the '
more powerful probabilistic aggregation techniques can be used.
At the moment 1 am inclined to take the conservative attitude because of the
gaping holes in our knowledge of the estimation process. On the other hand,
the desirability of filling these gaps with extensive empirical investigations
seems evident.
Evaluation:
Theory of Group Estimation
1
I
9 G A’.,S‘ Stael von Holstein’ “Assessment and Evaluation of Subjective Probability Distributtons, ’ Economic Research Institute, Stockholm School of Economics, 1970.
10. Girshick, et al., op. cit.
! 1. W. Edwards, "The Theory of Decision Making,” Psychol. Bulletin 5 (1954), pp. 380-417.
12. F. Knight, Risk, Uncertainty and Profit, Houghton Mifflin, Boston, 1921.
13. Hans Reichenbach, The Theory of Probability, University of California Press, Berkelev 1949
Section 68.
’
^alkey, "Experimental Study of Group Opinion," Futures I (September 1969), pp. 408-26.
15. N. Dalkey, B. Brown, and S. Cochran, “The Use of Self-Ratings to Improve Group Estimates ”
Iechnological Forecasting 1 (1970) pp. 283-92.
16. J. P. Guilford, Psychometric Methods, McGraw-Hill, New York, 1936, pp. 426ff.
17. N Moiseev, "The Present State of Futures Research in the Soviet Union,” in Trends in
Mathematical Modeling, Nigel Hawkes (cd.), Springer-Verlag, Berlin, 1973.
18. N. Dalkey, "Experimental Study of Group Opinion,” op. at.
19 C A“chJSOn^nd J gA^ C- Brown»
Lognormal Distribution, University of Cambridge Press,
1
I
I
20. N. Dalkey, “Experimental Study of Group Opinion,” op. cit.
21. T. Brown, “An Experiment in Probabilistic Forecasting," The Rand Corporation, R-944ARPA, March 1973.
22. Peter A. Morris, Bayesian Expert Resolution, doctoral dissertation, Stanford University, Stanford
California, 1971.
23. R. A. Fisher, “On the Mathematical Foundations of Theoretical Statistics,” Philos. Trans. Roy
Soc., London, Series A, Vol. 222, 1922.
24. N. Dalkey, "An Impossibility Theorem for Group Probability Funcn s.” The Rand Corpora
tion, P-4862, June 1972.
H
25. K. J. Arrow, Social Choice and Individual Values, John Wiley and Sons, New York, 1951.
$
1
References
1. John McCarthy, “Measures of the Value of Information,” Proc. .Mat. Acad, of Sci. 42 (September
15, 1956), pp. 654-55.
2. Thomas Brown, “Probabilistic Forecasts and Reproducing Scoring Systems,”
The Rand
Corporation, RM-6299-ARPA, July 1970.
3. L. J. Savage, “Elicitation of Personal Probabilities and Expectations,” J. Amer. Stat. Assoc. 66
(December 1971), pp. 783-801.
4. E. E. Shannon and W. Weaver. The Mathematical Theory of Communication, University of Illinois
Press, Urbana, 1949.
5. Savage, op. at.
6. S. E. Asch, “Effects of Group Pressure upon the Modification and Distortion of Judgments,” in
E. E. Maccoby, T. M. Newcomb, and E. L. Hartley (eds.), Readings in Social Psychology, Henrv
Holt, New York, 1958, pp. 174-83.
7
7. C. H. Coombs, “A Review of the Mathematical Psychology
oz of
- Risk
------, ”’ presented at the
Conference on f
'
.. .............................................
Subjective
Optimality,
University of Michigan, Ann Arbor, August 1972.
8. M. Girshick, A. Kapl.
. lan’ and A. Skogstad, “ I he Prediction of Social and Technological
Events,” Public Opinion Quarterly, Spring 1950, pp. 93 110.
261
1
ri
1
tj
I
P
si
l '*
. C. Experiments in Delphi Methodology’
Evaluation:
Introduction
’This study was supported by the Urban Systems Engineering Center, Northwestern University,
NSF Grant GU-3851.
The Delphi Method: Techniques and Applications. Harold A. Linstone and Murray Turoff (eds.)
ISBN 0-201-04294-0; 0-201-04293-2
Copyright© 1975 by Addison-Wesley Publishing Company, Inc., Advanced Book Program.
All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or
transmitted, in any form or by any means, electronic, mechanical photocopying, recording, or
otherwise, without the prior permission of the publisher.
2t?2
263
A number of Delphi studies have used high/lovv self-ratings of participant
confidence. Evidence of the value of such confidence ratings in improving the
results of the Delphi is somewhat limited, except under certain conditions of
group composition [4]. In this study, the use of high/low self-confidence ratings
is again evaluated, and the influence of a number of other personal descriptive
variables is tested.
Other design features include the application of short turn-around times
using a computerized system for supporting inter-round analysis of the Delphi
data. Although Turoff (Chapter V, C) has used a more complex, interactive
computer system for this purpose, a simpler program is used here merely to
accelerate accounting tasks.
J
M. SCHEIBE, M. SKUTSCH, and J. SCHOEER
The emphasis in the Delphi literature to date has been on results rather than
on methodology and evaluation of design features. The other articles in this
chapter do address the latter aspects. Still, quite a number of issues remain
unsolved, particularly those concerned with the details of the internal structure
of the Delphi. For example, the way in which subjective evaluation is measured
may affect the final output of the Delphi. .\ number of variables enter here.
Ostrom and Upshaw |1] have noted that the range of the scale provided has a
marked effect on judgment. Persons playing the role of judges who estimated
themselves as “relatively harsh" assigned average “sentences” of four years to
“criminals” when presented with a one-to-five-year scale, and twenty-one years
when presented with a 1 to 25-year scale. 'The difficulties involved with the
selection of a suitable scale range can be solved by the employment of an
abstract scale rather than one representing, for example, hard dollars or years.
An abstract scale allows relative measures to be made. Abstract scales are
particularly suited to the measurement of values, as for example in the
development of goal weights to represent relative priorities for goal attainment.
A number of psychological scaling techniques which result in abstract scales
arc available. 'This study reports on the comparison of several scaling techniqiies which were tested in the context ol .in cxpci imcntal Goals Delphi.
Another issue is that »»f the <•((<•< Is of feedback input, whic h form the sole
means of internal group (oinniunic.ilions in the Delphi pioeess. Il is important
io the design of Goals Delphis to determine the nature and strength of the
feedback influence. In the experiment reported below, the impact of feedback
was identified by providing participants with modified feedback data. The
resulting shifts of opinion were then used as measures of feedback effectiveness.
Methods for the measurement of consensus are also considered and a
redefinition of the endpoint of a Delphi is offered. Instead of consensus, the
stabibii of group opinion is measured. This allows much more information to be
derived from the Delphi, and in particular, preserves opinion distributions that
achieve a multimodal consensus.
Delphi Methodology
1
Description of Procedure
♦
The objective of this experimental Delphi study was the development and
weighting of a hierarchy of goals and objectives for use in evaluating a number
of hypothetical transportation facility alternatives. In the terminology suggested
by Wachs and Schofer [2], goals are long-term horizonal aims derived directly
from unwritten community values; objectives are specific, directional, measurable
ends which relate directly to goals. Previous experiments by Rutherford, et al.
13] had indicated that Goals Delphis should be initiated by the development of
objectives rather than goals, for the tendency toward upward drift in generality
can be minimized if the Delphi participants are first asked to work at the more
specific level. The development of goals, once the objectives have been defined,
can be a< <omplished with much greater ease than can the reverse process.
1 he flow chart of Fig. 1 illustrates graphically the process by which the Goals
Delphi .ind the design experiments were carried out. First, the initial list of
objectives was generated. The process administrators presented the hypothetical
transportation situation to the participants by means of a verbal description
(Appendix I of this article), and a map (Appendix II of this article). The
participants were then given a set of five blank 3x5 cards and asked to list no
more than five objectives which they felt were applicable in the hypothetical
situation. In all, seventy-seven objectives were submitted.
1 o derive goals front the list of objectives, and to eliminate the overlaps
between them, a grouping procedure was followed. The process administrators
first rejected those objectives that were exact duplicates of others, and assigned
the remaining ones to sets. Each set represented objectives tending toward a
common goal. Nine major goals were established, and these were “named”
appropriately. Statements which were not strictly objectives were left out of the
grouping process. The complete list of goals and objectives is given in Table 1.
These goals were then returned to the participants for their evaluation. They
were given the opportunity to add new objectives, and several were received
and incorporated into the goal set.
1
• J
*!
It
£
•j
M. Scheibe. XL Skutsch. and J. Schofcr
Evaluation:
Delphi Methodology
265
Table 1:
J
Goals and Objectives Suggested for the
Hypothetical Transportation Scenario
I
r H...
GOAL 1:
Minimize the adverse environmental impacts ot the system.
Objectives:
T
Minimize air, noise, and water pollution.
Minimize negative illumination
and vibration effects.
1
KA» Gi'AlS
RAIL i.i’Alkvl’S? u.Sr
GOAL II:
I
I
Preserve the recreational and social
environment.
Obj ec t ives:
I
Provide compensation for parkland removed.
Minimize the amount of parkland taken.
Minimize the demolition of historic buildings.
Minimize the amount of non-urban land required for the new facility,
Minimize the adverse social consequences of the transportation
4
Wk HE iiW.t.s:*
------- 1------ -
facility location by providing adequate
KATE GOA!.-KUlTiD rHRH.
I l
WRITE ■.‘'j-'r!.:•
compensation to families
and businesses (and employees).
1
Minimize the amount of urban land
required for the new facility,
Minimize the number of residences relocated.
>
SI •
Minimize the disruption of existing
neighborhoods, people and businesses.
KA IE t.i'ALS
RUIKI> > 01 K
goal
II
1
I
III:
Minimize operating and construction
T
I
costs.
Objective:
Minimize operating and construction
I
costs.
11
GOAL IV:
'
I PHI
; 1 "SNA Im
Maximize mobility and accessibility for the urban
Obj ectives:
I - 11 • >N’NA1 Rt >
{
area residents.
Minimize travel time.
Minimize monetary travel cost.
Reduce congestion
rr>rh>:.NAii<t>
Fig. I.
I he Goals Delphi expci^menta! design.
Locate the facility to increase mobility and accessibility
in the area.
Provide sufficient imobility for all members of society.
I
.I
-
M. Schcibc. M. Skutsch, and J. Schofer
Evaluation:
Delphi Methodology
!
267
Table 1 (cont.)
i
GOAL V:
Design the facility to operate as safely as possible.
Obj ec t ive:
Design the facility to operate as
fely as possible.
GOAL VI :
Coordi natv the transportation system with la nd u s <• d ev e 1 opmen t .
Obj ec t ives:
Maximize the flexibility of the facility so as not to hinder possible
further urban development in the area.
Encourage the desired land use development pattern as stated in the
land use plan.
GOAL VII:
Design the faciliiv
that
tb.e'ic appeal is in harmony with
the enviFonnienl .
Objectives:
Use transport facilities to highlight the character of the city to
increase awareness, interest, and participation by the facility users.
Design the facility to be visually pleasing to the surrounding community.
GOAL VIII:
Maximize positive economic benefits for the entire area.
0! ..ctive:
Maximize positive economic benefits for the entire area.
GOAL IX:
Minimize the adverse impact s occurring during construction.
Ob ject ive:
Minimize the adverse impacts occurring during construction.
■
Following the development of the goals hierarchy, a decision was made
(largely because of time constraints) to concentrate attention at the goal level.
The objectives, therefore, were not included in the weighting procedure.
Objectives, however, were at all times appended to the goals related to each, so
that participants would always be aware of the specific meaning of each goal.
Participants were first asked to do a simple ranking of goals. As discussed
elsewhere in this paper, one of the purposes of the Delphi was to compare
different scaling methods. Participants were therefore asked to follow the
ranking with a rating analysis. Nine-point Likert scales were used (0
= unimportant, 9 = very important). This type of scale was felt to be easily
understood by the participants. In addition, when the ends are anchored
adjectively, as in semantic differential scales, this scale is commonly found to
have intenal properties. Using the computer program developed for this study,
the results of this first round were analyzed. The program was processed using a
remote terminal; goal weights served as inputs, and histograms and various
distributional statistics were produced as outputs. Frequency distributions of
scores for each goal prepared by the computer were presented to the participants, along with the mean for each distribution.
Participants w'ere asked to once again rate each goal on a nine-point scale,
using the information from the previous round as feedback. In addition, those
participants whose score on any goal was significantly distant from the group
mean value for that goal were asked to write a few words explaining the
reasons behind their positions. These statements were edited and returned in
the next round. This procedure continued for a total of four rounds. The results
are given in Fig. 2 and 3, which show- the histograms produced in the first and
the final rounds.
After the fourth weighting round, the participants were asked to perform a
pair-comparison rating of all the goals. This was done to compare this scaling
method with the nine-point rating scale and the ranking methods.
The initial development of the goals was accomplished during one two-hour
class period. I he rank ordering of the goals and the first three weighting rounds
were conducted in a second two-hour period two days after the first. The fourth
round look place an additional five days later, while the pair comparisons were
made a week after the first weighting round.
!•
■
I
■
I
An Experiment on the Effect of Feedback
There has been quite a bit written about the uses of feedback in the Delphi
technique. Most of this, such as the work of Dalkey, Brow i, and Cochran [4] at
Rand, has concentrated on the effects of different types of feedback, such as
written statements and various statistical measures. The effects 0f this fccdba&k,
particularly in the almanac-type Dclphls, have been measured by comparing
r
M, Schcibc, M. Skutsch, and J. Schofer
1 *2 '34' s' 6 '? ’ 8'9
XI
1 2
3 4 ' s' 6'7 ' 8 9
GOAL 2
MEAN =6.57
STD. DEV. = 2.06
GOAL 1
MEAN =7.00
STD. DEV. = 1.26
I 2
3 4 5 '6'7
Evaluation:
Delphi Methodology
269
8*9
GOAL 3
MEAN =6.14
STD. DEV.= 1.77
'1
2'3 ' 4
' 7 18 ' 9
GOAL 1
MEAN = 7.33
STD. DEV. = 0.99
'1 ' 2' 3 '4 ' 5 '6 ‘ 7’8 ' 9
' 1 1 21 3 '4 * 5 '6 ' 7'8 ’ 9'
GOAL 2
MEAN = 7.47
STD. DEV. = 1.43
GOAL 3
MEAN =5.86
STD. DEV. = 1.49
I
'j
tVt
4
89
5 6 '7
GOAL 4
MEAN = 7. 14
STD. DEV. = 2.22
1 2
3 4
5 6 7
8 9
GOAL 5
MEAN =6.81
STD. DEV. = 1.99
' 1 2 ' 3'4 5 '6 '7 ’ 8'9
GOAL 6
MEAN - 7.86
STD. DEV. = 1.01
TT 5
6 7 8
9
GOAL 4
MEAN = 7.38
STD. DEV. = 1.76
1‘ 2'3 X 5 *6 ' 7 ‘ 8 ' 9
1 1’ 2*3 ' 4' 5 6 ' 7'i ' 9
GOAL 5
MEAN =7.10
STD. DEV. = 1.54
GOAL 6
MEAN =7.90
STD. DEV. = 0.92
I
1
3 '4 ' 516 '7 *8^9 '
'1’2 ' 3 ’4 ‘ 5* 6 ’7 '8'9
GOAL 7
MEAN = 5.24
STD. DEV. = 1.70
Fig. 2.
GOAL 8
MEAN = 6.00
STD. DEV. = 1.98
1 *2 1 3 '4 ‘5 ‘6'7
8 *9 1
GOAL 9
MEAN = 3.48
STD. DEV. = 1.96
Results of round one rating analysis.
1 2 3
V4 5 6
9
GOAL 7
MEAN =5.86
STD. DEV. =0.94
Fig. 3.
1 2 3 ' 4' 5 '6 ' 7 *8 ’ 9
2'3'4'5 6 '7'8'9'
GOAL 8
MEAN =6.05
STD. DEV. = 1.73
GOAL 9
MEAN = 3.48
STD. DEV. = 1.40
Results of round four rating analysis.
1.
M. Scheibe, M. Skutsch, and J. Schofer
Evaluation:
Delphi Methodology
group given different feedback or no feedback at all. In Policy and Value
Delphis the effect of feedback is evaluated by measuring the degree of consen
5
CT.
sus which is reached and the speed with which it is reached.
I here seems to be very little in the literature, however, which examines the
round-by-round effect of feedback or investigates the manner in which the
o
4
1
3
2
I
feedback affects the distribution of scores in a particular round. In this study, it
w
w
o
V)
—
1'
was decided to investigate these aspects of feedback, since the kind and amount
of feedback used in the Delphi may be an important variable in its results. A
0
1
2
greater understanding of the impacts of feedback might lead to better Delphi
design. The method employed was to provide participants with false feedback
data, and then to observe the effect of this on the distribution of priority-weight
scores.
3
4
5 6 7
Round 1 (true)
8
:]
I
3
2
0
■
1
2
3
4 5 6 7
8
Round 1 (altered)
(returned in Round 2)
uted. During the experiment on feedback, one goal was chosen and the
distribution was altered by the administrators so as to change markedly the
position of the mean. Since this was done after the first weighting round, no
written feedback accompanied the altered distribution. The goal chosen for this
lest was Number 3 (“Minimize the operating and construction costs”). This
goal was chosen because it appeared to have a good consensus after the first
iteration. In addition, it was judged to be substantively important. It was felt
9
t
Bli
: -IS
:bI
Two types of feedback were used in this study. 'The first was a graphical
i epi cscniui ion <>( (he disli ibulion of stoics logcihci with a listing of the mean of
llu. disti ibulion. In addition, in latci rounds edited, anonymous commcnis by
the participants concerning the impoi tan< e of the vatious goals were distiib-
I
271
the accuracy of the opinions of a group given a certain feedback with that of a
I
2
; sig
')
V)
3
4
C
o
~l
3.
~1 £ 22
11
0
]
2
3
4
5
6
7
8
tJ'1 0 1
Round 2
2
WJ □ 1
3
4
5
6
7
8
k
I
9
Round 3
that most participants would be very surprised by the altered distribution.
*
The second-round distribution showed that the feedback had had an effect,
since a number of persons shifted their positions away from the true mean. By
the third round, the distribution was once again similar to what it had been in
the first round, although the distribution was shifted slightly to the lower end of
the scale and remained that way permanently, showing residual effects of the
gerrymandering. Figure 4 shows the actual distributions and the altered feed
back used.
In attempting to explain the reasons behind these changes, the following
hypothesis is offered. Upon seeing the first round of feedback information, the
respondents had three options: they could ignore the feedback and keep their,
votes constant; they could rebel against the feedback and move their votes to
the right, in the interest of moving the group mean closer to their true desire; or
they could acknowledge the feedback and move their votes nearer the false
mean. If they had followed either of the first two options, it would indicate that
the feedback was not effective in changing individual attitudes. That the third
option was in fact taken, however, indicates that the feedback did have an
effect on the participants.
1
7
•I
6
cp
I
5
I 4J
V)
H
w
o
Ju
%
3
2
1
—
0
Dg. 4.
E□
4 5 6
7 8 9
Round 4
Distributions generated for Goal 3.
I
I
M. Sdicibc, M. Skutsch. and J. Schofcr
Evaluation:
The third round, as a result of the feedback of the second round, also shows
'he effect of feedback. I he second-round distribution showed that participants
Acre attempting to increase the priority for (Joal 3. although with respect to
heir true initial opinions, they were actually decreasing this priority. It seems
likely that many respondents, upon seeing this, felt that the group was moving
■loser to their original position and they decided to return to their first-round
ole, since it no longer appeared that this position would be far distant from the
mean value of the group.
This experiment suggests that the respondents arc. in fact, sensitive to the
.■edback of distributions of scores from the group as a whole. These results seem
io indicate that most repondents are both interested in the opinions of the other
Members of the group and desirous of moving closer to the perceived consensus.
Comparison of Scaling Techniques
Vilh the exception of Dalkey and Rourke [5|. there is little discussion in the
literature of the different methods of scaling which could be used in a Delphi.
< he two most common methods which arc used arc simple ranking and a
Jkert-type rating scale. Even when these methods are used, there have seldom
been attempts to ensure that the scales developed are. in fact, interval scales.
The necessity of having an interval scale is seldom emphasized in Delphis.
' here is the suspicion that on some occasions the scales derived are ordinal
icales. An ordinal scale merely shows the rank order of terms on the scale, and
.<> statement can be made concerning the distance between quantities.
Ercsumably. the primary reasons for using a Delphi, especially when com
paring policies or measuring values, include the determination of not only
hich policies are considered most important, but also the degree to which
• ich policy is preferred over the other possibilities. In order to assure that this
an be determined, an interval scale must be obtained.
In this study, three methods which usually yield interval scales were tested.
1‘hcse methods were simple ranking, a rating-scale method, and pair compari-ons. The purpose in trying three scales was to determine if all three methods
ielded approximately equivalent interval scales. If this is found to be the case,
hen in future designs any one of these scales could be used. In this situation, it
•could probably be wisest to choose that scaling method which was considered
asiest to perform by the participants in the Delphi. In this study it was found
|hat the rating-scale method was considered by the participants as the most
mnfortable to perform. 1 he limitation of the pair-comparison method is that it
lime consuming, l or example, to apply this method to a set of ten objectives,
•ach participant must make forty-five judgments. The ranking method is fairlv
asy for a small number of goals, but becomes increasingly difficult as the
umber of goals increases, for il cssenlialk requires the participant to order the
iitire list of items m his mind. In addition, mans parin ipants fell uncomfort
I
I
Delphi Methodology
273
able performing this method because they were prevented from giving two goals
an equal ranking (i.e., forced ranking). While this dilemma might possibly have
encouraged more thought concerning underlying priorities, ii’.vas felt that the
frustration caused had a negative effect on the end result. The rating-scale
method was found to be quick, easy to comprehend, and psychologically
comforting. 1 he participant’s task is easy, since he must rate only one item at a
time. 1 lie problem that remained was to determine whether such a scaling
procedure would yield a scale with interval properties.
In this experiment il was< found
methods yielded
found that
that each
each of
of the
the three
three methods
yielded
somewhat different scales. Using the Law of Comparative Judgment [6], scale
values for each goal for each round were derived. These values were then
translated onto a scale from one to nine. Graphical representations of these
scales are given
(
in Fig. 5. Because of the presence of feedback, the four rating
rounds aree not independent. Each one depends on those previous to it. The
scales derived in each successive round should not be identical, for if the scale
remains constant from round to round, the justification for using an iterative
approach is lost. In addition, because of the order in which the scales were
developed, the ranking scale can only be compared with the first-round rating
scale and the pair-comparison scale can only be compared with the fourth
round rating scale. Because of four rounds of feedback between them, the
ranking scale and pair-comparison scale should be compared only cauti ously,
and should not be expected to be identical
The interscale comparison shown in Fig. 5 is not especially encouraging. The
pait-comparison method is known to produce interval scales, and the similarity
in results of this approach and the round-four rating results is not strong. The
scales produced by ranking and round-one rating are, however, not too
different fiom each other. It is possible to interpret the progression of rating
scales from round one to round four as a movement in the direction of the
pair-comparison scale. This experiment did not pursue further weighting
rounds, but, as discussed below, major changes in weights beyond round four do
not seem likely. In addition, later pair-comparison responses might differ from
that shown in Fig. 5. Given the complexity of the pair-comparison method for
participants, however, it may not be unreasonable to accept cautiously the
results of simple rating methods as fair approximations to an interval scale.
■
:.1
i
-
Id
I
j
The Effect of Personal Variables on Participant Behavior
I
Dal key, Brown, and others have considered and used the confidence of par
ticipants in their responses to reach more accurate estimates of quantitative
phenomena in Delphi exercises. Working with almanac-type data not available
to the pat ticipants, they found that by selecting for inclusion in the feedback
only those iespouses considered “highly confident’’ by their proponents, a
I
i
M. Schcibe, M. Skuisch, and J. Schofer
Evaluation:
9
H
7
COALS
observation, therefore, several •
simple hypotheses were tested. It was felt that
confidence might involve more
- than simply confidence in individual answers,
and therefore a selection of variables r*
representing various aspects of personal
confidence were sought, as well as high/lo-..
■>w confidence in each response,
Flank ing
1
2
3
7
7
Ra ti ng
Round 1
4-3
4
2
43
4
9
Racing
Round 3
5
6
jy +6
73S
1 25 6
SCALE VALUES
4
GOALS
7 38
wu
2
3
4
4
4
articlC Sh°WS the questionnaire7^ued to all participants
elphi. The variables measured were as follows:
SCALE VALUES
5
GOALS
4
'9
2 6
4 4 4- q.
6
Comparisons of goal weights on various ca
7
8
9
Pre-Delphi Survey
Quest ion Nuinbvi
legorical judgment scales.
1
I
I
Inde pendent Variable
i
I
1
A.
Perceived academic standing relative
to other participants
2
B.
At-oneness with other participants
3
C.
Familiarity with other participants
4
D.
Confidence of own capacity in the
subject area of the Delphi
I
5 and 6
E.
Experience in the field
s
7
F.
Expectation of the Delphi process
(confidence in Delphi)
Each of these confidence variables
was then correlated against dependent
variables describing the amount of r^Vemrenl actuaHy made by each participant toward the center of the distribution. It
It was, of course, not possible in this
value-judgment Delphi to test
accuracy as well, as was done in Dalkey’s
experiments.
SCALE VALUES
'll
J
GOALS
1 1 I1 “J 4
8
1
!
SCALE VALUES
738
Fig. 5.
9
1
Rtting
Round 4
Pai r-Comparison
ore t e
goals
4
9
bJfr0^
SCALE VALUES
H
5
SCALE VALUES
GOALS
4 44
17
41
9
83
9
Rating
Round 2
8
Delphi Methodology
275
1
II
~- -
kr
(a) 1 otal amount of change from round one to round four
1
slightly superior result was achieved [7]. Later, they found that in situations in
" “ch 'Native confidence was measured and in which the "highly confident”
d) Number of responses exactly on the mode in round three
group was reasonably large, a definitely superior result could be expected [4|.
•Studies in the psychology of small groups, however, indicate that highly
unfident persons should be less influenced by group pressure than those with
less confidence, and therefore it would be expected that highly confident
It Z 7
individuals move less toward consensus than do others in the Delphi context
,ater, Dalkey el al. [4] showed that “over consensus” may occur, and the ratio
>1 average error to standard deviation may actually increase, if consensus is
■orced too quickly. In order to reach some greater understanding of theory and
? reSP?T Wi,hin three places of the mode in round three
amounts of tmaDeliange mon'V°uld be
confidence with a h^h degr e of cZn^
7
II
the consensus in round thfee
Hi^^“?ZnZZn
§ ■"
with small
‘OW
3 high COnf°rmity to
n^tidual rr (~d by
I
i
M. Schfibc. M. Skutsch, and J. Schofer
1 ercent of
highly confident
Delphi Methodology
277
answers, however, was cross-correlated positively
with perceived academic status, although this was not significantly connected
with either movement variable. Amount of change in the second round was also
just correlated (positively) with the "at-onencss” variable, although there was
no relationship at all between "at-onencss” and percent highly confident. These
represented the only significant correlations found.
1 he evidence for the effect of confidence on the tendency to conxerge is
somewhat sketchy. The only conclusions that can be drawn from the experience
is that the initial surprise on being confronted with some distribution of group
opinion may to some extent cause the less confident members who believe that
they associate with the rest of the group to move toward the center of opinion,
but that this tendency is certainly not an overwhelming one.
At the end of the Delphi a second questionnaire (Appendix IV of this article)
a
as used to determine whether the kind of feedback provided had any
Ads
conscious effect on movement in the Delphi. The variables were as follows:
Post-Del phi Sun ey
Question Number
I
Evaluation:
Independent Variable
G.
Satisfaction with the results
strongly cross-correlated.) Clearly, the people who
.
,
> were strongly conforming
were not happy
I
...
with
the -Delphi at all. The question of what is cause and what
is effect, however, remains to be answered. Yet <
- --------------- .
one might speculate that,
pecially in a value-oriented Delphi, the group pressure from some forms of
especially
feedback can be overly
strong, forcing
forcing participa
participants to take positions which they
iverly strong,
find uncomfortable. While compromise may be uncomfortable in any situation,
the real danger here is that participants may leave the process without really
compromising their feelings at all. That is, perhaps the anonymity of the
Delphi itself may have encouraged participants to capitulate, but only on
"hard” feedback which tends to force what might be an irrational consensus,
one which might be only temporary.
5 and 6
I.
Lrustration with conununication levels
Opinion Stability as a Method of Consensus Measurement
7
J-
<)ptirnisrn for Delphi
In most Delphis, consensus is assumed to have been achieved when a certain
8
K.
Keeling of being rushed
percentage of the votes fall within a prescribed range—for example, when the
interquartile range is no larger than two units on a ten-unit scale. Measures of
this sort do not take full advantage of the information available in the
endency to converge strongly has elsewhere been shown by Schofer and
>kutsch 18] to be due to emphasis in the visible consensus and on the need to
reate consensus. Satisfaction with the process was also negatively correlated
with the conformity variable. (Satisfaction and agreement with feedback were
1*1
£
was carried out tended to discourage such responses. As a result, histograms of
value weights formed the bulk of the feedback. It is just this kind of limited,
Success of feedback in the learning
process
However, the success-of-feed back variable was strongly and negatively correitcd with the propensity to conform to the mode in round three. In other
words, those who did conform to the visible majority had difficulty in giving
and taking ideas from the feedback. 1 his is interesting in that it indicates the
iffcient kinds of feedback that may affect people in different ways. The
I
if
A cautionary note is relevant at this point. Another study by Skutsch [9] has
shown that the form of the feedback itself influences consensus development.
Despite the fact that participants in this experiment were encouraged to report
their verbal rationale for their positions, the rapidity with which the, process
H.
x i haps indicating that those people who were not caused to change their
opinion radically were in better spirits after the Delphi than the others.
i
paper. I hey may later hold to their original views, and, if the results of the
Goals Delphi are used to develop programs to meet their needs, participant!
might ultimately be quite dissatisfied with the results.
2,3 and 4
These variables were correlated with the same dependent variables. Both
optimism for the future of Delphi and satisfaction with the process correlated
significantly and positively with the number of monotonic changes made,
4
I
k
I
distributions. For example, a bimodal distribution may occur which will not be
registered as a consensus, but indicates an important and apparently insoluble
cleft of opinion. Less dramatically, the distribution may flatten out and not
reach any strongly peaked shape at all. The results of the Delphi are no less
important for this, however. Indeed, considering that there is a strong natural
tendency in the Delphi for opinion to centralize, resistance in the form of
unconsensual distributions should be viewed with special interest.
A measure which takes into account such variations from the norm is one
that measures not consensus as such, but stability of the respondents’ vote
distribution curve over successive rounds of the Delphi. Because the interest lies
in the opinion of the group rather than in that of individuals, this method is
preferable to one that, would measure the amount of change in each individual’s
vote between rounds.
To compare the distributions of opinion between rounds, the histograms may
bc subtracted eplumnwlse and the ahwlntp value of the
JQken.ln
2
■II
-1
I
4
<v,
Nl. Schcibc. M. Skutsch, and J. Schofer
chis approach is applied to the weight histograms reported for Goal 5 in the
irst thiee rounds of the Delphi. Columnwise subtraction between the first and
second, and the second and third, rounds gives the results shown in Table 2.
1 he .ibsolutr values of the differences between histograms arc aggregated to
“"i,s
‘•hatigr. but siinc anv one pain, ipanl’s . hange ol opinion is
^fleeted m the histogram dillc!cnees bv tsxo umls «.( change, net personhanges must be computed by dividing total units of change by two. Einally,
he percentage change is determined by dividing net changes by the number of
participants. Clearly, in the example shown in Table 2, the distribution of
alue weights for Goal 5 became more stable between rounds one and three.
The question of what represents a reasonable cut-off point at which the
.espouse may be said to be unchanged, and therefore finally in its stable
position, poses some problems, however. Since there is no underlying statistical
’hcory in* what has so far been proposed, no true statistical level may be set, as
'light, for example, be possible with a statistical change in variance test.1
Empirical examination of the responses in the Delphi, however, showed that
n any point in time a certain amount of oscillatory movement and change
within the group is inevitable. This might be conceptualized as a sort of
mderlying error function, a type of internal system noise. What is needed is a
confidence measure which allows the distinction to be drawn between this
isind of movement and strong group movements that represent real changing
pinion. Such an <estimate has tentatively been made from studies of observed
nobability of movement.
Leaving aside objective 3 (for reasons imade
' apparent earlier), the propensity
f the individual to alter his score as a function of distance from the■ center
ooint was measured. This was done by calculating the proportion of rrespon.cots at each scale distance from the mode that moved toward the_* mode
between rounds. The
I he results, displayed in Fig.
big. 6, show a strong tendency for
increased amounts of movement with distance from the center point. They also
how that a percentage change is to Ibe expected among respondents who are
Irtady dead on the mode itself. The amount
------ ----- of
~f movement at the mode (about
15%) has therefore been taken to irepresent the
’ base of oscillatory movement to
•e expected, and this is supported by the fact that the_• amount of change at the
ciilroid does not alter appreciably between rounds.
Using the 15% change level to represent a state of equilibrium, any two
istributions that show marginal changes of less than 15% may be said to have
cached stability; any successive distributions with more than 15% change
should be included in later rounds of the Delphi, since they have not come to
.ie equilibrium position.
Evaluation:
"I
•. ir
Rating
1
2
3
4
5
6
1
8
2
I
5-'»
3
Absolute difference in number
selecting rating, rounds 1-2 (a)
0
0
0
2
2
2
4
0
2
■t
Total units of change
12
Table 2:
Example of Stability Measurement Computations
for Goal 5
Net person-changes
(b)
(£)
6
Number of participants
21
Percent change
28.67.
(D
r
Rat ing
1
2
2
4
5
Absolute difference in number
selecting rating, rounds 2-3 (a)
0
0
1
0
1
Total units of change
4
Net person-changes
(b)
(£)
2
Number of participants
21
Percent change
9.57.
(d)
6
1
8
9
0
0
1
(a)
These numbers are the absolute differences in the histograms for the
two successive rounds.
(b)
These numbers are the sums of the absolute differences in the histograms,
(£)
Net changes are total units of change divided by 2.
(d)
Percent change is net change divided by the number of participants.
i
I
. Conventional variance tests were found to be unsuited to the case of change in histogram shape
n this context. Most rely on independent samples; none is strong enough to pick up small changes
i shape, and none robust enough to deal with non-normal distributions.
f
Delphi Methodology
o
•' I
I
iI
l
't
M. Scheibe, M. Skutsch, and J. Schofer
First round
G>
s
Evaluation:
Delphi Methodology
Table 3:
orid Round
281
Results of Stability Analysis
'i
•
Second round
Third round
£
Third round
Amount change
Rounds 1-2
Amount change
Rounds 2-3
1
13% stable
10%
2
17%
10% stable
3
38%
14% just stable
4
10% stable
10%
5
26%
107. stable
6
2% stable
2%
7
24%
33% increasingly unstable
8
22%
147. just stable
9
24%
147. just stable
Goal
Fourth round
■L
IJ
100
ii
11
°§ -io
G 'g
E
80
Average ' movement
over three rounds
60
40
£!
20Z _
1
1
J.
1
3
units from mode
iig. 6. Proportion of respondents shifting their positions, as a function of distance
from the mode.
I
I
I
JL
4
1 9
participating group. Use of this stability measure to develop a stopping criter
ion preserves any well-defined disagreements which may exist. To the organizer
of a Goals Delphi, this information can be especially useful.
I
•»]
The results for all nine goals included in this experimental Delphi using this
analysis are shown in 1 able 3. I rom these data, there can be no doubt as to the
general tendency toward stabilization. ()nly one goal. 7. had not reached a
.stable position by the end of the third round, although 3, 8, and 9 were all only
just stable.
In general, this method seems to have a number of advantages, l irstly, it
allows the use of more of the information contained in the distributions. There
are applications in which, at the end of the Delphi process, the entire distribu
tion may be used, as for example in linear-weighting evaluation models where
goal-weight distributions are treated stochastically, such as that by Goodman
110], In addition, this stability measure is relatively simple to calculate, and has
much greater power and validity than parametric tests of variance.
Perhaps most important, one of the original objectives of Delphi was the
identification of areas of difference as well as areas of agreement within the
Delphi Service Program
In order to make several iterations possible
in the
ofa very short time /
,
. - ...
-------space
r--------period, a computer time-sharing terminal was used to process the results of this
Delphi experiment. Unlike the systems described
VII, B),
described by
by Price
Price (see
(see Chapter
Chapter VII,
B),
the program used in this Delphi was an accounting device only; verbal
g
feedback was compiled and read to participants by the organizers.
In this application, histograms produced by the computer terminal were
copied by hand onto an overhead projector transparency to provide immediate
feedback to participants, who themselves determined their positions in the
distiibutions relative to the group. It is anticipated that, for future experiments,
computer-generated histograms will be produced in multiple copies,, one of
which will be provided to each participant:
4;
I
'JB2
M, .S< licibc, M. Skutscli, and J. Schofvr
This type of computer support, oriented toward the use of a single terminal
for all participants, may be especially desirable for Goals Delphi applications,
where, because of the lay nature of (he respondents, it seems especially
desirable to keep all of those involved in a single room, and to maintain a
relatively high rate of progress throughout the survey.
Closure
The potential applicability of the Delphi method to goal formulation and
priority determination for public systems is very great. Yet, because the detailed
characteristics of the design of the process can have important effects on the
nature of the outcomes, it will be important to tailor the Goals Delphi to the
problems at hand. The structuring of internal characteristics which are
appropriate to a Goals Delphi should be based on a rather complete under
standing of the linkages between form and function in the Delphi environment.
While considerable experience must be gained before Delphi can be offered as
a routine goal-formulation process, this discussion has suggested some structural
and process features relevant to this important application of the Delphi
method.
Evaluation:
Delphi Methodology
283
Appendix I: Hypothetical Decision Scenario
I hc following transportation-facility-location problem is offered as an
appropriate context for developing local-scale transportation planning objec
tives. Within this context, there is a need to establish an objective set, and to
evaluate quantitatively several alternative plans in the context of the objectives.
.A two-mile transportation link is proposed in an urban area. It is to run from
the Central Business District (CBD) to new, developing suburbs to the north.
This area is presently served by a four-lane boulevard with an average daily
traffic (ADT) of forty thousand vehicles, and by a four-lane street with an
AD T of twenty thousand vehicles. This street, however, circles an historical
area by means of four 90° turns, and traffic must travel this section at
twenty-five mph. The southeast corner of the historical area comes within five
hundred yards of the edge of a lake, and the main street presently is only one
block from the lake at this point. A tollway also passes the suburb and proceeds
in a southeasterly direction. The tollway passes within one mile of the CBD,
with its alignment located in a ravine. The elevation of the ravine is such that
to build a connector to the tollway from the CBD would require a great deal of
earthwork, and even with this the grade would be about 3%.
The alignment of the boulevard is such that it begins in the CBD, proceeding
northwesterly through a low-income area to a large park, where it turns and
continues in a northeasterly direction through a middle-income residential area
to the suburbs.
The four-lane street heads due north from the CBD, passes through an
industrial park, and then makes four sharp turns around the historical area and
proceeds directly into the suburbs.
a
Citizen opposition can be expected in four areas. Public opinion l^as long
been against any changes in the historical area. A citizen group can be
expected to form opposing an alignment through the park. One can also be
expected to form opposing removal of houses in the middle-income areas north
and east of the park. Problems can also be expected if the alignment goes
through the low-income area, requiring relocation of some households.
1
M. S
.■ t li. »Hid J. Schaier
Appendix 11: .Map of Hypothetical Iranspurtation Scenario
Evaluation;
Delphi Methodology
285
I
Appendix 111: Pre-Delphi Self-Rating Form
Code Number
PRE-DELPHI SURVEY
SL'BL'RB
1.
MEDIAN
FAMILY
INCOME
$9500
MIDDLE
INCOME
RESIDENTIAL
2.
HISTORICAL
AREA
F"
I
I
PARK
I
I
I
L.
-J
MEDIAN
FAMILY
INCOME
$12500
\\\
As a transportation planner, in a
class of transportation planners,
my skills in planning would put me
about here, relative to the others.
Very Highly
Skilled
I chink my ideas are, in essence,
in agreement with the rest of
the class.
Yes, absolutely
\\
\>
\
3.
RAVINE
I know most of the people in the
class very well.
4.
lake
3
4
5
6
’ 7
No, not
2
3
4
5
Yes, pretty much
1
MEDIAN
KAMI LY
1NCOME
$6300
2
at al 1
1
AX
\ \\
MIDDLE
INCOME
RESIDENTIAL
1
No Skill
at all
2
3
I have some definite ideas atout
what the goals in transportation
planning are and should be.
Yes, lots
1 have been in transportation for
longer than most of the other
people here.
Yes
I have a lot of experience in
planning outside of school.
Yes
1
2
3
4
6
7
No, none
at all
5
6
7
5
6
F
I
7
I
7
Ji
No, none
4
I
□
5.
•/)
i\
I
MEDIAN
KAMI LY
INCOME
$3200
UNDEVELOPED
WOODLAND
LOW
INCOME
RESIDENTIAL
6.
1
1
No
2
3
4
5
6
d
No
2
3
4
5
6
7
INDUSTRIAL
AREA
CENTRAL
BUSINESS
DISTRICT
7.
I am anticipating that the
Delphi is going to be a good
thing for goal setting.
Yes, I think
it will be
1
2
3
No, I think
it may be a
waste of time
4
5
6
7
!
I
286
M. Scheibe, M. Skutsch, and J. Schofer
Evaluation:
9
2S7; 'J
Delphi Methodology
1
Appendix I\: Post-Delphi Evaluation Form
References
Code Number
1
POST-DELPHI SURVEY
1.
2.
3.
4.
5.
6.
7.
8.
I feel satisfied with
with the results in
general.
I learned ideas from
the feedback.
In general, I agreed
with the ideas in the
feedback.
I could express my
ideas OK this way.
I feel as if I really
wanted to talk to
people.
1
3
4
5
' No.'l (XT'1'’-
6
7
1
2
3
4
5
6
7
1
2
3
4
5
6
7
1
2
3
4
5
6
7
1
2
3
4
5
6
7
I have a feeling
people didn't under
stand or think about
my reasons.
1
2
3
4
5
6
7
1 think the Delphi
could be operational
in goal setting more
generally.
!
2
3
4
5
6
7
1
2
3
4
5
6
7
1 think it went too
fast.
T. H. Ostrom and H S. Upshaw. “Psychological Perspective and Attitude Change ” in A. O
I'm not really happy
with the results at
all.
I didn't learn a thing
from the feedback.
I disagreed with every
thing in the feedback.
1 couldn't really write
what 1 wanted to say.
I didn't feel the need
to talk at all.
Va“dASn'Z HiXa/C
23,
American Society of Civil Engineers, forthcoming.
g J
4' P
D?lkTeBrown’ and S- Co<:hran. "The Delphi Method IV: Effect of Percentile
5 K,CCebnCk,Land 7 7; ° DRelevant Facts." Rand Corporation, RM-6118-PR, 1970.
' Vai5'
and. n ' F?Urke' “Experimental Assessment of Delphi Procedures with Group
v alue Judgment, Rand Corporation, R612-ARPA, 1971,
6. m
W. ‘ 3?rgerson’ Theory and Methods of Scaling, Wiley, New York, 1958
Dalkey,
7. N. C. Dalke
^ ‘7hc De’phi Method: An Experimental Study of Group Opinion,” Rand1
Corporation, RM-5888-PR, 1969.
H
x^ana
8’ ^.SkrUUCh an£J‘ L- Sfh°fcr’ lGoals DelPhis for Urban Planning: Concepts in their Designs,
Socio-Economic Planning Sciences, 7, pp. 305-313, 1973.
9. M. Skutsch, ‘‘Goals and Goal Setting:
Delphi Approach.
Setting: A
A Delphi
Approach.”” unpublished master’s thesis.
Department of Industrial Engineering and Management Sciemi. Northwestern University,
I
I •'
!
3
10’ iVth000^™’ uA GorpPu!erized Investigation and Sensitivity Analysis of a Linear Weightg cheme for the Evaluation of Alternatvies for Complex Urban Systems,” unpublished
masters thesis, Department of Civil Engineering, Northwestern University, 1971.
i
I think people under
stood my reasons pretty*
well.
I don't think it could
be operational, really.
I think it went too
slowly.
3
&
1
. D.
Change Responses
Propensity to Change Responses
Evaluation:
in a Delphi Round as a Function
of Dogmatism
what they should be. The latter set made each person a fully qualified
pert.”
With these question categories as a base, it was possible to use the questions
to define the respondents as expert or nonexpert.
The Delphi procedure was continued through three rounds. During rounds
two and three each subject was given the group median, interquartile range
and his own response to the previous round for each item. He was asked to
"review’ [his] projection on the basis of the information provided” and to
change his answer if he wished to do so.
NORMAN W. MULGRAVE and ALEX J. DUCANIS
Since one of the assumptions of Delphi is that it “reduces the influence of
certain psychological factors such as ... the unwillingness to abandon publicly
expressed opinions, and the bandwagon effect of majority opinion” (Helmer,
1966), it would seem of interest to examine the effect of personality upon an
individual’s performance during several Delphi rounds. Specifically, the ques
tion can be raised concerning the willingness of a more dogmatic individual to
change his answer in a Delphi round (whether he is an expert or a nonexpert).
Since dogmatic thinking is characterized by resistance to change (Rokeach,
1960), it might be posited that the dogmatic individual would be less likely to
change his position when confronted by the opinions of others. It might be
further presumed that the type of question asked, i.e., those upon which the
individual could be considered either more or less expert, might also affect
performance of highly dogmatic as opposed to less dogmatic individuals. It was
(herefore predicted that the number of < hanges made by a high dogmatism
gioup (1)H) would be less than a low dogmatism group ( />z ), and that the ( /)/z)
group would change less on questions on which they might be considered
expert than on questions on which they would be considered less expert.
Results
The Dh and DL groups were identified as those scoring in the upper or lower
27 percent of the class on the FCD Scale (Berger, 1967). Second- and thirdround changes were tabulated for those who were both inside and outside the
interquartile range for each of the four sets of questions. The results of that
tabulation are shown in 1'ables 1 and 2.
Table 1
Proportion of Individuals Changing Answers
Round Two
I'ypr of Item
The Delphi Method: Techniques and Applications, Harold A. Linstone and Murray T uroff (eds.)
ISBN 0-201-04294-0; 0-201-04293-2
Copyright © 1975 by Addison-Wesley Publishing Company, Inc., Advanced Book Program.
All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or
transmitted, in any form or by any means, electronic, mechanical photocopying, recording, or
otherwise, without the prior permission of the publisher.
Percent Changing
High Dogmatism
1 nsidc
Range
Outside
Range
I
Percent Changing
Low Dogmatism
I nside
Range
Outside
Range
■
Almanac
"School” Questions
Predictions of Values
What Values Should Be
Method
The subjects for the study were ninety-eight graduate students enrolled in a
class in Educational Psychology, most of whom were school teachers.
Berger’s (1967) revision of Rokeach’s Dogmatism Scale (the FCD Scale) was
administered on the first day of class. Subsequently the class was used as a
Delphi panel and asked to make certain estimates. Four types of questions w'ere
utilized. 'Ten questions defined the subjects as nonexpert, such as the number of
farms in the United States. 'Ten other questions concerning class size, teachers’
salaries, length of the school year, and similar items defined the subjects as
experts.
Eighteen other questions were value-oriented items. Subjects were to respond
in terms of what certain values in the United States will be in 1980, and also
289
29.5
20.3
13.4
12.8
76.2
64.7
74.8
71.9
13.6
6.1
4.5
7.2
63.1
54.0
50.8
59.3
I
•e
Table 2
Proportion of Individuals Changing Answers
Round Three
Type of Item
1
Almanac
“School” Questions
Predictions of Values
What Values Should Be
Percent Changing
High Dogmatism
Percent Changing
Low Dogmatism
Inside
Range
Outside
Range
Inside
Range
Outside
Range
6.9
4.1
8.9
9.2
29.2
19.7
6.8
3.4
1.9
15.8
1.9
14.5
14.8
6.7
6.0
18.6
I
:■
■
iNorniiHi W, Mulgruvu and ,-\lcx J. Ducanis
A significant difference was found Ixrtween the groups in the number of times
they changed their answers. For round two the value for chi-square was
computed as 18.48 with 7 degrees of freedom. 1 his value is significant at the .01
level. The corresponding value for the third round was 14.78, which is signifi
IV. E.
The Performance of Forecasting
Groups in Computer Dialogue and
1'ace-to-face Discussion
cant at the .05 level.
KLAUS BROCKHOFF*
Discussion
The Problem
It would seem that personality characteristics of the individual involved in the
Delphi panel have some effect upon his propensity to change. Of interest as
Advances in mathematical and statistical techniques, the availability of
efficient computers as well as ideas and attempts to utilize certain organiza-'
tional structures in the compilation of expertise are basic
elements for
basic elements
for an
well was the finding that the High Dogmatism group exhibits significantly
more changes. Thus the prediction that they would be less likely to change is
not upheld. A possible explanation for this may be that if the dogmatic
individual looks to authority for his support, then in the absence of any clearly
defined authority the dogmatic individual would tend to seek the support of
whatever authority seems present. In this case authority would be the median
of the group response.
I he second prediction that the High Dogmatism group would change less on
questions where they could be considered expert, i.e., “school questions” and
“what values should be” than on questions where they could not be considered
expert, i.e., “almanac questions” and “what values will be,” was upheld on the
Second round (chi-square 6.622 with one degree of freedom) but not on the
filial round. I here were no significant differences on either round for the Low
Dogmatism group,
These results seem to indicate that the High Dogmatism group is less likely
to change an answer to a question on which they consider themselves expert
than one on which they consider themselves less expert, but that in the presence
of some “perceived” authority such as the group median, High Dogmatism
groups will exhibit more change than Low Dogmatism groups.
References
\ . I-. Berger, “Effects of Repression and Acquiescence Response Set on Scales of Authoritarianism
and Dogmatism,” unpublished master’s thesis, University of Oklahoma, 1967.
N. C. Dalkey, “The Delphi Method: An Experimental Study of Group Opinion,” Rand Corpora
tion, 1969.
O. Helmer, “The Use of the Delphi Technique in Problems of Educational Innovations,” The
Rand Corporation, 1966, P-3499.
O. Helmer and N. Rescher, “On the Epistemology of the Inexact Sciences,” Rand Corporation
I960, R-353.
J. Martino, “The Consistency of Delphi Forecasts,” The Futurist 4 (1970), p. 63.
M. Rokeach, The Open and Closed Mind, Basic Books, New York, 1960.
intensified discussion of the problems of forecasting specific future developments
and events. As J. Wild has shown, those conditional forecasts which are derived
from models by certain statistical techniques are based as much on empirical
knowledge as on ad-hoc extrapolations, projections, or expert opinions.1 How
ever, the isolation of independent variables from the surrounding conditions
and the intra-personal process of information processing often do not become
clearly visible in the latter. Thus there is a danger that uncontrolled, or
uncontrollably, misinterpretations and false judgments may occur.
We do not want to infringe upon the controversy on the superiority of
forecasts as compared with “projections,” which is being carried out in the
theory of science2 as well as on an empirical-pragmatic3 level. It has by no
; 4'
i
I
■
■ ‘ia
•i T?
j
I
■
means been settled for the forecasts. This seems particularly true when the
comparison is drawn on the basis of a benefit-cost relationship.4 If only for this'
reason we are interested in the question whether or not the utilization of the
empirical knowledge of groups of experts in the derivation of statements about
future developments or events can be improved upon by organizational ar
1
rangements. Improvement is meant as an increase in the accuracy of these
statements. This is one reason for the development of the Delphi method?
•In collaboration with D. Kaerger and H. Rehder.
‘J. Wild, “Probleme der theoretischcn Deduktion von Prognosen,” Zdlschrift fur die gesamle
Staalswissenschaft (ZfgS) 126 (1970), pp. 553-75.
2Ibid. and E. v. Knorring, “Probleme der theoretischcn Deduktion von Prognosen,” ZfgS 128
(1972), pp. 145—48; J. Wild, “Zur prinzipiellen Uberlegenheit theoretisch deduzierter Proemosen”
ZfgS 128 (1972), pp. 149-55.
8
E.g., R. M. Copeland, R. J. Marioni, “Executives’ Forecasts of Earnings per Share versus
Forecasts of Naive Models,” Journal of Business 45 (1972), pp. 497-512, and the literature quoted
there.
Thus the suggestion in H. A. Simon, D. W. Smithburg, V. A. Thompson, Fublu Administration,
New York, 1961, p. 493.
O. Helmer, N. Rescher, “On the Epistemology of the Inexact Sciences,” Management Science 6
(1959), pp. 25-52.
I
I
The Delphi Method: Techniques and Applications, Harold A. tinstone and Murray Turoff feds'!
ISBN 0-201-04294-0; 0-201-04293-2
k
Copyright © 1975 by Addison-Wesley Publishing Company, Inc., Advanced Book Program.
All rights reserved. No part of this publication may be reproduced) stored In a retrieval system) er
transmitted, In any form or by any means, eleptronip, mechanical pholdropying, recording, qr
otherwlsj.’. whhnMt ih? prior prrmMan df the ptihlhhpr
5
I
■i
i'/Hn kiiuii
The conditions of group performance have been investigated in thousands of
studies. Only few studies have been devoted to the question whether the
peculiar organizational structure of the Delphi group6 leads to higher group
performance than the face-to-face discussion in a group in which the each-toall pattern of the communication system can be activated.7 Beyond this the
unwieldy, nonhomogeneous, and inaccurate definition of types of tasks by
w'hich the performance of groups is judged8 and thus the classification of
concrete formulations of the question make it very difficult to derive statements
as to the particular capacity of groups in forecasting,<J’
A large proportion of the statements as to> the superiority of certain forms of
group organization compared with others ’was obtained by observing group
performance in solving certain kinds of problems and by assuming that the
results would apply to tasks which appeared comparable. Thus references to
the ability of groups to forecast particular future events were judged on the
basis of their performance in responding to almanac-type questions. Martino
has demonstrated that the answers to almanac-type questions observe the same
type of distribution as the answers to forecasting questions.10 However, it has
not been investigated whether the parameters of the distribution vary signifi
cantly from one type of task to the other under conditions which are compar
able otherwise. 1 hus it appears desirable to reconsider the original assumption.
In the following report we try to investigate some of these questions experi
mentally.
2.
Initial Hypotheses on the Performance of Forecasting Groups
Evaluation:
293 1
Perfonnancc of Eorccasting Groups
The group size is determined by the number of members of
a group. Th||p®
measure refers solely to formal criteria. Thus a person who does not contribu® i
to the activtty of the group, either because of his own reticence or because
I
of a
|
formal system of communication which does not accept his contributions, is stfll7
considered a member of the group.
U
Group performance can Ibe synthesized by a statistical aggregation of
individual performances.12 However,
I’
such groups lack the essential
characteristic of communication which
in natural groups. In the follow----- exists
-------- .a
I
mg we will report entirely about experiments with natural groups.13
/
It may seem natural to study group performance of groups with a consideMfll
able number of members. However, in the experiments to follow we havSiH
deliberately concentrated on small groups of four to eleven people. One reason' W
for this is that very many small and medium-sized organizations are applying i 1
elphi. I hey can call in only small groups of experts. Even so they maylWfl
wonder about their performance, and how to measure it. With regard to
possible objection that the results from the observation of small groups may be'
subject to considerable "noise,” we may say that basic to most evaluations is theWi
use of the median of individual responses. The median, however, is not sensitiveXffl
to large dispersions, even if they are one-sided. On the other hand, it goes
without question that it would be desirable to repeat our experiments in order^iW
to check on the reliability of the results.
Very different things can be understood by the “performance” of a group:
|
be understood by the “performance” of a group.
:
The tnpel, number of pieces of
of information
information exchanged,
exchanged, time
time needed
needed for
for solving:
solving: :<-S
a problem and number of mistakes, can be considered a “classical” yardstick of performance. Ziegler ascribes the origin of this tripel to a paper written M
by
■:
Bavelas in 195014. As Barnard’s definition of performance—“the accomplishment of the recognized objectives of cooperative action”15—makes clear, how-’: jO
ever, this classical tripel is not compulsory. Indeed, it generally remains unclear
whether performance refers to the goals (recognized objectives) of the member
of the group, to those of the group, or to those presented to the group.
I he task given to the groups is to find an answer A to a question which j
deviates as little as possible from the answer A' which can be verified now or'in
'
the future. Increasing performance then means that
approaches 0. In
order to make comparisons between different questions or different groups a
,
‘
.
2.1 Group Performance and Group Size
2.1.1. Measurement of Variables II has been shown in various studies that the
performance of a group may depend on its size.11
6Cf, below, section 2.3.1.
7On the restriction of the each-to-all pattern cf. M. E. Shaw, “Some Effects of Varying Amounts
of Information Exclusively Possessed by a Group Member upon His Behavior in the Group,”
Journal of General Psychology 68 (1963), pp. 71-79.
flFor the differentiation of types of tasks cf., e.g., M. E. Shaw, Group Dynamics: The Psychology of
Small Group Behavior, New York, 1971, pp. 59, 403ff.; A. P. Hare, Handbook of Small Group Research,
New ^ork, 1962, pp. 246ff; C. G. Morris,“1 ask Effects on Group Interaction,” Journal of Personality
and Social Psychology 4, (1966), pp. 545-54. For problems of definition also R. Ziegler,
Kommunikalionsstruktur und Leistung sozialer Sysleme, Meisenheim a. Gian, 1968, pp. 96ff.
I he limited choice of type of tasks and the strict assumptions on experimental group problem
solving are most criticized. H. Franke, Gruppenproblemlosen, Problemlbsen in oder durch Gruppen? Problem
und Enlscheidung, Heft 7, Munchen, 1972, pp. 1-36, here p. 26 el seq.
IOJ. P. Martino, “The Lognorinality of Delphi Estimates,” Technological Forecasting, 1, (1970) pp.
355-358.
Gf., e.g., J. D. Steiner, Models for Inferring Relationships between Group Size and Potential
Group Productivity,” Behavioral Science 11 (1966) pp. 273-83; F. Frank and L. R. Anderson, “Effects
of I ask and Group Size upon Group Productivity and Member Satisfaction,” Sociomelry 34 (1971),
pp. 135-49.
--
,
■
■■■:
'1
,2For a chronology of the publications on statistically “synthesized” performance, cf. I. Lorgc, D.
ox J. Davitz and M. Brenner, “A Survey of Studies Contrasting the Quality of Group
hereOppa367fand nd‘V,dua Performance» 1920-1957,” Psychological Bulletin 55 (1958), pp. 337-72,
,3Here natural group does not mean only a natural group with a:
ach-to-all pattern of the
Se7 KUnBr^khoHS,“7’ F f ' ' 1
narrOwer definition in my earlier publication.
.
K. Brockhoff, Zur Erfassung der Ungew1Ssheit bei der Planung von Forschungsprojekten
(zugleKh em Ansatz zur Bildung optimaler Gutachtergruppen),” in H. Hax, Enlscheidung bei
unsicheren Em-artungen, Koln, 1970, pp. 159-88, here pp. 167f.
R. Ziegler, op. cil. p. 18; see also p. 55.
C. J. Barnard, The Functions of the Executive, Cambridge, Mass., 1962, p. 55.
1
■>
i i
Evaluation:
! sl.i.r-. ii-•*. khoff
standardization is necessary’
the correct answer is used:
Performance of ru»ccu»iing Groups
295 i
The assumption of a decreasing error with increasing group size is based bn
the probability model of search.21 From this model one deduces the possibility
Thus the relative deviation of an estimate from
I
of compensating individual errors by calculating the mean for the group.
In natural groups, however, the rigid conditions on which alone the state
ments of the probability model are valid cannot always be fulfilled. Since,
however, a consistent system of other factors that influence group performance
as well (as the direction of their influence cannot be given), we may formulate:
H - -i'|
.1'
Wc call this expression the “error.” If the error refers to a person, we speak of
i
an individual error. If the error refers to group performance, we speak of a
group error. If .1 is the median of the estimates of all members of a group, we
Hypulhests (J: With increasing group size, the group performance increases cetens paribus.
speak of a median group error (MGE).
The “mean group error” as used by Dalkey16 is not identical with the MGE
It becomes clear in section 2.5 to what extent the restriction ceteris paribus can
l
be repealed in our experiments.
as given here. The basic difference is that Dalkey uses the logarithm of the
V
quotient A /A' in order to test hypotheses about the distribution of his “mean
group error.” The distribution is of secondary importance for our present
considerations regarding performance. For this reason we do not use logarithms
J
2.2 Group Performance and Expertise
here.
'The MGE is used here directly as a measure of performance. It is not pul
' . •
2.2.1
The Measuring of Expertise H. A. Simon represents “expertise” as a
possible basis of authority or as a form of authority.22 By expertise we mean
into relation to the expenditures made for its derivation.
Further measures of group performance which arc mentioned arc the ability
of the group to survive in a changing environment, its satisfaction, and the
expert knowledge upon which professional authority can be founded. This
expert knowledge can be “proven”23 by demonstration or by recourse to
'i
I
confirmation through third parties. A “proof” by recourse to third parties can .•>
hardly confirm more than a refutable conjecture as to the expertise of a person.
If influencing variables of group performance are being sought, the experiments
which make these variables visible can hardly contain an analyzable test of
habitual change of its members.17 We do not intend to study group perfor
mance in such a broad context (although we have unsystemancally collected
remarks on member satisfaction).
expertise at the same Itime. It is necessary to measure expertise as an independent variable by some: other means.
One could proceed by testing which persons demonstrate expert knowledge
in solving fact-finding questions. When such persons have been found, they can
be engaged in forecasting. This takes for granted that the answering of both^
tvpes of questions can be considered to be identical types of tasks.24
24 Until now,
'The Relationship of Performance to Group Size’ 'The study of hypotheses
' „
,> size and group performance occupies a
about a relationship‘ 1between
group
in small-group research. A brief survey of the diversity of
prominent position ... -......... «
,
empirical results» was compiled by lurk.
Turk.118 /.
A -■
uniform result cannot truly be
expected,
because
the
individual
studies
were
expected, because the individual studies were carried out under different
2.1.2
conditions (types of tasks, performance measures, etc.). With respect to forecast
ing it has been hypothesized that the mean group error decreases with
increasing group size.19 It should be taken into account, however, that this
statement has been formulated only for synthetic groups, i.e., a statistical
aggregation of individual judgments, and with reference to the performance of
however, no empirically tested statement to this effect exists.
2lC(., e. g., P. R. Hofstatter, Gruppendynamik. Die Krihk der Massenpsychologie, 11th ed., Reinbek,
1970, pp. 35ff., 160ff.
22H. A. Simon, Administrative Behavior, 3rd ed., New York and London, 1965, p. 76.
23F. Landwehrmann, “Autoritat,” in E. Grochla (ed.), Handworterbuch der Organisation, Stuttgart,
1969, col. 269-73, here col. 270, refers to H. Hartmann, Funktionale Autontat, Stuttgart, 1964,
24For a procedure oriented thus, cf. M. A. Jolson, G. L. Rossow, “The Delphi Process in
Marketing Decision Making,” Journal of Marketing Research, 8 (1971), pp. 443—48. Another proce
dure, based on the solution of test questions and a test of the understanding of professional
tcrminology, is described by A. J. Lipinski, H. M. Lipinski, R. H. Randolph, “Computer-Assisted
Expert Interrogation: A Report on Current Methods Developm
Technological Forecasting and
Social Change 5 (1973), pp. 3-18, here pp. 9f. (The same in S. Winkler fed.], Computer Communications,
Impact and Implications, New York, 1973, pp. 147-54. The authors also test the “quality of
respondents’ comments” [presumably on factual questions], the degree of attention and the degree
of optimism [with the aid of a price list for old phonograph records] and inquire from this a rank
order of expertise.)
I ’
the group in answering fact-finding questions.
16CI. N. C. Dalkey, “The Delphi Method: An Experimental Study of Group Opinion,” Rand
Coro. RM 5888 PR, 1969. Also H. Albach. -'Informationsgewinnung durch struktunerte Gruppenbefragung—Die Delphi-Methode”, Zeilschnft fur Beinebswirtschaft {ZfB), Suppl. 40, \ r. 1970, pp.
11172Summldzed, e.g.. by M. Deutsch, ’'Group Behavior.” in D. L. Sills (ed., International Ency
clopedia of the Social Sciences 6, New York. 1968. pp. 265-75, here p. 274.
'8K Turk “Gruppenentscheidungen. Sozialpsychologische Aspekte der Organization kollektiver
Entscheidungsprozesse,” ZfB 43. (1973), pp. 295-322, here p. 302.
19N. C. Dalkey, op. cil. pp. 9f.
20“These were questions where the experimenters knew the answer but the subjects did not”: N.
C. Dalkey, op. at., p. 10, In.
I
KL.H Bh. ki. Ji
i .iHiia i , pue pouiq cuiiMder whcthei ex|jcnise lutings by third parties can be
u.'icd. However, it may be of interest to maintain the anonymity of all persons
who may possibly participate in a forecasting
f
group.25 Another problem that
arises is what criteria should be used. iin choosing (hose persons who are to judge
the expertise of others.
Thus there remains the possibility of determining expertise by self-rating. For
(his purpose an ordinal scale is generally used, from which one value can be
chosen to indicate expertise. We worked with a scale of real numbers graded
from 1 to 5, in which low numbers must be used to express a low degree of
expertise while high numbers may be used to express a high degree of expertise.
Such a determination of expertise is employed already in some forecasting
groups by their management.26 These results of individual forecasts are
vseighted according to the self-ratings when a group judgment is derived.
Measurements of expertise which are obtained for individuals should also
permit statements as to the expertise of the total group. Since self-ratings are
measured on an ordinal scale it is not permissible to form an arithmetic mean
of all the self-ratings of the members of the group. We therefore characterize
the expertise of a group by the median of the individual self-ratings.
Whether such self-ratings have a high positive correlation with ratings by
third parlies has not yet been studied in realistic situations. For this reason it
remains an open question whether corresponding confirmatory results of
psychological tests27 can be applied to real situations, which generally are not
free, of conflicting interests.
Even if no significant positive correlation exists between ratings by third
parties and self-ratings concerning expert knowledge, it is not determined
which of the ratings is more correct. Thus in the approach used here, which
involves self-ratings, the question remains whether the participants in the
experiment rate themselves correctly when compared with an (inapplicable)
objective standard.
~.2.2. The Relationship between Expertise and Group Performance
We assume that
gioups with high self-ratings of expertise perform better than groups whose
members rate themselves as less qualified. With this assumption we follow
Dalkey, Brown, and Cochran28. I heir results must, however, be examined with
edic insofar as they were obtained from answering fact-finding questions.
1'111 thermorc, the subjects were able to compare all questions to one another
25Cf. Section 2.3.1.
•<>D. L. Pyke, “A Practical Aporoach to Delphi: TRW’s Probe II,” Futures 2 (1970), pp. 143-52;
H. P. North, D. L. Pyke, “Probes of the Technological Future,” Harvard Business Review 3 (1969), pp.
63-76; A. J. Lipinski, L. M. Lipinski, R. H. Randolph, op. cit., pp. 1 Iff.
27M. A. Wallach, N. Kogan, J. Bern, “Group Influence on Individual Risk Taking,” Journal of
Abnormal and Social Psychology 65 (1962), pp. 75-86, here p. 83.
C; Dalkey, B. Brown, S. Cochran, “ I he Use of Self-Ratings to Improve Group Estimates,”
I ethnological Forecasting, 1 (1970) pp. 283-292.
1'A'alu.iiion:
PeHui inancc of i’<
oups
297
before rating their expertise with rrespect to each
‘ question. In many applica- I
lions it is not possible to present all questions at once. We shall therefore^ollow
a different procedure by presenting tasks in a sequential manner. Even so, we
assume that the basic relationship is still valid. Fhus, we arrive at
Hypothesis E: With increasing expertise, group performance increases ceteris paribus.
2.3 Group Performance and Communication System
2.3.1 The Characteristics of Face-to-Face Discussion Groups vs. Delphi Groups The
question whether decentralized or centralized organizations exhibit higher
performance is another of the classical questions in organization research. The
attempt to set up a universally applicable rule of organization to answer this
question had to be dropped because little by little conditions become known bn
which first the one organizational structure and then the other seemed more
advantageous for accomplishing specific tasks. Fundamentally, it is assumed in
these studies that all organizational structures considered are capable of ac
complishing certain tasks. A formal organization is characterized essentially by
its communication system and the distribution of competence among its mem
bers. It is further assumed that fact-finding questions as well as forecasting
questions can be answered more accurately by groups than by individuals, if
the (expected) error” is taken as the measure of accuracy.
Fact-finding questions and forecasting questions can be discussed in natural
groups with an each-to-all pattern of communication. If it is desired-to have a
group judgment, this task can be left up to the group, or a rule for aggregating
the group judgment from the individual judgments of the participants can be
given. Depending on the situation, the use of such a rule can be restricted to
the case where the group does not agree on a group judgment within a given
period of lime.
°
Particularly Carzo’s results indicate that in natural groups in which com
munication is not limited, solutions to complex tasks are reached rapidly, with
few errors, and to the satisfaction of the group members.32 One objection to this
is that this form of organization may also produce dysfunctional effects.
A first dysfunctionality may arise as certain group members consciously or
unconsciously influence the group result to a greater degree than their expertise
‘’This is made particularly clear by H. Albach, “Organisation, betriebliche,” in HMrlabuch i
Sozialwiss. 8, Stuttgart, Tubingen, Gottingen, 1961, pp. 111-17.
ren
als betricbliches Prognose-und Planungsvcrfahren, Zeilschnfl fur betnebswirtschafthche Forschung (ZjbF), N. S., 22nd Yr. (1970), pp. 128-37 here
particularly fn. 8.
1
’
F^C- ^epeTrim'natS by B- ,COntini' “The Value ol Timc in Bargaining Negotiations-Some
Empirical Evidence, American Economic Review 58 (1968), pp. 374-93.
Cf. R. Carzo Jr., “Some Effects of Organization Structure on Group Effectiveness,” Administra
tive Science Quarterly 7 (1963), pp. 393—424.
nmuira
..s
I
!
i
I
I
J
Evaluation;
Performance of i oiccasung Groups
warrants,44 (\ further dysfunctionality arises if the exchange of information is
the fourth round.39
interfered with by “noise.”34 A more far-reaching possibility for the occurrence
of dysfunctional effects is that the transmission of information necessary for task
accomplishment from some group members to others is blocked by somebody
interested or that the transmission of information from some group members
in the time period given for accomplishing the task is altogether impossible. In
the latter case the participation of the individual group members in the
exchange of information may be independent of the degree of expertise. Then
participants with a high degree of expertise cannot necessarilv influence group
performance.
'
r
One can summarize that possible dysfunctionalities in a natural group with
• in «•;.« h-u,-all palK Tii of <<,iiu„uni. ati.di
..ijli/ing i|„. , ,,i,,„i.-.,
lion syslcm .m.l lhe system ol < omp<■!< „, <•
|, ,|(l(.s
1|(.
postl.v. ly Wtth th,- .leg,,-,. „| exp,-, use Axsumtug (I,.,, these elf,-. Is often
mteilcie wtth group petlonimnee, tulcs should be set up which reduce lhe
effects of dysfunctionality or prevent their appearance. A set of such rules was
suggested and introduced by Helmer and Dalkey36 and given the name
•Delphi.” In spite of diverse variations in procedure, the applications known to
date have as their primary objective: “...the establishment of a meaningful
group communication structure.”47
299
I J
This leads to
■
Hypothesis R : The performance of Delphi groups increases ceteris paribus with increasing
number of rounds.
Hypothesis R: The performance of the .Delphi groups increases ceteris paribus with
increasing number of rounds only at first. Finally, the increase in performance can be reduced
and ini’Htrd. The ,/urslmn m which round the "per/,„m,mce inversion" begins must be
an.\u>tifd cmpitually.
According to Dalkey’s studies, Delphi groups de
monstrate a certain, though not significant superiority when compared with
certain other groups in solving fact-finding questions.38' We apply this perception to our
1
r
Hypothesis V: The variance of answers around the median decreases ceteris paribus with
increasing number of rounds.
The Relationships
■
We test this hypothesis here for up to five
f
rounds. It cannot be expected that Vt
hypothesis R' is valid for an unlimited number of rounds, since: growing
|
dissatisfaction of the participants and increasing time requirements make
it C
________
seem senseless to continue the consultations indefinitely. Therefore, we m<
modify
hypothesis R' to__________________________________________________________ ’ ‘?||
Finally, Helmer and Dalkey40 showed an interest in the observation that the
variance of the responses around the median decreases with increasing number
of rounds. The reduction of variance is not in itself a criterion for increased
performance. One must view this observation on the basis of hypothesis R:
together with it, variance reduction gains importance in the sense that it means
increasing certainty and accuracy of the answers.41 We therefore also test this
question by investigating
d.3.2.
K
Two statistical measures of variance are at our disposal for testing this
hypothesis: average quartile difference and average variance from the median.
I he latter measure offers certain advantages for a comparison between groups
of different sizes. For this reason it is given preference here.
Hypothesis D: The perfonnance of Delphi groups is ceteris paribus higher than the
performance of natural groups with an each-to-all pattern of communication.
I he diverse arrangements of Delphi experiments make it necessary to
investigate some of their special features as well. Of particular importance is
the question whether the performance of Delphi groups is the same in each
round, or whether it increases with increasing number of rounds, at least up to
'ij
■ ;f
i
2.4 Group Performance and Type of Task: Fact-finding Questions and
Forecasting
“This very abridged presentation must be viewed on the basis of the entire discussion concerning
the question: which conditions promote behavioral conformity in individuals in groups- cf A P
Hare, Handbook .... op. at., chapters, 2, 13 (there in reference to status rivalry); L. Festinger, E.
Aronson, I he Arousal and Reduction of Dissonance in Social Contexts,” in D. Cartwright, A.
Z-ander (eds.), Group Dynamics: Research and Theory, Evanston, III., 1960, pp. 214-31
A. P. Hare, op. at., chapter 10, pp. 272ff.
”On this broad field see H H. Kelley, J. W. Thibaut, “Group Problem Solving," in G. Lindzey,
L^Aronson (eds.), Handbook of Social Psychology 4, Reading, Mass., 1968, pp. 1-101, here pp. Off,
36N. C. Dalkey, O. Helmer, “An Experimental Application of the Delphi Method to the Use of
Everts, Management Science 9 (1963), pp. 458-67.
M. Euroff, “Delphi and its Potential Impact on Information Systems," AFIPS Conference
I roceedings, Fall Joint Computer Conference (Fall 1971), 39, pp. 317-26, here p. 317.
N. C. Dalkey, op. cit. passim, particularly
1
particularly p.
p. 22.
22.
Only a few of the generally available results of forecasts by Delphi groups can
be tested againstt reality, because they mainly refer to events which are
39Cf also J. B. Martino, “An Experiment with the Delphi Procedure for Long-Range Forecast
I
ing, IEEE Trans, on Engineering Management 15 (1965), pp. 138-44.
20400 HelrnCr’ S0Cial TecKno10^' New York and London, 1966, pp. lOlff.; N. C. Dalkey, op. cit., p.
4,They are based according to the Delphi method, on renewed intrapersonal conflict solution
and problem solving, after being provided with additional data. On the interpersonal process which
is to be eliminated here, cf. J. Hall and M. S. Williams, “A Comparison of Decision-making
Performances in Established and Ad Hoc Groups,” Journal of Personality and Social Psychology 3 (1966),
pp. 214-22. On the practical organization of the elimination of dysfunctionality and pressure to
conform, cf. 3.2, below.
‘
I
i
0
1
Klaus Brockhoff
expected to take place in the distant future. For this reason, the comparison of
performance of face-to-face discussion groups and Delphi groups is made by
observing tasks which appear similar.4 The problem of solving fact-finding or
almanac-type questions is assumed to be similar to forecasting. The answers of
such questions are, as a matter of principle, unknown to the participants but
known to the experimenters. These two applications of Delphi, its use in
forecasting and its simulation with only subjectively unknown bits of know'ledge, exist as yet side-by-side without comparison. Since the complexity of a
task is an important determinant of group performance, but the criteria for
determining tasks of varying degrees of complexity are not clear enough, we
want to test directly
Hypothesis F: The performance of a group in answering fact-finding questions is ceteris
paribus equal to that in forecasting.
The hypothesis is deduced from the assumption that both types of tasks
exhibit the same degree of complexity. The tests should be carried out
separately for face-to-face groups and Delphi groups of the same size. If the
hypothesis is refuted, many of the statements about Delphi-groups which we
rederived using fact-finding questions cannot be maintained. In order to test
hypothesis E, we chose only facts referring to events that had occurred, on the
average, six months before the experiments. 'The forecasts, on the other hand,
refer mainly to a period of time which did not exceed six months after the tests
were carried out.
2.5 The Relationship between the Hypotheses
The repeated use of ceteris paribus in the hypotheses leads us to assume that they
arc in fact related to each other. It seems senseless to describe the great variety
of possible combinations. The rejection of certain hypotheses can reduce the
test program considerably as it leaves only a small number of relationships
which need to be tested. Hypothesis E, e.g., can be tested for a given group size,
a given type of question, a given type of group, and, in Delphi groups, with
regard to the results of any round. No matter what the result is, it is irrelevant
for the further experiments, if expertise should be distributed equally in the
different groups. Since the distribution of the actual expertise becomes known
from the results of the experiments, it is advisable to bring about the necessary
clarifications at first.
The situation is similar when discussion groups and Delphi groups arc
compared with each other. 11 hypothesis /? is not tested for the latter, it cannot
■‘2N. C. Dalkey, op. cit., pp. 9f. Dalkey cites (p. 23) a paper by Campbell, in which similar
methods evidently were used to the ones planned here. The original publication was not available.
Evaluation:
Performance of Forecasting Groups
301
il
be determined from which round the results should be taken if, compared with
the performance of the face-to-face discussion groups.
The presentation of our results in Section 4 is organized according to such
reasoning.
3.
The Experiments
3.1 Participants, Group Formation, Place, and Time
All experiments were carried out as part of a lab. course listed in the University
of Kiel catalogue. It was planned to have “students” and practitioners work in
separate groups and to compare the results. However, since we could not give
credits for the course, too few students registered to be able to form even one
small group.
Practitioners were designated and chosen from the permanent staffs of the
local banks with the assistance of the bank managers. Bank employees were
chosen because hardly any other line of business is represented in the area by
enough individual organizations with personnel trained in economics and with
relatively uniform fields of business. At the same time, the size of the participat
ing organizations is generally so large that persons who perform specialized
functions (long-term credits, short-term credits, investment brokerage, etc.) and
who differ with respect to the lengths of their employment could be chosen.
This seemed desirable in order that definite differences could show up in the
self-ratings of expertise in reference to the individual tasks.
The thirty-two participants were randomly assigned to four groups, having
five, seven, nine, and eleven participants, respectively. At the face-to-face
discussions, however, registered participants were absent for various personal
reasons, so that the groups had four, seven, eight, and ten participants only.
The experiments began with an introductory lecture about forecasting
methods and an exercise in the use of the displays which were used in the
experiments with the Delphi groups. After that, each subject participated in a
session in which the members were organized as a Delphi group. At a later
session a face-to-face discussion took place. After the experiments were con
cluded, an opportunity for criticism was given. Eight months later, the results
wt^e communicated. We have tried to motivate our participants to cooperate
well in the preparatory lecture and demonstration. Besides, we offered book
prizes for outstanding performances with regard to different types of questions
4
i
and the two basic group structures.
Fhc experiments were conducted in May and July 1973. With three excep
tions they were scheduled for Thursdays to conform with late closing time of
banks. Fhc Delphi groups worked in the computer center of the University of
Kiel; the face-to-face discussions were carried out in a library room.
1
I'.v aluaiion:
....
* i»*>‘ vJtgaiii/.aijoii of the Ucjphi Groups aud the Faeoto-Face Groups
1 he Delphi groups were set up so that the participants received all information
from the experimenters on a computer-generated display.43
1 he participants in a group were not supposed to establish immediate
contact with each other. They responded to all questions by writing an
alpha-numeric text in their normal language. The responses can be divided
inio ihtee classes: (1) responses that were known onlv to the experimenters; (2)
I espouses whic h, aftei the iespouses of all pat licipanls had been teceived by the
expetirnenter, became objects of computing procedures, the results of which
weie made known to all participants; (3) responses which were recorded and
made known to all participants without any changes. The first class includes
the name of the participant and the degree of expertise that he expresses with
regard to each question. This is handled differently in the face-to-face dis
cussions groups. A response of the second category’ is an individual estimate.
After the computation of the median of the responses of the group members,
(his figure is made known to all participants. The third category includes all
arguments for divergent opinions of those whose responses lay outside of the
lower or upper quartiles.
Computer communication has been praised as a means to enable experts to
communicate with each other even though they are separated from each other
by large distances. In the real world this could mean savings in travel expenses
and in the efforts expended in coordinating dates for groups of experts. Beyond
(his it is of importance for the experiments that the computation of quartiles,
and the preparation, distribution, collection, reproduction, and renewed distri
bution of questionnaires do not have to be carried out by hand during the
sessions. This gives one the chance to shorten the experiments considerably.44
1 he entire exchange of information between the participants as well as
between the experimenter and the participants during the sessions, with the
exception of certain recurrent standard formulations, was stored on tape as a
record of the experiments.
Figure 1 shows part of a record. A separate data file, an abstract from the
iecords, is kept on tape. It serves as the data base for the diverse computations.
I he abstract from the record shows the beginning of a session of the Delphi
group with seven participants. Vertical lines on the left edge of the text signal
those portions of the texts which appear in the same form on the display of each
paiticipant. In section 1 the names of the participants are given to the
<xpeiimenter. Section 2 contains the first fact-finding question for the group. In
Pcrfonnaib ; • (
:t
:
u|)S
303
Igeben sie bitte ihren namen an
BRIDSTRUP
CARL MOELLER
BOLTE
1
JOERG BISKUP
THOMAS M. RIECKEN
GLOCKNER
KAEMPFER
FUER WELCHEN BETRAG KAUFTEN DIE KREDITINSTITUTE
DER BRD 1972 FESTVERZINSLICHE WERTPAPIERE
2
i
IN MRD.DM
FAKTEN... KAUF VON FESTVERZINSLICHEN WERTPAPIEREN
DURCH KREDITINSTITUTE IN DEN JAHREN 1967 BIS 1969
JEWEILS ZWISCHEN 10 UND 15 MRD.
3
BRUTTOABSATZ FESTVERZINSLICHER WERTPAPIERE 1971
I
30,3 MRD. DM
IGEBEN SIE IHREN EXPERTENGRAD--ZIFFER 1. BIS 5.-
I
EXPERTENGRAD
FRAGE
1 TEILNEHMER
3
3
EXPERTENGRAD
FRAGE
1 TEILNEHMER
4
2
EXPERTENGRAD
FRAGE
1 TEILNEHMER
5
1
EXPERTENGRAD
FRAGE
1 TEILNEHMER
6
1
EXPERTENGRAD
FRAGE
1 TEILNEHMER
7
3
EXPERTENGRAD
FRAGE
1 TEILNEHMER
8
u
EXPERTENGRAD
FRAGE
1 TEILNEHMER
9
u
IGEBEN SIE IHRE SCHAETZUNG FUER RUNDE
SCHAETZUNG
ZU DIESER ERAGE AN..
if
U
II
1 AN. .
I
1
FRAGE
1
TEILNEHMER 3
50.00
SCHAETZUNG
1
FRAGE
1
TEILNEHMER U
US.??
SCHAETZUNG
1
FRAGE
1
TEILNEHMER 5
35.00
SCHAETZUNG
SCHAETZUNG
1
1
FRAGE
FRAGE
1
1
TEILNEHMER 6
TEILNEHMER 7
50.00
48.00
SCHAETZUNG
1
FRAGE
1
TEILNEHMER 8
U0.00
SCHAETZUNG
1
FRAGE
1
TEILNEHMER 9
21.50
i
5
‘
ABWEICHUNGSFRAGE AN TEILNEHMER
TEILNEHMER
9
GESTELLT
6
GESTELLT
I!
1
9 -INFORMATION..
I
KEINE
1 he programming was done by I). Kaerger, who will report separately on problems that arose
herewith. The program had to be in FORT RAN IV. It was run on a PDF 10.
For a brief discussion of the advantages and disadvantages of “computer communication’’
compared with direct communication, cf. A. I. Lipinski, H. M. Lipinski, R. H. Randolph, of), at.,
particularly pp. 11-12.
ABWEICHUNGSFRAGE AN TEILNEHMER
TEILNEHMER
6 -INFORMATION..
1972 WAR EIN BESONDERS GUTES JAHR FUER DEN ABSATZ VON WERTPAPIEREN
FUER WELCHEN RETRAG KAUFTEN DTE KREDTTTNCTTtutf
DER BRD 1972 FESTVERZINSLICHE WERTPAPIERE
6
11
7
IN MRD.DM
Fig. 1.
Abstract from a record
I
Klaus Brockhoff
Evaluation:
IFAKTEN... KAUF VON FESTVERZINSLICHEN WERTPAPIEREN
BRUTTOABSATZ FESTVERZINSLICHER WEHTPAPIERE 1971
32,3 MRD.
DM
u5.22
MITTELWERT =
KEINE INFORMATION
1 97 2 WAR EIN BESONDERS GUT.ES JAHR FUER DEN AESATZ VON WERTPAPIEREN
IHRE
SCHAETZUNG FUER RUNDE
2 AN. .
SCHAETZUNG
2
FRAGE
1
TEILNEHMER
3
SCHAETZUNG
2
FRAGE
1
TEILNEHMER
SCHAETZUNG
2
FRAGE
1
TEILNEHMER 5
35.22
SCHAETZUNG
2
FRAGE
1
TEILNEHMER
6
5 2.22
SCHAETZUNG
2
FRAGE
1
TEILNEHMER 7
48.22
SCHAETZUNG
2
FRAGE
1
TEILNEHMER
8
35.22
SCHAETZUNG
2
FRAGE
1
Ieilnehmer 9
4 2.32
ABWEICHUNGSFRAGE AN TEILNEHMER
TEILNEHMER
5
9
I
IGEBEN SIE
I
52.22
^5.22
12
I
I
t
GESTELLT
5 -INFORMATION..
FIGURE 1 : ABSTRACT FROM A RECORD
Fig. 1.
(continued)
section 3 you see information which is considered relevant for the judgment of
the problem and which is given to each participant. Additional information of
this sort is not given to participants when forecasting questions are asked. This
difference is justified by the assumption that the subjects need some information
to refresh their memories with respect to judging “facts” which are about six
months old. It is expected that they do not need help in evaluating present-day
facts as a basis for their forecasts.
In the following section, 4, participants are asked to give their degree of
expertise. It is given only once for each question. In section 5, we enter the
response portion of the first round.
Fhe numbering of the participants in sections 4 and 5 serves only as an
internal identification. Il begins with “3”, because “1” and “2” are reserved for
the experimenter and the tape on which we store the record.
After the estimates have been made (in section 5) the 0.25, 0.5, and 0.75
quartiles are calculated. The participants whose estimates lie outside the 0.25
II
; I
BIS 1969
JEWEILS ZWISCHEN 12 UND 15 MRD.
ZUSATZINFORMATION..
305
and 0.75 quartiles are shown the quartile values which they exceeded or fell
short of, and are asked to give reasons for this divergence, in case they think
they possess particular additional information. The text of the questions, whichI
is repeated at this place, is not put out in the record shown here. In section 6,. I ,•
I
data necessary for the analysis are recorded.
1
Fhe second round begins with section 7. First, the question is repeated, then
the relevant information. In section 9, additional data are given, namely, group
responses from the previous round and any additional information which was
collected in section 6 of the previous round. This information is presented in the
following order: first, additional information from those participants whose
estimates fell short of the lower quartile; then, information from those whose
estimates exceeded the upper quartile. In the present case, there is only one
item of additional information. The original text accompanying this informa
tion, which does not refer to its sources, is not reproduced here.
Beginning with section 10, the process which was described for section 5 and
the succeeding sections is repeated. Five rounds are carried out for each
question. This scope was chosen in compliance with the observation that after
the fourth round generally the results do not improve (see section 2.3.2). An
additional round is added here to test this statement.
After the five rounds are completed the next question is asked. It is a
forecasting question. The two types of questions are asked alternately s > that
possible effects of learning or fatigue do not influence only one type of qu stion.
In face-to-face discussion groups, the group members are asked to int xluce
themselves to each other by their name, field of employment, official pc dtion,
and the number of years spent in banking. The idea of this was to provid ^each
participant with a basis forjudging the experience of the discussion partners in
the following discussions. To what extent this information was take i into
account in the formation of the group judgments could not be registered
explicitly. Furthermore, the participants were asked to specify their degree of
expertise for each question on a record. They noted their personal estimates for
each question before any discussion took place. A discussion of the problem was
expected to follow and a unanimous group estimate was demanded. A discus
sion leader was not appointed.
8
DURCH KREDITINSTITUTE IN DEN JAHREN 1967
Performance of Forecasting Groups
I
I
3.3 The Questions
i
::
Fhe fact-finding questions and the forecasting questions refer to finance, J
banking, stock quotations, and foreign trade. We could choose from a list of
ninety fact-finding questions and thirty forecasting questions that were kept on -J
a separate tape. The questions were chosen at random from this stock. Each |
question of the two different types had the same chance of being chosen. ■
However, no question appeared twice in a group.
IE.:
1.;.. Ua.ff
Evaluation:
Pci lot uiau of j’. j
CEuups
307
? _
however.47 I he following points can be considered as possible explanations for
•
the grraier length of time spent on the Delphi rounds: (1) Participants write
more slowly than they speak. (2) Communications between the experimenter
and (he participants takes place “sequentially," i.c., if participants j and k are
involved, j < A, the message of participant k to the experimenter or back can be
exchanged only after the same kind of message has been exchanged between
participant j and the experimenter. This pattern of sequential communication
is determined by the available technology. (3) Communication among the
participants and between the participants and the experimenter can take place
only during the periods of time in which the computer, which operates in a
time-sharing mode, is available for the job. Although the CPU was not busy
with batch operation during the experiments, the demand for memory space for
other jobs which were also initiated at remote terminals was noticeable during
the experiments. The participants considered such delays very disturbing,
b inally let us point out that since the available teletype terminals type more
slowly than the displays, preliminary experiments indicated that the former
were not suitable for the experiments.
The operating system allowed for the connection of 13 displays at one time.
I his determined a possible maximum of group size. However, as each partici
pant was supposed to join each type of group only once, we have limited
maximum size to 11. The minimum size of groups was determined by the
consideration that we wanted to have two clearly determinable participants
whose estimates lay outside the quartile values. This can be achieved with five
people. The fact that we had one man less in the discussion group did not
interfere with this principle.
All questions refer to items which are reported in the monthly statistics of the
German ledcral Bank (Deutsche Bundesbank), the daily slock market quota
tions of the Frankfurt Sloc k Exchange, and the market i< ports of the big banks.
< )nly very few of the fact-finding (jucstions icfcr to facts whic h .ire repot ted in
the foreign trade statistics.
In all cases the correct responses can be verified objectively at the time of the
experiments or at a later dale. In the opening lecture it was called to the
attention of the participants that, for example, the questions about certain past
or future interest rates did not refer to the rates of the respective local
institutions, which may depend largely on effects of local competition. Rather,
they refer to the rates which are listed as averages in the statistics of the Federal
Bank.
In five cases the wording of the questions was unclear to the participants.45
1’his resulted partly from an inexact formulation on our part and partly from
imperfect knowledge of the definitions as used by the Federal Bank on the part
of the participants. In the face-to-face discussions, clarifications could be made
immediately. In two cases in the Delphi groups, the “correct answer" in the
sense in which it was understood by the participants was carried on rather than
the answer to the original formulation of the question. Further cases of general
misunderstanding did not become evident.
In a final discussion many of the participants expressed the feeling that the
fact-finding questions were rather irrelevant and annoying: all the facts could
be looked up with no trouble. This point of view was not expressed with regard
to the forecasting questions. It should be recorded, however, that both sets of
questions refer to the same objects, although at different points in time. (Thus,
for example, one question asks for the price of a share of RWE common stocks
six months before the experiments and another for the same quotation six
months afterward).
4.
The Results^
1
i
I
I
¥
'F
-1•I
4.1 The Distribution of Expertise
3.4 The Volume of the Tests
It was originally planned that each group of different size and organization
should be asked to give ten forecasts and to answer ten fact-finding questions.
1 his plan was carried out with one exception. The Delphi group with eleven
participants was able to handle only eight questions of each kind.
1 he time spent on the discussions amounts to between 140 and 200 minutes.
In the Delphi rounds between 200 and 240 minutes of connect time were spent
per participant.46 (This length of time does not correspond to the CPU time,
15On the significance of the wording of questions and its influence on the level of estimates, cf. J.
R. Salancik, W. Wenger, and E. Helfer, "‘rhe Construction of Delphi Event Statements,”
Technological Forecasting and Social Change 3 No. 1 (1971), pp. 65-73.
46With 20 questions, that is 10 to 12 minutes; with 16 questions, 12 to 15 minutes. Lipinski,
Lipinski, and Randolph, op. at., report 15 minutes per question, on comparable hardware.
I
We want to investigate whether the expertise of the group members is distri
buted evenly or unevenly in the groups. It can be assumed that the distribution
is even, because the participants and the questions were assigned to the groups
at random.
However, the preliminary question whether expertise is rated differently with
regard to fact-finding questions and forecasts must be clarified. Only if the
-1
I
I
I
47D. Kaerger will contribute to the further analysis. See also, Institute for the Future,
"Development of a Computer-Based System to Improve Interaction among Experts,” First Annual
Report to the National Service Foundation, 8/1/73, p. 6, Table 2. The relationship of CPU time
to connect time varies from 1:110 to 1:135.
I he tests quoted in the following are described, e.g., by S. Siegel, Non-Parametric Statistics for the
Behavioral Sciences, New York, and London, 1956; G. A. Lienert, Verteilungsfreie Methoden in der
Bioslatistik, Meisenheim a. Gian, 1962.
I
1
job
Klaus Brockhoff
Evaluation:
Performance of Forecasting Groups
hypothesis of an uneven distribution is rejected can a comparison between the
309
Table 2
Distribution of Expertise (%)
groups be made with aggregated data from fact-finding questions and forecasts.
1 herefore, vve first test ‘‘auxiliary hypothesis 1": expertise is rated differently in each
i>roup with regard to fact-finding questions and forecasts.
The comparison of the quartiles given in Table 1 reveals different results in
only five out of twenty-four cases. However, the narrow limits of a scale from 1
■w
41
4
■■3
!
Degree of Expertise
Type of Group
Group Size
1
~2
Face-to-Face
Discussion
Groups
4
44
32
13
5
7
21
38
34
6
1
8
30
44
19
4
10
45
33
15
6
3
1
5
39
32
16
9
4
7
15
35
37
9
4
9
29
28
25
13
5
11
39
30
18
9
4
to 5 does not allow this result to appear sufficiently reliable.49 We therefore
compare the differences between the cumulative relative frequencies of the
expertise ratings within each group for fact-finding questions and forecasts. The
Kolmogorov-Smirnov lest shows no significant difference of the distributions
on the 5 percent level.
Delphi
Groups
Table 1
Quartiles of Expertise in Fact-Finding Questions and Forecasting Questions
3
4
5
h
Quartiles
Lower
Quartile
I Type of
Group
Group
Size
Face-toFace Dis
cussion
Group
4
7
8
I lelphi
(iiotip
Upper
Quartile
Median
FactFactFactfinding Forecasting finding Forecasting finding Forecasting
questions questions questions questions questions questions
1
2
10
1
1
2
1
1
5
7
9
11
1
2
I
2
2
2
2
3
I
1
1
1
2
2
2
3
3
2
3
3
1
1
2
2
2
3
2
3
3
2
2
2
3
2
1
2
2
3
3
3
3
Within each type of group comparable results appear, as expected.50 The
distribution of expertise is in both cases indistinguishable between the smallest
and the largest groups; the distribution varies greatly between the group with
seven participants and the largest and smallest groups, respectively (cf. Table
3).
‘
I
Table 3
Maximum Absolute Difference between Cumulative Distributions
of the Relative Frequencies of the Degree of Expertise between
Pairs of Groups (in %)
luuc-lo-l’acc Discussion Groups
Group Size
7
8
4
23
Ii
I
Delphi Groups
10
Group Size
7
9
11
14
5
5
24
14
2
15
24
7
14
24
15
9
1 bus, auxiliary hypothesis 1 is rejected. We can use the entire set of data to
investigate auxiliary hypothesis 2. It says that between the groups no differences occur
in the relative frequencies with which the different scale values of expertise are chosen.
I able 2 shows the relative frequencies of the distributions of expertise.
7
8
12
I
I
5
1 his conjecture is based on general reflections and empirical results on the correct construction
of a scale. On the inferiority of the scale with five divisions compared with scales with more
graduations in estimates of decisionmaking groups, see G. Huber and A. Delbecq, ‘•Guidelines for
(combining the Judgments of Individual Group .Members in Decision Conferences,” manuscript,
riMS-meeting, Detroit, 1971, pp. 5ff. It could not be investigated whether these statements can be
applied to our results, because of the difference in the task and because American subjects may be
less familiar with a scale using five divisions than are Germans,
50A comparison with the groups with the same rank of size within the type of group shows no
significant differences at p > 5 percent, neither with the Kolmogorov-Smirnov Test nor with the
sign test. The latter test was carried out in compliance with the possible
that ihg Safiibis
were intcrrelaied.
r
§
Kia.
kholf
An investigation of the observations for all groups of a given type leads us to
reject auxiliary hypothesis 2. If one were to formulate auxiliary hypothesis 2 for
a pairwise comparison between groups it would be refuted in all cases except
when the smallest group is compared with the largest group or the second
largest group respectively (Kolmogorov-Smirnov Test on the 5 percent level). A
consideration of the data in 1 able 2 now gains increased importance. The
significant differences arc due to the fact that in the middle-sized groups, and
particularly in the groups with seven participants, the ratings of expertise seem
higher than in the smallest or largest groups. It is obvious, and is not examined
more closely here, that the lower degrees of expertise are chosen much more
frequently than the higher degrees. Whether this is an illustration of the often
assumed pyramid of qualifications and abilities, or whether it only reflects a
fear of using the higher values on the scale, cannot be determined definitively.
The second assumption is supported by the observation that in the Delphi
groups, where greater anonymity is guaranteed. 9.6 percent of the self-ratings
fall into the categories four and five, whereas in the face-to-face discussion
groups, where the self-ratings were occasionally asked for directly by other
members of a group, only 5.5 percent fall into these categories. The results of
die next section contribute to the extension of these reflections.
1.2 Phe Significance of the Self-Ratings
A first indication as to the validity of hypothesis /: can be gained by observing
individual errors and self-ratings. We test whether the lowest individual errors
in each group and in relation to each question arc attained by those persons
who rate their expertise highest. Individual errors (see section 2.1.1) are taken
as absolute values. In the Delphi groups we can determine this error in every
round. We restrict ourselves here to the first and the last round. In the
iacc-to-face discussion groups the only data forjudging individual errors is from
(he questionnaires filled out before entering the discussion of each question. We
test auxiliary hypothesis 3:
The distributions of the highest self-ratings ivilh regard to each question and the
self-ratings of those who attain the highest level of performance (/.r., the lowest individual
error taken as an absolute value) coincide.
Auxiliary hypothesis 3 is tested separately for fact-finding questions and
forecasting questions in face-to-face discussion groups and Delphi groups. In
the latter case it is also tested separately for the first and the last rounds. In all
cases, auxiliary hypothesis 3 is refuted at a high level of significance in a \'2 test
(0.01). 'Phus, one must assume that in the situations investigated here, selfratings with regard to expertise do not give enough information as to which
persons actually possess expertise. For this reason either the ability to give a
Evaluation:
Performance of Forecabun^ Groups
I
311
self-rating which corresponds to actual expertise must be studied more closely
and, if possible, promoted or other methods of determining expertise before the
beginning of the questioning must be tested for their effectiveness. The Institute
for the Future emphasizes the latter problem51 in its recent studies.
Auxiliary hypothesis 3 was based on individual performance. Hypothesis Et
on the contrary, refers to the performance of the entire group. We attempt to
operationalize this viewpoint by testing the rank correlation52 between group
performance with respect to each question and the average expertise of the
group with respect to the same question, separately for each type of group, each
group size, and each of the two types of questions. The skew distribution of the
degrees of expertise already lets us expect that important information cannot be
gained from such a test. Indeed, we do not find any significant rank correlation
coefficient in the relationships tested for fact-finding questions. A classification
of the data in a 2X2 contingency table according to the criteria: low and high
degree of expertise vs. upper and lower half of the scale of the rank figures for
group performance, does not lead to significant relationships. With regard to
forecasts, significant relationships (at the 5 percent level) show up in Delphi
groups with five and nine participants in the third as well as in the fifth round.
However, as is shown in Table 6, these groups are not noted for particularly
good overall results. In this respect the correlation seems unimportant.
Since the conclusion that objectively existent expertise is not an essential
factor of individual or group performance contradicts the definition of expertise
and thus is not tolerable as an explanation of the results, it can only be
concluded that the self-rating of expertise by practitioners for tasks of the
/VI 111 their
UIVU WJJWllW
llOVi
present type does not coincide with
objective expertise.
?\fter the experiments were concluded, this unsatisfactory result led to tl e
question whether explanations can be found for the choice of the rank figures >fif
bje< s
expertise by the individuals. We attempt to explain the behavior of our subjet
by the following auxiliary hypothesis 4:
The degree of expertise with regard to a question is related to the number of years that a
subject spent in banking. Furthermore, it is higher whenever the subject matter of the
question coincides with one of the fields handled during the years in the profession.
0
I
I
I
I
hl
The necessary data were collected by questionnaire. Answers from up to
twenty-eight participants were available. In the analyses a high positive corre
lation showed up between age and number of years in the profession, so that
separate hypotheses for these two variables were not tested. Further, .• >ositive
correlation showed up between the number of fields one had experience in (a
list of possible fields was presented which could, however, be supplemented)
51Cf. Institute for the Future,"Development of a Computer-Based System to Improve Interaction
among Experts,” op. cit.
52Spearman rank correlation with correction for ties.
t
I
• ; .! ’J. ■ n-uiikr; of
in ti.'.1 j•:-T
■!>. Hi.k the r-s o e».mpuHcnis of
iiypothcsis 4 can no longer serve as mutually independent variables for explain
ing the degree of expertise. Therefore, we tested the hypothesis in a simplified
form, once for the influence of the number of years in the profession and once
for the influence of the fields of employment on the choice of the degree of
expertise. In neither case, a Kolmogorov-Smirnov I’esi refutes the hypothesis
(U.001 level).
Thus an observation that was gained in the face-to-face discussion groups is
confirmed. Once in a while during the discussions the number of years of
“banking experience" was brought into play to decide points of controversy,
i
jiU.'.iiHU.
Hypothesis R will be tested now lui judging performance. In order to
represent the performance of a group for a certain type of question, individual
I
performances must be aggregated. Since the individual performances are
“index numbers" only the geometric mean can be chosen for this. At first we
turn to the fact-finding questions. If considered individually, it becomes evident
that the lowest and the highest median group error (taken as an absolute
ic)
lie with approximately equal frequency in the first two rounds (cf. Fable 5). In
case of identical figures for the observed variable, its first appearance was
considered.
obviously with the view that this is a criterion for measuring expertise objec
tively. The same is true for the present field of employment of the persons in
question. Obviously, however, these criteria forjudging on the expertise are not
sufficient in the light of very special questions. It would probably be better to
consider, for example, the regular observation of special sections of the bank
Table 5
Relative Frequency of the Lowest (and Highest) Median Group Error
by Number of Rounds, Group Size, and Type of Question
Round (Fact-Finding Questions)
statistics.
(hoilp
Size
1
2
3
4
5
5
7
9
11
0.70(0.40)
0.40(0.70)
0.40(0.80)
0.635(0.40)
0.20(0.40)
0.30(0.20)
0.30(0.0)
0.125(0.20)
0.10(0.20)
0.10(0.0)
0.30(0.0)
0.125(0.10)
0.0(0.0)
0.10(0.10)
0.0(0.20)
0.0(0.10)
0.0(0.0)
0.10(0.0)
0.0(0.0)
0.125(0.0)
4.3 The Performance of the Delphi Groups
We formulated two hypotheses re;•garding the performance of the Delphi
groups. We first test hypothesis lz. To do so, we determine how frequently the
measure of variance for the last round is smaller than that for the first round
(cf. Table 4).
Hypothesis F cannot be refuted. When up to five rounds are carried out in
Delphi groups a reduction of variance of the estimates takes place.
A closer examination shows that it cannot be rejected that variance reduc
tion appears with equal frequency in all groups, and that variance reduction
occurs independent of the type of question (chi-square test, 5 percent level).
Table 4
Frequency of Variance Reduction in Delphi Groups,
Round Five compared with Round One
(^c of all possible cases)
I
’i:
I
Round (Forecasting Questions)
Group
Size
1
2
3
4
5
5
7
9
11
0.50(0.70)
0.70(0.60)
0.70(0.70)
0.375(0.865)
0.40(0.20)
0.10(0.20)
0.30(0.10)
0.50(0.0)
0.0(0.10)
0.0(0.20)
0.0(0.0)
0.0(0.125)
0.0(0.0)
0.10(0.0)
0.0(0.10)
0.0(0.0)
0.10(0.10)
0.10(0.0)
0.0(0.10)
0.125(0.0)
i-
The median values for each group, which are easily read from Table 5, all he
Group Size
Fact-Finding
Questions
Forecasting
Questions
5
7
9
11
100
90
90
100
80
80
100
100
in the first or second round. If we consider the lowest median group errors
(taken absolutely), w-e observe that round two evidently is of greater impor
tance concerning forecasting questions than concerning fact-finding questions,
whereas rounds one and three are of much greater importance for fact-finding
questions than for forecasting questions. On the other hand, the highest median
group errors (taken absolutely) are much more heavily concentrated in the first
round for forecasts than for fact-finding questions.
I
Klaus Brocklioff
Evaluation:
I his analysis docs not, however, take into consideration the degree to which
the medians deviate from the correct values. This may be evaluated by looking
at the geometric mean of the individual errors for each round, each group, and
each type of question. For it could be possible that good results deteriorate not
at all or only very little, while poor results improve greatly with an increasing
number of rounds.
Data to judge on this question are presented in Fable 6.
Performance of Forecasting Groups
I
If one assumes that people get bored with answering the same questions over
and over again and if one therefore cuts out the results of the last round, we
observe a minor increase in the test statistics. The calculated value for fact-
finding questions exceeds the significance level of 10 percent. For forecasts the
increase is so slight (x,2 = 5.77) that no further consequences can be drawn from
it. Taking everything together, our results seem to indicate that it is not
reasonable to extend the number of rounds in Delphi groups beyond the third
round.
This would support hypothesis R while it refutes hypothesis R'. Since in all
cases investigated it is not assumed that the self-rating of expertise varies with
'Fable 6
Geometric Mean of the Individual Errors
(taken as absolute values)
the number of rounds, the result cannot be tested further in this respect.
The observations of group performance are not supported by the results of
the best individual performance. In no case, i.e., neither
hen all rounds are
considered nor when the number of rounds taken into consideration is limited,
Round (Fact-Finding Questions)
Group Size
1
2
3
4
5
5
7
9
11
0.20
0.13
0.22
0.24
0.19
0.09
0.14
0.29
0.18
0.07
0.09
0.23
0.23
0.10
0.17
0.28
0.27
0.11
0.16
0.22
can a significant difference in the best individual performance be observed
which would vary with the number of rounds. The best individual performance
is very often maintained over consecutive rounds. So, if one would have
objective criteria by which one could pick real experts, one might expect a
I
greater stability of their judgments as compared with that of all members of our
present groups.
I•
4.4 The Size of the Groups
Round (Forecasting Questions)
Group Size
1
2
3
4
5
5
7
9
11
0,28
0.44
0.25
0.12
0.27
0.42
0.20
0.09
0.28
0.43
0.16
0.10
0.35
0.36
0.16
0.10
0.35
0.36
0.16
0.10
We test hypothesis G directly, separately for face-to-face discussion groups and
Delphi groups. We do not correct the hypothesis to include the distribution of
expertise (as determined subjectively by selbratings) between groups, as it has
practically a random influence.
When the data in 'Fable 6 are analyzed by the rows, significant differences
(on the 0.1 percent level) show up in Friedman’s analysis of variance by ranks
(X,2 = 12.84 for fact-finding questions and x2= Is f°r forecasting questions). It is
clear that the group performance in all rounds may be rank ordered
Data are analyzed by Friedman’s two-way analysis of variance by ranks.
A significant difference for the entire set of data for fact-finding questions
barely fails to be demonstrated at the 10 percent level (xr2 = 7.6 compared with
the tabulated value of 7.78). I he computed value of x,2 for forecasting questions
is considerably lower (5.75) and fails the 20 percent level. The phenomenon
that for fact-finding questions in all gioups except the one with eleven partici
pants the highest performance is attained iin the third round does not influence
the tests significantly. This is even less so for forecasting questions, where the
best performance is achieved twice in the second round and otherwise in the
fourth or fifth rounds.
315
Fact-finding questions:
Forecasting questions:
follows:
Group with seven participants on top, followed by the
groups with nine, five, and eleven participants.
Group with eleven participants on top, followed by
the groups with nine, five, and seven participants.
The groups of seven and eleven reverse their rank order of performance in
fact-finding questions as compared with forecasting questions.
This observation cannot be explained by varying self-ratings of expertise, as
can be read from the refutation of the hypothesis concerning a different
distribution of expertise in fact-finding questions and forecasting questions (see
section 4.1).
A
Klaus Brockhoff
Evaluation:
Performance of Forecasting Groups
317
If one considers the best individual perfortnances directly, the result for the
gloups is conhrrned, except for a shift in the rank orders of the groups with five
and eleven parttetpants for fact-finding questions, and of the groups with seven
and hve participants for forecasting questions. The individual estimates of the
pa.ticipants can thus be considered to be one factor which influences the result
1 he quality of the estimates could be determined bv the frequency of
Information exchange between the participants.1! The frequency of in ormation
however, that a significant result (5 percent level) can be identified only for the
group with nine participants.
Low, and in part negative, values for the rank correlation, particularly in the
group with eleven participants, can probably be explained by the fact that the
exchange is shown in Table 7.
opportunities for information exchange were used to transmit signs of im
patience toward the end of each session. These were not considered to contri
Fable 7
Frequency of Information Exchange between the Participants
bute to group performance in the stipulated sense, and thus were not counted
in selecting the data of Fable 7.
In the face-to-face discussions group performance is distributed differently
(see Table 8). For the fact-finding questions we discover the following:
in Delphi Groups
Absolute Number of
Information Exchanges
(iroup Si
I
After rounds
3
2
r>
7
9
11
7
10
43
26
13
33
17
')
the participants themselves, as keepers of information, channel varying
amounts of information into the group. If so, it should be possible to find a rank
correlation between group performance and the frequency of information
exchange with regard to each question within each group. It turns out,
I
11
17
II
% of All Possible
Exchanges of Information
After rounds
2
3
I
I
1>
17.5
25.0
53.8
40.7
15
20
1I
i
22.5
32.5
41.3
26.5
10.0
26.5
21.3
17.2
If performance is measured again by a median group error which is com
posed of individual estimates given before entering discussion, we find that the
group with seven participants attains the highest level of performance. The
groups with ten, four, and eight participants follow in that order. It should be
noted that these data .ire only approximately comparable to those for round
one in Fable 7 because the initial data used here arc given before the start of
•I
5.0
37.5
25.0
17.2
the exchange of information. If we take the geometric mean of the absolute
values of the differences between the group estimate and the correct values, the
gtoup with ten participants appears at the top of the scale of performance. The
groups with seven, four, and eight participants follow.
Let us now turn to the forecasting questions.
As before, a corresponding rank order of group performance cannot be
*,"rtr-.’ uwiowmi
We have aggregated data for fact-finding questions and forecasting questions
as we have found that the frequency of information exchange does not vary
s^nificantly with the type of question (binomial test, 5 percent level).
e find that die absolute frequency of information exchange between the
participants varies significantly between the individual groups (y2=19 8 is
sign, .cant on the 5 percent level). Nevertheless the group w.th nme partici
pant clearly leads the sequence, followed by those groups with eleven, seven
and five participants. If the frequency of information exchange is related to the
number of possibilities for information exchange (which depends on
Fact-Finding Questions
Forecasting Questions
the
num er o questions as well as on the number of participants outside of the
quart.l^, f course^ a significant difference between the groups can likewise be
eeimme (x, 15.0). In this case only the groups with seven and eleven
parncipants change their positions in the rank order as compared with the rank
Older of absolute frequencies of information exchange. One could assume that
of R^bilT®T^XUliOnS bLrT,S and ,ndividuals 10 3
ui nenaointy, journal of Educational Psychology 46 (1955), pp. 17-24.
Table 8
Geometric Mean of the Median of Individual Relative Errors
(taken as absolute values) before Discussion and the Relative
Error of the Group Estimate in Face-to-Face Discussion Groups
Problem a. Different Levels
Geometric Mean
Geometric Mean
Individual
Individual
Group Errors
Group Errors
Errors
Errors
Group Size
4
7
8
10
0.171
0.116
0.257
0.141
0.163
0.154
0.220
0.139
0.17 ■
0.103
0.072
0.187
0.184
0.147
0.121
0.212
■
i
I
J■1
Klaus BiockluHf
demonstrated for the two types of group structure. Here the group with eight
par ictpants leads, and the lowest level of performance is attained by the group
with ten participants (see Table 8).
y
5rouP
When these results are ir‘
. • nterpreted it should be noted that the fact that in
several cases one member of the Delphi groups did
1 not participate in the
ace-to-face discussion groups does not affect the results of
the ‘latter"i
— latter
-- in i a
uniform fashion.
Moreover, the direct observation of the
group activity suggests that the
number of members present in a discussion
group cannot be considered the
decisive variable. It is more i
important to consider the number of members of
Egroup
„up who actively take part in the ‘discussion,
:
onparticipants do not influence the group judgment in any wav. This situaO occurs in our face-to-face discussion groups. According to our 'observations
the group with eight participants, one member of the group only very rarely
took part m the discussions. In the group with ten members two persons d d not
express any op.mons during the entire experiment. When the group average
■Miparem"1^ th Y
^7 COnf°rmed to the majority opinion, which was
apparent. Ano her person only rarely joined in the consultations Thus the
at tne part of the group is almost always reduced to seven persons 54
... ' X.i"
'\° Sma"er gr°UpS ‘hC gr°Up itSelf recluired ,ha' each member
eX firLT« TUhCStlOn' ‘hTe the ■ aCtiVe” gr°UP corresP°ndS tn size with
the entire group. Thus our observanon material is reduced practically to
■ active chscussion groups with four, seven, and possibly eight participants
After these reservations as to the interpretation of the results it is'not
■urprismg that the rank order of performances, if partitioned with'respect to
■ act-finding questions and forecasting questions, as well as with respect to both
>pcs of group structures, does not generally exhibit comparable results in
groups of varying size. Results coincide in the
the smallest
smallest group only This
however, cannot be considered significant.
5
P
y'
I.') Hypothesis D
The problems of drawing a <
comparison between the results from the Delphi
gioups and the face-to-face discussion
groups have already been mentioned,
Besides, one has to find a
generally accepted criterion on which to base the
xr;, ;rcssi"ns oi ,h vau,hor a,ier “ch
no^xisi; ihcy would certainly have conlribulcd .^^'^Xuiio'n'of
du.urbTng iZna?di“u‘^ne aXhl' ab»l 7"S
^r. on a group
f- Wh°
dlSt'USS‘°nS
Evaluation:
Performance of Forecasting Groups
comparison. Dalkey’s statements are based
on the number of cases of superior
performance. However, the interest in group performance
can be based with at
least the same right on the amount of the
errors that occur. We take up this
latter point.
The geometric aveirage of' aHgroup
"
performances for fact-finding questions in
face-to-face discussion groups (0.167) is high
ler than the corresponding value for
the third rouind*'-•••
of Delphi groups, which is as low as 0.1 D . However, the result
is reversed for forecasts. Here "the
-----corresponding value of 0.209 for the third
round in I"
Delphi groups is h.gher than the result of 0.162 for the face-to-face
discussion groups. Thus
Derforman’
I t.
une<lulvocal rehtionship cannot be established. The
performance of the only group with identical size under different organiza
tional structures also differs greatly.
orgamza
It is further noteworthy that with regard to the forecasts, the discussion inlthe
to the forecasts,
the discussion
in I the
discussion groups shows no progress in performance
if the results
are compied
ormance if the results are compared
with the geometric means of individual errors b~r—
before discussion. On the other
hand the result of the discussion in all the Delphi
the error !
k. groups
xi_ is that
.1 , mean
is reduced up to the third round. Furthermore, we find that I
the performance
evel of the face-to-face discussion groups is approximately equal for fact■ nding questions and for forecasts, whereas the performance leve^of the Delphi
quesdons'1 '
■1
{|j
f°recas‘s is much lower 8S CQ«’pared with fact-finding
4.6 Hypothesis F
I he preceding statements have already made it clear that different icsults
are
efimtely produced when group performance with regard to f
fact-finding questions is differentiated from group performance with regard
I to forecasts (cf.
ables 6, 8). A general confirmation or
refutation of hypothesis F on the basis
Of group performance is not possible. We therefore formulate
and test the
auxiliary hypothesis 5 in addition to what we have found until
now. It says:
There exists a positive rei’lationship
’ ‘ ’ between the rank order of performance of individuals in
each group with regard to fact-finding questions
f and forecasts. Performance is defined as
the geometric mean of the individual errors (taken
as absolute values') in answering
fact-finding questions and forecasting questions.
r.
The auxiihary
“
hypothesis is refuted in each of the groups. With increasing
group size we calculate rank correlations of
-0.300, -0.391, +0.357 and
tTonsoJ the the7ajor‘ty ofkcases not even the sig" corresponds to the expectations oi the auxiliary hypothesis.
r
5. Summary
Although one should not overestimate the results from the very fe
:w experiments
presented here, they do lead to doubts as to f
the efficiency of the Delphi
method. Of course, it must be admitted that the
.e attempt to use the Delphi
■
Klaus Brockhoif
method for short-term forecasting is a comparatively tough test, for it was
originally designed for long-range forecasting. .At another occasion (sales esti
mates) it was also found that the errors of short-term forecasts can be very
much higher than those of long-term forecasts.,b If one assumes that the results
of the forecasts could be interpreted as an attempt to estimate an unknown
status quo at the lime when the experiments look place as well, then they
should be corrected by the average value of the relative difference between the
status quo and the realization of each topic. I hese corrections vary’ between
0.042 and 0.098 in the individual groups, according to the choice of questions.
I heir application docs not alter the rank order of the results as compiled in
1 ables 6 and 8. '1 herefore we may refer directly to the text with our summary:
(1) It cannot be discerned that fact-finding questions are suitable test
material for recognizing expertise or appropriate organizational structures for
forecasting groups.
(-)
general positive relationship between group size and group perfor
mance cannot be recognized.
(3) In face-to-face discussion groups the measure of the group size must be
determined by the number of active participants. Appropriate precautions
should be developed.
(4) Variance reduction almost always occurs in Delphi groups between the
first and the fifth rounds, but the best results are as a rule already known in the
third round. Further rounds may impair the results.
(5) Self-ratings of expertise show a positive relationship to the performance of
the persons questioned in only two of four Delphi groups. They tend to be
lower in face-to-face discussion groups than in the Delphi groups, and are
determined substantially by the extent of professional experience rather than
being set with regard to the questions in case. It is important to employ and
develop better methods for the determination of expertise.
(6) A direct comparison of Delphi groups and face-to-face discussion groups
was not possible because several participants dropped out. However, the results,
if separated for fact-finding questions and forecasts, do not point in one
direction.
(7) Proponents of the Delphi method will point out that our subjects, being
banking experts, are belter able to express themselves in a face-to-face discus
sion than on a display, even if its use has been explained to them. This should
apply particularly when the space for exchanging information among partici
pants is limited. The fear of making mistakes in the operation of the display
could lead to exaggerated caution. However, if one agrees with this argument,
the first point of this summary' has to be explained also, as the results are not
uniform in this respect. Anyhow, it appears to be important to avoid a “new ‘
56J. Berthel, D. Moews, Information and Planung in induslriellen Untemehmungen, Berlin, 1970, pp. 158
ff. (with data from fifteen firms).
Evaluation;
PcrforiHance ul t orccasiing Groups
321 n
Taylorism," on which some rzm-concepts are founded. However, it must be jp
granted that the originators of the Delphi method did not say that it has to be
operated as a computer dialogue.
(8) Only in the Delphi group with the greatest exchange of information did
we observe a positive relationship to group performance. The results indicate Jl
that in small Delphi groups more opportunities for information exchange E
should be given. However, it probably must be tested whether the information i\
given by the participants does coincide with what others would want to know, •
i.e., whether it adds to their knowledge.58 How can the “confusion effect” in the .
majority of the discussions, which is recognized when the “reference values”
mentioned there are compared (cf. Table 8), be explained without this distinc
tion and the assumption of a difference between information supplied and |
information demanded?
(9) li must Im- admitted that in our strongly discipline-oriented group there
has been relatively little opportunity for improving estimate by sharing infermation as compared to interdisciplinary groups concerned with other tasks.
However, this affects all our groups in the same way. This criticism would be.
valid if it could be demonstrated that different groups react differently
...........................
to these;,
two types of tasks.
7':J
■■ Il
_____________
... I
57See W. Kirsch, “Auf dem Wege zu einem neuen Taylorismus?” in H. R. Hansen, M. P. Wahl .
(eds.), Probleme ban Aufbau bdneblicher Infonnationssysteme, Muchen, 1973, pp. 338-48.
58E Witte (ed.), Das Informationsverhalten in Entscheidungsprozessen, Tubingen, 1972, especially, pp. :
44ff.; R. Bronner, E. Witte, B. R. Wossidlo, “Betriebswirtschaftliche Experimente zum Informa- |
tions Verhalten in Entscheidungsprozessen,” in E. Witte, op. cit., pp. 186ff.
i !|
: i
I.:.-,.
.<1 .M: o iu) Turuff
Hoheations in the 1^1.1
als()
Anoihcr
11T, is the Futures Gr
. up,
p ofit concern wdling to do Delphi studies on a
proprietary basis. While
•■•■e some of tts work ,s not available to the public, we have
included a list of its i
ccent efforts to mdteate the scope of application for Delphi
in this sort of environment.
In the next section we find
essentially all other articles on Delphi not
encompassed by these journals or
organizational headings. One can see from
ihis list the beginnings of university. ......... -V,
and governmental use of the
technique. Unfortunately, many users corporate,
haw
therefore, represented in this its.
not, t'
nO'
rCSUl'S a"d arfc
For a r
— * many years Delphi has been
great
associated with the subject of
I echnological Forecasting
,
.
T
c have provided a j
list of articles on
technological forecasting which do discuss Delphi. separate
Our caution
to the reader
about this set of references is that some of these
writings tend to define and
7,7lder DelPt?i.S0,ely "■'thin the scope of that one
application area. It has been
our intent in this
------ book
------ ; to present Delphi in the wide?
— context of an alternative
communication form.
1 here is evident in the literature
a set of new directions having to do with the
application of the Delphi concept in such areas as computerized cot '
'i areas as computerized conferencing
Systems, citizen feedback, j
' '
convenience we have placed (he.se reference's i
Appendix
591
i
Before
1970
Journal Articles
Rand Papers
14
34
11’1' and Futures Group
1970-1974
Total
54
68
48
4
67
13
45
3
61
80
140
44
64|
6
39
45
134
355
489
Related Work
136
45
181
Total
270
400
670
Other Delphi Articles
1 echnological Forecasting
New Directions
19
Foreign Language Articles
Iota! of Delphi References
95
25
A
other languages we
Delphi.
important to the
papers on Delphi
and asking what writings in other disciplines I.
have been referenced there, The
papers in this list represent work in the fields of (
J economics, operations research,
philosophy, planning, psychology, sociology, and statistics'
piovtdes a gutde to the techniques and knowledge in other ----- In essence, this
areas of utility to the
study and use of Delphi. As one would e— • •
expect, psychology and sociology make
up a large part of the referenced material*
cm for an update in a future edition of this book.
Distribution of Bibliography
s . . . . . .... —
P
Journal Articles
Alderson, R. C. el al., ’'Requirements Analysis, Need T
Forecasting, and Technology Planning Using
the Honeywell PATTERN Technique,” TFSC 3, No. 2 (1972).
mara, Roy C., “A Note on Cross-Impact Analysis.” FUTURES 4 N<
. Io. 3 (1972).
.and Andrew J. Lipinski, “Some Views on the
’ 'Use
’ of Expert Judgment,” TFSC 3, No. 3
(19 /2).
~T/«97^Gerald R SalanC*k’ “Forecasting: From Conjectural Art Toward Sciei
•ncc,” TFSC 3, No.
Ament, R., "Comparison of Delphi Forecasting Studies
in 1964 and 1969,” FUTURES 2, No. 1
(March 1970).
’’THl969)? SCaling CrilCrion of Qu“li°nnaire hems,” Soao-Economu Planning Sc^es 2, Nos. 2,
“^”^,^%^9;SEER: A
‘™Phic S'^ Examines Development in MediBjerrum, C. A., “Forecast of Computer Development
and Applications, 1968-2000,” FUTURES 1,
No. 4 (June 1969).
Rlack,n^n’ A- Wade, “A Cross-Impact Modei Applicable to
Forecasting for Long Range Planning, FFSC 5, No. 3 (1973).
- , ‘ 1 he Use of Bayesian Techniques in Delphi Forecasts,” TFSC 2, No 3/4 (1971)
................. ........
■sytfK
A
i
APP-ch Apphed m .nformation
App,“'h Appli“"
i
1
Appendix
1
^',^,d,',“ S'
'■d-
L' K Companies.” Lang Rang, Piann.ng 5,
C No"i XcMW108'31 FOreCa5,i0S in SiX
Dalltey, Norman C„ "An Elementary Cross-lmpact Model," TFSC 3, No. 3 (19721
Gr0,JP Opinion: The DelPhic Method," FUTURES 2. No. 3
(SepiemberX,^69)en,a
—, Analyses from a Group Opinion Study,'’ FUTURES 1 No G th ,■ »
ms-n
' 37viarBcM9n70a)nd S’
°f Sdf-Ralings
In>pruve Group Esdmates,” TFSC 1, No.
z1;r™3Fiannini; in “
'tuinm”
l'‘
K.
^n.
■Sunr' 1,1
A
Drvrlo|>in<-nt.
Pl&£UrA 1 h' wlphi M„clhod:,S“bslan“> Context, A Critique and An Annotated Bibliography »
Socio-Economic Planning Science 5, No. 1 (1971).
‘wnugrnpny,
Pyke Donald L., “A Practical Approach to Delphi,” FUTURES 3, No. 2 (June 1970)
An^yL”'Xr-OT£?2VNg,:3C<dnd
°' Random Numb'n in C^-Impact
analyses, t-u i UKES Z, No. 3 (September 1970).
Roland ''i'dc'°"TrhrOr>? Cr“s-’mP“‘ Matrices, and Pivotal Events,” TFSC 2, No. 1 (1970).
“, 2.
F‘™"g ind
t>,phi ^hnique-A Rep.y,”
Sar£SC 51No.4'u!97n3).a,iOn
S7l'>l”<7lldK7l'<i7l)l'r |,r,l''K'w
S‘ ('I'ydj1' p"’ 11 '
as a Means ,or Scicn,ific and 1 echnological
'•-mises in Applytng the Delpht
TFSC 3. So. 4J'(1972).
—,
S'ng ForeCastinS “ a »«ision-.Making Aid," FUTURES 2, No. 4
Cross-Impact Fechniques in Technology Assessment "Fl 'Tl 'RES 4 nt
~Ana;;^Fj^^ar(M^<,^An E,,eC,i-
i
l
V- FumS
' '“’d,"an'Jot1.’ "“phi and the Law ol Diminishing Returns," TFSC 2 No 2 F19701
Tilec' TOC"; No. 2AW2).a' ,he FU'Ure ,mPar' °' C°mPUl"-C—'cations on Everyday
^TUREs'i,
Ma'n“S: An ,llUS,ra,‘On
Policy Analysis,”
‘‘Potential Changes in Employee Benefits." FUTURES 2. No 2 (1970)
Cmt
S-a> po«cy
3X^)9™; KXPeri,”Cm in SimUla,i°"
Method of Forecast.
htg ”lhC
8se^:^^e!:;a^^rT
Crablx-, Eugene M„ and Donald 1 Pvke ‘
7
Processing Technology and Applications,"’ JsC 4X 2 pX*
J.
•' ^8
I
•t
ol Delphi Event Statements," TFSC
Pl-1""'"«: IWining Sulfi.ient Futures," FUTURES 3, No. 4
■,clpht: A Regional Planning Application," TFSC 3, No. 4
Envir°nm'nt-A DdPhi p—<"
Rangl Planning 5, No. 4
MmPt,'LTentS|tc “Pb'/Cross-Impact." FUTURES 5, No. 3 (June 1973).
TFSC i No I y^e 1969a) PP
l°
Integralion °f Technological and Social Foreeasts,”
~k?^p\?rntrWl%n9le<:hnO‘OgiCal and SOCia‘ ChanSeS: A FOreCaSting MwielFUTURES
in thc i99°'s: Expcrience with a
i
4
p—”
Iur°fC Murray, “An Alternative Approach to Cross-Impact Analysis,” TFSC 3, No. 3 H972).
(1970) CPhl Conferencing: Coniputer-Based Conferencing with Anonymity,” TFSC 3, No. 2
------ ’ “The Design of a Po,icy Delphi,” TFSC 2, No. 2 (1970).
.“Meeting of the Council on Cybernetic Stability: A Scenario,” TFSC 4, No. 2 (1972)
No^ig^y1’ UIS Greater Citizcn Part‘cipation in Planning Possible and Desirable?” TFSC 4,
J
Welty, Gordon, “Plato and Delphi,” FUTURES 5, No 3 (June 1973)
pL^r2ONo'T(M‘^^n and
of Technologica. Forecasts,” Lang Rang,
“d
6 (Decemlxr 1969).
,n,Ormalion
! '
"Cr“ss’Impacl Gaming," FUTURES 4. No. 2 (June 1972)
Huckfeldt. k aught, E„ and Robert C.Judd, "Issues in Large Scale Deiphi Studies," TFSC 6. No. !
IUXXOrkt.VXmr970)“raPU'a,iOnal ASPeC,S <>f C-‘mp-
S"(Decen^r' 1972).^
1
■i
1
A8gregated lnpUtS in,° DclPhi Forecasts: A Regression Analysis,”
and E"fn Helfer’ “Thc
El El RES 6. No. 2 (Apnl
............. ................. . ........ *...... N<1,
(December 1970)°^ ‘*
593
Attitude Questionnaire Items,”
^casting,"
X jX
97
1,ajZ''PerS°nnel Adminis-'en in 1980. Adelphi Study," Long Raig‘, Ptann.ng 5 No. 2
^-““TlmOm' F? 'Tn' Log."°™ali,>-o' “'Ph' Estimates." TFSC 1, No. 4 (Spring 1970)
. The Opl.mtsm/Pessim.sm Consistency ol Delphi Panelists." TFSC 2 No 2 (1970)
oZr
’buL R 7,!”-P 0 ^P ^rna'eS'" 7TOC '• N0' 3 <°March
‘W
Hanning JLNo.’ 4 June™,''
f
Delphi “hnique," Long Rang,
1 ?972)Ktnb’ Analyt,cal Techniques in Government Science Policy,” FUTURES 4, No.
1 (March
Rand Papers in Chronological Order
Sackman H., Dflphi Assessment. Expert Opinion, Forecasting and Group Process, R-1283-PR, April 1974.
Brown, I homas, An Experiment in Probabilistic Forecasting, R-944-ARPA, 1972.
Dalkey, Norman C., An Elementary Cross-Impact Model, R-677-ARPA, May 1971.
—, and Bernice Brown, Comparison of Group Judgment Techniques 'with Short-Range Predictions and
Almanac Questions, R-678-ARPA, May 1971.
Schmidt, D. L., Creativity in Industrial Engineering, P-4601, March 1971.
ai/tey’ ,N°rrnan D',/.ndAnanid L’ Rourke’ ^f^i^lal Assessment of Delphi Procedures With Grout
Value Judgments, R-612-ARPA, February 1971.
Quade, E. S„ On the Limitations of Quantitative Analysis, P-4530, December 1970.
—U war’tJ iA” A Preli"li^ary /nilu"? ^to the Software Estimation Process, RM-6271-PR, August 1970.
Hi',
L^1S’,and D‘ Snyder, Measurement and Analysis of the Quality of Life: With Exploratory
Illustrations of Applications to Career and Transportation Choices, RM-6228-DOT, August 1970
t^uade, E. 3., An Extended Concept of "Model," P-4427, July 1970.
I
-
I :
.......... .. . ■ ‘
‘
■ -d M ut ( ay Turoff
‘
& f.
Appendix
595
.970,
",
1'970
1 ublications of the Institute for the Future*
/r
no™;„47Z ^“d^w.tCr. R7G2FARPA’Febru^ ,97°-
‘II1: Use °f Self Rating to
Improve Group Estimates, RM-6115-PR, November^eg '
" Structure of Enpermenls,
rnara, Roy C., A Framework for National Science Pvlic
Policy Analysis,” P-18 (published in IEEE
an^On5c on S>'stfms’ Man and Cybernetics, Vol. SMC-2 N<
2. No. 1, January 1972).
------ ’ ... he Soc,al Responsibilities,” P-16, April 1972.
——, ‘Toward a Framework for National Goals and
Policy Research,” P-20 (published in Poli9
Sciences, Vol. 3, 1972).
and ‘hr Jud^‘ P’^ • ' d>^enlal Decision Esample. P-3620,
——. and Andrew J. Lipinski, “Some Views on the 1’
yT ?n-,En*pCrt JudSment’” P-19 (published in
^Technological Forecasting aid Social Change, Vol. 3, No. 3, 1972)'.
,M182•September 1969. ’
Capability, P-3540, Sep'temltr
Jr” Eilabl,ihmfnt # « Cong-Range Planning
“ RM^FpRjte0^"- and N' Dalk<:y’ nr
PJukne ?969O" ‘hr Dmm
. and J. K. Salancik, “Forecasting from C
Conjectural Art Toward Science,” P-14 (published in
Technological Forecasting and Social Change, Vol. 3^ No 4 1972)'
'*
Dalkey, Norman C.. The Delph, Method An Eupen,nental Study of Coup Opinion,
RM-5888-PR, June
j969of Opinions of Experts,
Quade E. S and W. I. Boucher, Systems Analysis
and Policy Planning: Applications in Defense,
California, June 1968.
Dalkey^ Norman C., Experiments in Croup Prediction, P-3820, March
1968.
. Quality of Life, P-3805, March 1968
Baran Paul. The Future of ^sprint, 1970-2000, R-16, December 1971.
'J^f,ariese Competition in the Information Industry, P-17 May 1972
____ ’ : On
aiS™nar
Broad-Band Communications, WP-1, February 1970
7 ’
j /he mpact of the New Communications Media upon Social Values ” P 3
hr k
•
jLau. and
PrMms, Vol. 34, No.
Spring 1969)7repXtf
P’3 (pubhshed ,n
1971 "
' a' “ Dman,tf0T Two-Way Infomation Smica to th< Homi, 1970-1990, R .
Tpldal'n^ re^n5^'
Dd key, Norman C„ DELPHI, P-3704, October 1967.
P'M^ ■'
F~g
,n
7? .967
Februa"'1967.)
7“' 7/T*
#
■"
^llecual Climate, P-3576,
P-3529'
____ '
1 9 70 .
Ik
September 1971,
i
Pr^, Conoid an,
A,nsl‘ac' P^ram Planntng, WP-7, April 1970
ngard' Cons'dt’‘‘'‘o^ »n a Framework for Community Action PlanningfwP-9, June
g' rd, Raul de, and Olaf Helmer, Some Potential Societal Developments- 1970-2000 R 7 Anrll lQ7n
Eneer Selwyn, A Case Study Us.ng Forecasting as a Decision-MakinT^^ D^’r ^69
------ . Cross-Impact Techniques in Technology Assessment P-15
December 1969.
Tnefl WO a''</ CmS'lmpaC‘ TtCKni,pUS: A" Eff,a,ve CmMm f”
p^es Anafysis, WP-8,
19ft7-
Ju'’C 1966' (Transla-d -
Helmer, Olaf. Th, Use of th, Delphi Technique m Problems of Edueo.ional InnocaUons, P-3499, December
’ l'rdr'a'/S‘a“ Science Policy and Connecticut.- A Futures Research Workshop, R-24 October 1971
I
•
Md Mustnal Aspects of the Space Program, P-5, November 1969
____ ’
!fv,lot"n^‘s m Plastics and Competing Materials by 1985, R-17, January 1971
' Z''"
Housing by 1985, R-13, January 1971. J
October .966.
Experts, P-2986, SepTember8!964 '
'970-,9S5~
a:
Haydon, Brownlee, The Year 2000, P-3571, March 1967
' uader EOIs''77 f ‘Kt ET Tr D,,r,,n
P-35M- March l9e7-
Neis^tr R
F“'“" °f
December
D,v,hpmm‘ Ac“m p,a^1
‘7/
BT0Sfi“ts°f Tech^logical Progress. California. P-3643, August 1967
- . Alethodo^ of Societal Studies, California. P-3611, June 1967
------ , The Future of Science. P-3607, May 1967.
Reseller, N., The Future as an Object of Research. P-3593, April 1967
-
-,s" -
■
“ “ “"'
lh< R,habll"y of Estimates Obtained from a Consensus of
- "-I ‘
Gordon, Theodore J., and O.af Helmer. Report on a Long-Range Forecasting Study. P-2982, September
?•
]970
Helmer Olaf, Conie,gmc, of Expert Consensus Through Feedback, P-2973 September 1964
7 , et' OHi 77 lm A,“T
R
^-mber >964 ”
964'
r>gard, Issues and Opportunities in the State of Connecticut 1970-2000, R-8, March
I^’po^R^M^ch *973Freder,Ck D- LaZar’ Sorne Considfrat^ns Concerning Bankruptcy Reform, IFF
m.;;. “7Z yui'dr-
■'w... ..
1 ’,7; nTFr •: ■ a"‘l * IF 1
s"‘"‘ '■<".....bi o,......... ...... 777
■ n' ' -
Jrojeci Delphi Experiment, RM-727-PR, November 1951
1- 'hr
? ‘■!,;hoV L' Sk°Ss,ad’ and M A Schick. Th, P.edKl.0,1
. \ pr11 i y.
.... <>u oj ll<>mhutf> l<r<iuitfuients
J
imp\i / "‘nZVrS nl^7^'w'
Insurance SysL: Current Status
for Sw,al ChanRe bv 1935 and Their
^St
of Social and Technological Events, P-93,
^Reports—Formal documentation of Institute Studies
forking
Preliminary contributions to the Institute’s work for its sponsors
A^r-Indtvtdual contributions by Institute staff members to the professional literature
■i
1I
) 1 urofi
Gordon, Theodore J., A Study of Potential Chang.
;es m Employee Benefits, Volume 1: Summary
and
Conclusions, R-], April 1969.
------ - Study of Potential Changes in Employee Benefits. Volume IL National and International Patterns, R-2.
April 1969.
A Study of Potential Changes in Enploier Benefit Volume III Ddp/u Study, R-3. April
. - I Study nJ Potential Changes m EmpR.
m .mpmee Heuelih. I
/| Appnidurs h, the Delphi Studi. K 1,
Apul I')(>'».
Potential Institutional Arrangements of Organizations hivohed in i' "
the Exploitation of Remotely Sensed Earth
Resources Data P-6 (published as A1AA Paper No. 70-334. MarLh
•
1970).
. Some Possible Futures of American Religion. P-4. May 1970.
, The Current Methods of Futures Research. P-1 1 August 1971
‘
e/
s,mr
and
7^
ml
Robtn
and
BucMc'
Appendix
597 4
•
Sahr, Robert C., A Collation of Similar Delphi Forecasts, WP-5, April 1970.
J pR ’Tcheodore J- Gordon, and Neale Adams, On the Nature of Economic Losses ArisL fro,
Computer-Based Systems in the Next Fifteen Years, R-23, March 1972.
j S
,va!lce’
H al’’ Gr0Uf> Communicalion Through Computers, R-32 and R-33, July 1974.
I'
• Ison, Allx-rt, and Donna Wilson, Toward the InUilulmnaliealion of Change, WP-I I, August 1‘ 70,
Delphi Studies by the Futures Group
Since some of these studies
are not available for general public distribution, a short description i>
included of some of the items.
■'
US-
R.2i'n\pDri|'hVr"’™'
Busmess
turner, Olaf, Long-Range Forecasting—Roles and Methods. P-7, Mav 1970
*
------ , Multipurpose Planning Games, WP-17. December 1971
------ , On the Future State of the Union, R-27, May 1972.
, Political Analysis of the Future, P-1, August 1969
—, Report on the Future of the Future-State-of-the-Union Reports. R-14, October 1970
- hu’tu^March
#
™ Memorandum. Institute for the
Scptembef?969n,^/ °f Lon^Ran^ Forecasting Methods for Connecticut: A
Summary, I EE Report, R-5,
- . and Helen Helmer, Future Opportunities for Foundation P
Support. System
R-l 1. June
Institute for the Future, “Development of'a Computer-Based
to 11970.
.........
ImproveGJ-^^^X:
Interaction
Aug°u"f 19^^ HrSt AnnUal ReP°rU Nali°nal
F°-da-n. Gr^nt
''sZh^h'MarcThv?^'1 Assessment
A~n‘ of Mediated
ica.ion: A
Workshop
lohansen. Robert. Richard H. Miller. Jacques Vallee. -Group Communication f
‘
through
Icdia. Fundamental Choices and Social Effects.- Working Paper. March 1974'
~ Electronic
Kt annsh, Arnold, The Non-Proliferation Treaty al the Crossroads, P-2, October 1969
‘^z:;x;xoAu^
l.il.inski Andrew J
■J
o/a
Toward a Bramtwork for Communications Policy Plannine WP-19 December 1971
J
a
Simulation. Some DcfmUwns, WP-b, April 1970
’
, SI--IPOL. Ap/imdix to the Simulation Carnr .Manual. VVP-14 October 1970
197“' Raul dc Br'8ard. Sunulatwn. Futunsm, and the Liberal .-I rts College: .4 Case Study . WP-15, April
Prelmunary Assess.
Dona for the National Advisory Commission on Criminal Justice Standards and Goals; member,
of the Commission served as respondents. May eventually be published as part of the report
the Commission’s work. Time horizon; to 1980-1985
o me report on
■
Fe’bruaiy 1972°re
~pri|nd972.TaibOt'
Effid (FPE). Report 43-01-06, February 1972.
ordon' Th' Etleralure of Cross-Impact Analysis: A Survey, Report 41-24-01/2,
“ D'lt,h' S,Udy ,‘fFu‘m
Systems, Report 49-26-01,
I he first Delphi conducted entirely through interviews; also the first (to our knowledge) to have
as its goal the conceptual design of a physical system.
snowiedge) to have
ThrmW S"
InTrti’ Fo
Tlch’'0‘0Sml “dd S«<al Changes: A Delphi Study of Opportunities and
I hreats for an Industrial Firm, Report 76-35-02, January 1973.
soc.'ri anTicTh^™10^1’ h,ard-,echnolo8y corporation; the aim was to identify prospective
in cite who
hrT
f
5a
developments relevant to the diversions, determine their probability,
hapSn and dehne ^l™51 lnSlrUmentalfin makin8 lh="’ happen, assess their impact if they did
r„
i , dc,'n5P°1 C1CS appropriate for the company to take. 1 ..-..e horizon- to 1982 +
/?« n1Z1 t
01 A‘‘‘ne Pr“dMs-lfs Decisions That Count (A Case
Mody of R Cd D Methodology), Report 24-15-01, February 1972.
'
—, and J. Cohen, An Investigation of Future Technological and Social Developments and Their Implications
for Entry into the Produce Market, Report 66-36-01, October 1972.
implications
The third “interview Delphi’’; here the attempt was to determine whether, in view of possible
anges in technology and society generally, it made sense for a particular company to c^ider
becoming a produce supplier. Time horizon: to 1982.
company to constder
Re^ onanFo1ZrMeplr0’
IDB°Ucher’ Cha“™&“ ^d Opportunities in the Photographic Industry:
Keport on a Focused Planning Effort, Report 85-42-03, June 1973.
&
197^
1 ’ A DelPhl Study °fthe Potential of a New Communications System, Report 58-26-02, August
-
•i
■I
1
■
A
■
T'^S^I and
Societal Developments up<.n^O^l‘U.!ftfi^^
WP^ IpXr71J- G°rdOn’
7Wi
AffM
fOCUS W1 °n th™PtUa' d^"
a
V«r Several Decades.
-^Richard Rochberg, and Richard Feller, STAPOl. Simula,mn-Game .Manual. WP-13, March
Other Delphi Articles
^n“7oJApGrnri09n70and
5 |!
L’A"! MuriS’i C’ I” “Society’ Po,itics and Economic Development: A Quantit live
Approach, Johns Hopkins Press, Baltimore, 1967.
?arCy’ and °- He,mCr’ “The Educat*on Innovation Stu
Behavioral Scientist 10, No. 7 (1967).
' Am icon
it
I.
Appendix
inul. Bruce, A Study of Consensus on Psychological Factors Related to Recovery from Nuclear Attack 1 luman
Psychological Factors Related to F
fences Research. Inc.. HSR-RR-71/3 Office of
of Civil
Civil Defense,
Defense, Mav
May 1971
1971.
s^lm^STO
'rfth, iMpin MrtM. numan Scicnccs Research ,nc
,h' Snr",,'i and
a|-& 1
w-
racl. J. 1... and R. J Twery, "The Applicadon of ForecasUng Techniques
in Research and
I levelopmem I lanmng, Printed at AICHE rnrrtrng. November 1969
9
and Wal,er Pricc' "Technological Forecasting," in Richard F Vancil fed f
l'a'‘"!nS,S^‘tms~IS7'' Harvard Business School, Cambridge Ma»
"‘he ‘"U,UrC
Commu'"cauons Services in the Home." PrU Canada,
11, Daniel “The Oracles at Delphi." The Public Interest. Fall 1965.
en er, A. D., A. E. Strack, G. W. Ebright. and C>. von Haunalter. .4 Delphic Study of the Future of
luhcine. Smith. Kline, and French Laboratories. January 1969
■ ghofer
E -CJenera! Education in Post-Secondary Non-University Institutions in Alberta”
" EdUCa,i<’n' N0' 9 (EdmOn'°n' Albena: Alber,a
•.XTltUApHM9W
P'^
Naval Supply
.ku. Harold, “Predicting Needs in Librarianship and Informat
ion Science Education,” Proceedmgs of the ASIA Annual Meeting, Vol. 7. 1970
■Ued
,he K"—"
IT •Ocmteri19re/OrCCaSlinS by he'alive Guess,ima*io"-”
«pi—
Experts,"
Er,g„lrrr.ng
.tplx'll Rolxn M„ "A Method,logical Study of the Utilization ol Experts in Business Forecastg.
h.D. dissertation. University of California. Los Angeles, California, 1966
and Dav,d Httchtn. "The Delphi Technique: Implementation in the Corporate Environ-
P0"
GradUa,C Scho<jl
Businras A^in^ion, U.
.toll. Marvin J., and Don Overly, "Toward a Consensus on the Future." Innoualm, No. 31, May
''eb.'uaryJ19C7l"
——, and Olaf Helmer, “A
An.........
Experimental Application of the Delphi Method
to the Use of
Experts,” Management Science 9, (1968).
Darling, Charles M., Ill, Jn Experimental 1968 Public Affairs
Forecast, Available from: National
Industrial Conference Board. New York, N. Y.
Davidson, Frank P., “Futures Research: A New Scientific
Discipline,” Proceedings of the Social
Statistics Section, American Statistical Association, 1969.
°f Conlpu,cr and Communication. Service.," AFIPS Conference
^We.rern R1
,h' Exl’,°ra,°V Dadof,mmt Project Evaluate Experiment, Ca«
Western Reserve Un.verstty School of Management. TM 165. November 1969.
Plonmnftf-'
,
DeV"’’ A
SMe! for Problem Identification and Program
llannmg, University of Wisconsin, Madison, October 1970.
8
Doak, Nick, ‘Future Rubber Processing," Rubber Journal, November 1972.
o>4e, Frank J and Daniel Z. Goodwill, “An Exploration of the Future in Educational Technology, Bell Canada, January 1971.
and Daniel Z. Goodwill, “An Exploration of the Future in Educational Technology,” in H
A. Stevenson R. M. Stamp, J. D. Wilson (eds.), The Best of Times, The Worst
Times—
Contemporary Issues in Canadian Education, Holt, Rinehart and Winston, Montreal, 1972 ?
^and Daniel Z. Goodwill, “An Exploration of the Future of Medical Technology,” Bell Canada,
AgCnCy for Ef,eclive planning," Hanard Burins Rmrw. January-
En/n/^^1Uy/^ “AppIica‘ionj of ^tutres R^arch to Society’s Problems,” in Nigel Hawkes (ed.),
International Seminar on Trends m Mathematical Modeling, Venice, December 13-18, 1971 (New York
Springer-Verlag, 1973), pp. 243-87.
V
Feldman, Philip A. “Technology Assessment of Computer-Assisted-Instruction,” Bell Canada,
Business 1 lanning, Montreal, August 1972 (internal document).
‘TmAV’ Martin_A7 an,?nL- S“Forecasting and Assessing the Impact of Artificial
genc/ s^nfoM
ProceedlngAs from International Joint Conference on Artificial Intclligence, Stanford Research Institute, August 1973,
FOfinn^ Phrel’ pA CritiC^ kook at the Ooss-Impact Method,” Research Report RR-5, Educalonal Policy Research Center, Syracuse University Research Corporation, August 1971.
7i^l 'Omputer and Educatlonal Futures Research,” The Potential of Educational Futures, Warren
Fulmer Robert M._ “Forecasting the Brave New World of 1984,” Manage, March 1971.
. Forecasting the Future," Managerial Planning, July/August 1972
,h'
’'P0" '°r "'e A,ncrican Society of Association Executives,
. "Technology A^ettsment by Cross-Impact,” paper presented a, the Engineering Foundation
..onjerence on Technology Assessment held at New England College. Henniker, N. H„ July
~ ’ On Forecasting the Future,” Inspection News, May 1971.
•
Glazier. Frederick P., “A Multi-Industry Experience in Forecasting Optimum Technological
Phen ITederick, and Walter I.. Gant. "The Delphi Technique," Joumai of Trackr. Education 21,
Foucaa
' P1'1 Jam M
pnS1ht and M- E' R Shoenian <eds-)’ A Guide
tactical Technological
forecasting, Prentice-Hall, Englewood Cliffs, N. J., 1973.
£
GOM,nS’iQF^"in’ A Technol°gicai Forecast in Transportation 1975-2000,” 39th ORSA meeting,
( IJ/v).
■un'ry 1971^ L’
ft
Pmlei H JF
Jh E W-T
for Alber^ Westrede Institute, Edmonton, Alberta, 1970.
R -n’e i ; ? t r J
\VllhamSOn’ T^^attve Methods for Eliciting Estimates for Health-Care
NovemU"^^9141
ReS°UrCeS’ Presented at Joint ORSA-TIMS meeting, Atlantic City,
...p.mm, K. L and F.N Cleaveland, Mrrtmg thr Xrrdr of Tomorro^t Publ.c Srrvic, Report of the
idiional Academy of Public Administration, January 1973.
H
uke, S. C. T., and H. T. Coutts, “The Future of Teacher Education,” u
npublished manuscript,
dinonion, Alberta: Faculty of Education. University of Alberta 1971
-er. J. Kenneth, "A Practical Method for Technologv Assessment." paper presented at the First
Ju"ie“"o730n,''rCnCt
Cd"‘Ol"Ky
i" The I lague. The Netherlands, May
‘K
•I
“f'ht F^'- n—
-,n
'" c'.' March 1969.
59!
Dalkey, Norman C., Studies in the Quality of Life (Delphi and Decision-making), Lexingu
:on Books,
1972.
. a X:
Part of NSSE Series on Con,cmfMra Education- chaH“
lcchnicluc: A Casc Study," PHI DELTA KAPPAN,
D- Z-. “An Exploration of the Future in Business Information,” Bell Canada, October
and Walter L. Gant, “The Delphi Technique: A Tool for collecting Opinions in Teacher
ducation, Joumai of Teacher Education 21, No. 3. (1970).
’’Xir
‘‘Pr.aClica’ Refinerne"ls 10 the Cross-Impact Matrix Technique," in Industrial
ophcations of Technological Forecasting, Cetron and Bartocha (eds.). Wiley-Interscience, New York,
I
GAu^stP1974nd MacReyn°,dS’ W" “°ptimal Urban Forms.” Journal of Regional Science 14, No. 2,
Gordon Theodore J., “Forecasters Turn to Delphi,” Futurist, February 1967
"Forecasting Policy Impacts,” in Walter A. Hahn and Kenneth F. Gordon (eds.), Assessing the
future ahd Policy Planning, Gordon and Breach Science Publishers, New York, 1973/
RrLhi Th Approaches to De,pb‘’” in Technological Forecasting for Industry and Government, James
Bright (ed.), Prentice-Hall, Englewood Cliffs, N. J., 1968.
J
r
I
ii
I
-Ur.I Miiif
.
i I.ik.hl S BC( j,(., • | |1(.
Reseatih Management, July 1'172.
■ -‘".I R I. I.rlilcu, T.n.pltn
601
L,i(
Thf
Marnm.
I,"<’ P'B'.- Hu.o.d Huune. Heueu. Jailuaiy
,
DelP^ Procedure for Long-Range Forecasting, IEEE, February
1967. ’JOSCPh 1 ’’ An Expfr’mgnl w,l/l
UnnZIv ",f w'se^.m
I Dlff(renCeS ln SubJf^ Likelihood
Estimates Made by Individuals, University
■ion
°f a ^^l Hemodialysis Popula-
M1 a11r'H Ii r.
F
.
rx
__
Bright
Englewood Cliffs, N J
.
1973
1
Futurist, October 1969.
‘O Res'arch Planning,” in James R.
l° PTachcal T^hnological Forecasting, Prentice-Hall,
’ ’
James Bright ^ed.). '"bo^on,A|n9^UStra"0"’
n,
"■ ■ "W«-» *■»
ion of Protein Ingredient:
Cornell 1'
Research. Center for Put
HelmOMPr'mp/\°'
S. C., June 1969 [ reprint in "hcAno/
°f Xianne Transportation Systems 1970-2000, U.
in the United sfates,” Vvihfngton^N^
Environmental and Political Implications
ln 'Plagued Fvrecactmg for Indudrg and Government,
of Government and
Hi h
K1 Iechniclue and Educational Innovation " ’
Inventing Education for the Future, W.
Hirsch (ed.), Chandler, 1967.
~7; ‘^ew Development in Early Forecasting on
Public Problems, a New Intellectual Climate,”
I ital Speeches 33, June 1967).
Social Technology, Basic Books. New York. 1966.
Research. Graduate School of’Businc^Adn^
Cen,er for Futures
—
°f T^S Fu,m.’ TRW C-P°™iom July 1966.
'SXrm
I
(unpublished).
"’g ‘n
ectronics,
Thompson Ramo Woodridge, Corporate Report, 1968
^ayJuneD19n6t
Pyl‘e'
°f
T-t'-l^Ral Future,” Harvard Busing Rlmtw,
Spectrum, January1"!^''
Tcchnologlcal forecasting in Planning f
Company Growth,” IEEE
University of Southern California, Los
Angeles, July 1973.
P^Xn, JFaku^l\S970hemiCal IndUStry “ the '98°'S~A DelPhi Method pr°«le,” CPemistr, aad
PSl‘lsa:,',,1J- (APri1 19®)-
liut-kfeldt VangX”""AF FoX^ot Chan' ■
Commission for Higher Education, Bo'ulde^ ColoradTl^T^ EdUCa,ion’" Wes,crn Interstate
Commission for Highe^Educatiom IWdJr. XadoXz1'’’’1 SlUdieS’" WeS,ern Interstate
Educational Policy^search" Xr "s^cuw UX
1969.
Appendix
Imp.,, i M.m.x Appio.i, i, ... I ... | „ |
.
I. I<> l<<|.
1 1
• < < liimlogy .\.ss< ;».siiieiitt ‘
lebruary 1970.
Lthelrh00rf
i Gfoi!
E*Pl°R "* Allernalives Ior lhe future,"
enter' ^acuse University Research Corporation, December
'''^
Educational Poi^^temc^X^SvmcZX^^r^
■^p.em^l^^'1" Wi‘h ‘he APPliCati°n Of lh' D'‘Ph*
P1Memor7andfm?NoM83^C^^^
CKem^ and
B.bliagraphj,, Technical
Pyke, D. L. d Practical Approach tt^Delphi
! °,f Mana8ement. M»y 1970.
^Institute of Chemical Engineers, Novlmter 1969
°
"S
Lons-Kan^ Planning, Ametican
'd
I1
D‘lfh,'
.'’lXSs,^Un^77 J!X7Aus]^9^ ,he ”Clphi T«t’"i<|ne.” Graduate Schoo! of
Journal of Marketing «««rrl 8,’ (buXlber’197u''ph‘
Marke,ing decision Making,"
- . "DHph^Me^hod:' Compmer°"edn-Oracl'eOn Wce/"^
University Business 53. No. 1 ( July 1972)
'
2’ No' 3' l97L
3 “ Consensus Formation,” College &
nclphi f;rO"S UP “ C°"^ Needs," CoHege and
Ex™and
RPr«^0/;A;^
Science Research,"
^OR,' C^ WesurnRLte UmveX^c™^^^
Rtie^' cti^
DaP‘-
ojsvc.ai Agenacc, Dept, of OR, Case Western
Rttde. Davtd, "Rubber Industry: A Ghmpse of the Future." Che,men, and Engineering Heutt, April
2000 A D””
^Tai:; and^X^^^^^
KXd, FXoXXr NXX'iX""
U' S Ar-
Research Institute. MenloXk. tZl^'A^^^ug^^ciiemT-'n'1”"
ComS'anf°rd
Per^E'
Trnn^mo:0/ M
Research, Grtduat Sch«l TfTu in^ Ad
rX
Angeles, April 1973
’=3 —
jBS “ -
'Tr*
and the Future,
Gemn^t, Summer
Cent, for Futures
Administration, University of Southern California, Los
-
“as
».i .< r™ o,£L"r££.“n"'fS- H'‘‘" a-‘-«—
1972.
J
I i.--
scot.', RVnda^rE.lnd'l^
• •' Mun ay i’ari.ffi
DATAMAnOH May l974
’ '
603
Productivity and the Delphi Technique.”
'
^,s^^F;“-^, care'Mph- -d<—-
.
Appendix
Sotit,y" 'Study), SET, Inc., 1970.
“o^M’amhXr
Hi-hi >-d" R-arch Report No.
S"-rl'
Shimmen, Torn, ”A Shor, Paper on Crosy-lmpac.
Analysis: A Basic Cross-Impact Model."
Institute for Future 7'echnology. Tokyo. 1973.
I ecling-Smith, J.. Medicine in the !990's. Office of Heihh I',-, n
i
i
,•
1969.
Health Economics. London, England. October
No°20 UunT^J116 'V°rld'S Future: Asim°'- D^Icd■"
-/Mr
’
^r^^Hg-” ^viet Cybernetics Review 3, No. 5 (May 1969)
, liindow into the Future, Mysl, Moscow, 1970
^rtsTKhnO'Og,eal
P™
Pemaquid
—<edE)VaFa*'n/8 S‘g,ni-S Ol Technolo8ical Change." Harvard Businas Rmsw 48 (l-'f))-l
C—
Engt^^
. Prentice-Hal),
JuE Aui?r9'7 L " "H°W ,O ChOOSe thC Righ’ Forecas,ing Technique.- Harvard Businas Raisu,.
. u.vvsuuug in a consumer Uoods
Vacation, 1 Issue 3 (1970).
Educational Policy Research Center, Syracuse’ Universes
—’ -rhe DCelnh FrCaSlin§ MZth0d'”
Dflla ^/-. January 1971.
“
r
Po a,ions•-in Th' 'T
, z°^ £1?
-~,Thr ^h. Mo^BarhgZdZ
The Potential of
coney Research Center, Syracuse
Po^y Research Center. Syracuse
University, June 1970.
Ttd^i^n^X.^"''’^ Drl'Sn and
(eds.), A Guide to Practical Technological
forecasting, Prentice-Hall. Englewood Cliffs, N. J. 1973
De Houghton. Charles, William Page, and Guy Slreatfeild,.
• ”
■and^Now the Future: A PEP Survey of
EhH 7 "Sum' M ^°?dShe?„529’ 37 (Lond0": PEP- AuP“' >971).
(1971).
C
eI
° US‘Ca 1 roblcms of Forecasting," Transactions, AS Estonian SSR, 20, No. 4
Girshic^M ArhK’ “,TcChn°lofic:' Forecasting,” The Journal of Busmas 44, No. 1 Oanua,
Ad^u±dW.Wngmm^C.°' 972e Dclphi T,Xhnit'Ue-" “"C
^na^T^ >n^.^^:;^^us"97r,enis;'38ih Scssion ot ,he■
1971).
f
’
£= £“ s" kr
^.^^roblems ol Select,ng Experts for Delphi Exercises." Aoadonis of Manage Journal 15,
w'..
A,Gm'
... < ’KSA. ! IMS. .
' ....
..... ;■■■pi....... . ..
»•' ||-
' "■
tie
“ “
uXt"/ NUrcte
07
’• "T* pi' inlx i
( >1
I)R'P“ of Medical Care, Johns Hopkins
1 echnological Forecasting
Edinburg University Press, Edinburgh, 1969
f--- -
..Crs
Sept’r^MSyO
'
'lOUrgC°,S
'
HMl'.hRn,adcaKprincemn. N.' Morgenstern^ Long Tenn Projections °of
f PM
“Ca‘ and Military Power,
Political
- ----------- ' •
' J', J“‘ruai y
1
I J,
Y'"’ David
natid M. (ed.). “The
Tl»e Futures Business"
article). C^,
Kiefer.
Business” (survey articled
Ch^tm! „ a v
xr
August 11, 1969.
' " ' article), Lhemical and Engineering News,
l-anlord H. \y .4 Syn,hosts »/ Toehnologtral Forocasttng .MothMugios (Wright-Patterson Air Force
iMay
........
■r—■>—■ A.r Force 'systdns CotnX’d,I a“
* ‘‘A".' ?. 1 ' J
-.....
197''
-.^tio Ep.dom.nlogy o/Broas, Canto,. Department of Medical Care. Johns Hopkins Univer-
F"aTjRi V
Y-.L, 1969.
Hayden. Spencer, "How Industry Is Using Techno................
□logical Forecasting,” Management Review, May
.
'
I
'
■ ’JS
|
°fA,f,hwh °f I'ulures Research, Report NORD-3 (Pretoria: Institute for
« m opinent South African Human Sciences Research Council 1973)
_artino Joseph ‘The Paradox of Forecasting,” The Futurist, February 1969
____ ’ . ^nnka F0T.eC^f°T
. D^ion ^ing, American Elsevier, New York, 1972
____ ’ ..whtT g*C F°r^aStl^g IS AHve and Wel1 in Indust«7.” The Futunst, August 1972
ZZ’ “T^ fC°?PU;erS ^ay Do Tomorrow,” The Futurist, October 1969.
, 1 ools for Looking Ahead,” IEEE Spectrum, October 1972
Np/Vt” L0ngFan^ Torecasting Methodology, A Symposium Held at
Holman AFB, Alamogordo,
New Mexico, October 11, 1967.
Massenet. M.. “Methods of Forecasting in the Social S’
- - Sciences, in Three Papers Translated from the
aOnXien^>
C~
Mass.; AmeHcan X^myT^
P=b-ary 1970.
MhroTian,'
Mythology?” IEEE spectrum, March 1973
Forecastlng and Assessment: Science and/or
■" R“ ■"d
NJanuaJ;r!969Q- “'r'Ch,,°l‘*ical F~"g - Hanning for Company Growth,” IEEE Sfootrum,
Mankind.” PMual .,ffalrs<
mM 12^No°4g('|C969EOreCaS"ng “ A'd ReSearch and Development Planning," Rosoarch ManagoGoiomrnent.
j
+'Cnt8’”
Technological Forecasting and
Doctora!
^van Statist
:
Ssfb
i
i
|
• .led Murtay l uroff
Appendix
and Donald L. Pyke, Technological Forecaslmg in Industry. NATO Defense Research Group
Conference, National Physical Laboratory. England, November 1968.
Pry, Robert, et al.. Technological Forecasting and Its Application to Engineering Materials. National
Materials Advisory Board, NMAB-279. March 1971.
national
Quinn, J. B "Technological Forecasting," Harvard Business Review, March-April 1967.
\osen, Stephen, "Inside the Future," Innovation Magazine. No. 18 1971
FOrCCaS'inB 'he 1'U'Ur; ‘S Losi"« “s
605
Part II (Der
Kand Edward Ward “Citi,
Kimbel D
mbcr ,972)'
' Sta.us,"
So^‘ologtcal Associa-
t t
^formation Systems " w
c;:'"’ndphic:on'"™"c^'Ba"
■A
1
Murray, Communication Procedures in Technological Forecasting," IEEE 73 INTERCON
ScSSIOn ?8: 1 ichnological Forecasting Methodologies. March 1973.
^ills, Gordon, et al.. 7echnological Forecasting. Penguin Books. Baltimore. 1972.
Lickhder, J. C. R„ “Th c
n°
Bulletin of Al ■ ? .
April 1968.
C^puter as a Communication D •
^Scientists,
Linstonc, H 4 "(->
Device, international
Research Needs inTmUnications
Futures Res.
u ..
Technology,
be published in W ^BoXh^^^11’1974aPer presented a' th. Confer
New Directions
Amara, Roy, and Jacques Vallee. "FORUM: A Computer-Based System to
Support Interaction
among People, IFIP Congress, 197-f Proceedings August 1974.
Bahm, Archie J., “Demo-Speci-ocracy," Policy Sciences 3. No. 1 (1979)
7umre“Marc\0iC9nOnferCnCin8 Arrang,:n,enl ,or a"
■"‘errogation,” Institute lor the
hX"m 72' d R ' °PmCnl' 'n,CrnatiOna' Co=
“ » X,XrrO^un,
gtiOn: A cations,
•
Problem, ant^I Po.ibkA^" H Randolph “Conle
- . Hubert M. Lipinski, Richard H. Miller, and Robert H. Randolph. “ARPA Policyormulation Interrogation Network. ’ Semiannual Technical Report April 1973
Haver, Raymond, -‘Societal Feedback," The Annals 2, September 1967
B.ll.ngsley Ray V - “System Simulation as an Interdisciplinary Interlace in Rural Development
sZ: TexUansP(A“Sei972r AgriCUllUra'
S'3110" l eChniCal
Carter George E„ “Computer-Based Community Communications." mimeographed, Cornputerdased Educat.on Research Laboratory, University ol Illinois. Urbana, Illinois (1973)
TVb, ■Syond Gcn«at‘o" Conferencing Systems," Computer-Based Educational Research
Laboratory, University of Illinois (November 1973).
Conrath David W “Teleconferencing: The Computer. Communication, and Organization”
' we^ of the First International Conference on Computer Communication (1972).
, and J. H. Bair “The Computer as an Interpersonal Communication Device: A Study of
Augmentanon Technology and its Apparent Impact on Organizational Communication,” Second International Conference on Computers and Communications, Stockholm, August 1974
Lnglebart, D. C., The Augmented Knowledge Workshop.” AF1PS Conference Proceed™; 42 1973
Joint Computer Conference.
—, “Coordinated Information Services lor a Discipline-or-Mission-Oriented Community," Second Annual Computer Communications Conference. San Jose. Calif. (January 1973).
, “Intellectual Implications of .Multi-Access Computer' Networks.” Interdisciplinary Con
ference on Multi-Access Computer Networks. Austin, Texas (April 1970).
•■iziom, Amitai, “Minerva: An Electronic Town Hall." Policy Sciences 3, No. 4 (1972)
tum^lom July
A .........
Gl197°ir)’
A reP°rt lu thc N-°-'
RlC<>’S CUlZeU Feedback Syslem' 1'^'hnical Report 59, OR Center, M. I. T. (April
Hall, T. W., ‘'^plementation, of an Interactive Conference System.” Spring Joint Computer
Conference, Vol. 38, AMPS Press, 1971.
J
f
Havron, M. Dean, and Anna E. Casey Stahrner, "Planning Research in Teleconference
Systems,”
Department of Communications. Canada. November 1973.
t‘^r^^3Steve ° U,,OnS’”
E-^nment: Some
o'
C,““n Fcedbacl‘ System- The p
Tr°PP'
System
S. ^Dr’ McKendree, “EMISARI p
August'wi "d 'nIerna"°naI Conference on
Miller, Richard H “Tr, a -
I
R1C° Model,”
Th,: R“our« Interruoti
m
omputers and Communications" ^On“orin8
BI-"-
„ ,b.
Renner, R(>d
' ro11 “cal Sciences, Univenhtv ofC Fr
** SociaJ
TM.225, Office Of p
,yO,Ca,,,Orn—
Cn f
™-21^ O«.'ce of Em
I
Pre^"-. AD
^plc'"com
Santi, V De et t n
Computer and Information S
?nd Invo,ve
Scaler
O{^e of Economic o
.
enCe)’ Member
Assisted^Instrucdon cRobert,Rohansen, ‘‘OR^^p^7’June l97l-
October 1972.
I
I*
Conferencing in a Co
ysteni,” International Conference
'P""r
'
"January 1971.
ealJacl<- New Technology for
fXXTco; Gr0UP
and Socia! Ch
•
^-ens, ChiZ^^ences.
^1PS
.____
January 1971.
Gmzen Feedback and Societal 9
^Paon. Gordon, “Moloch or Aflua • "
Conference on c^Ompu,eri2ed Conferencing
I!
’rence
'
Sckho,m- August 1974.
( 971),
M- I T
I
i
1
t
d International
^-national
“COnfere"^^^^
LEE, November 1972.
! i
1 utolf
S'An'^ICi"Lia Nc“''Orks'”
.January 1973.
39,' AFn>s Pre^97SI
len,ial lmpaC' °'' ln,urma'io1' -Sm" Conference
607
Proceedings FJCC
+ Computers + Communications = J” in / Industrial
‘
Applications of Technological Fore-
’
Appendix
paper presented at the Summer Schoo! of
G^^W^7sX^^' Phi-TeChni<’- APPfed » ‘Be Urbanization Process,"
x“
Melallurp'” Trend 4
' ..puma,n Commumcaiion,” EKISTICS 35. No. 21 I (June 1973)
Coinp^n^TiX:
S'S" '-'
Ihe Application of the DeIPh* Method in the Research of Iron
' ‘-'-„ce on
c—" ><-
in
FA- -
^N7t3S(A7iHP7^ Confcrc"d"8 ■" Sloping Coumries.- ltmM of
ndustnal and Business Organizations,”
Research Report, Prague, 1972.
USi^r;il^XjS^“/',''°"; ■'
‘h'
S^‘ Planning ERL,
""OlV,nl! ,hC ,,Ublit
Social P‘-"-8."
^aw, Judgmmis
l-utu^. Universily of’
Industry) journal,^7 IS^w'^k^mXeden^ Bui'derS’” Byggnadsindustrin (Building
andg.uhbrerLaat'GStu^enU^8;’
Al,fn,at,,,
— e of Graduate
'' En"X N:“ X^^ma^9^
Compuier-Based mforrnalion Sys.enV FJCC Conference
l'^"™5 3nd
iL^i^
B-d Com-
UCal,°n Research
-Based Education
oershy of Illinois. Urb™lP
U-
II
Application of, the Delphi Technique
to a EoXt’otthiTiXi^
sanstalt/The Swedish National Defense Res a k i
■-.den T. and L. O. Te.ma I • Xit“
O.8lcal !;^Penment>” Forsvarets Forskning-
Sallee. Jacques, "Network Conferencing." Datamation. Mav 1974.
(1970).
S"
■“ ......... ........ 7;;;;
'•Vlieox, Riiharil,
puu-ri/ed (I,,.,ununi...,i„„s 1 >i,,.s ..........
....... ar'd ‘<c^ Admimsnadon Conference (IRAC, Mav'19 [970 '
v.Xt„^-^PS
Wilson, Stan, "A Test of he Techni. , !
Laboratory, Texas A^f Li^tm
■ A ^Bmatton of Delphi and Cross-Impact Matrix Methods,” Trend 5
Mu^sXZ'.;?XUim0' ,he M~ °n th' HiS^ Of Civiliaadon,” Nordic
■
yS"'"'S'
'"’a‘
■" a Dynamic En-
?P, "
IEEE, October 1972.
C:°nSU1'inS Usi"’ API-'' AP‘-
Sornmr,u„l,m,M. No. 2 (I9G9)“ O.p^nhagw'Znmark Bt lPh' S'Udy'" '"^nnatim Fra
PeXrs!mpn\Blro"rhVOFkS AB' "?‘Phi 1985 ” O^’und. Sweden.
Platmanufakmr AB (pZX d caTliT 'Or Pe™orP ” P'">°rP. Sweden, 1969.
in the Beer Field.” M^md. Sweden
°
packa8in8)’ “Technological Forecasting
\atIUeS andfTheir Ganges in Sweden,” Malmo, Sweden
Research Institute for Construction Materials, " ‘ "
Some Foreign-Language Articles
AIM«h,od^’zfBn^
Mruk.ur.er.e Gruppenbefragung-Die Delphi-
»r:5SE?7sxlaTe^:in,heI
Th°e Royal"collepe nf
t-zechosiav...,
Sweden'. 1973^ °
°r“lry' "The FutUre R°lc of the Forest—A Delphi Study.” Stockholm,
Arising from Computer-B'ase^SystCms^n che^ew ' fF' <v" the,^ature of Economic Losses
Memo Park. Calif., Report R 23 Sch "972
XHas Copeo MCT AB. -The Driving o( Tunneis-l.Need.s and Technologies,- Naeka, Sweden,
'J7±orog>S T;ra73he(19??)Ca'iOn 01 ,he Dcl|jhl Mc'hod
"
InS'i,Ute 'Or the F“‘“re,
alS Betriebhches Prognose- and Plammgsverfahren,” ZfB,
'h‘ r— ling oi Machinery
Skandia AB (Insurance Company), ”The Internal Delphi-Study of Skandia," Stockholm, Sweden
“^^“eZ
The Swedish Board L ^chnLl rt
“°n "
“In“°n- ^mentation and
, A'lmann.fnForla8«' Stockholm, Sweden, 1971.
Sweden.” Stockholm, Sweden, 1 SeT “P"1'"'’
kdniL^™
s,ockho,m Schoo‘ °l
Ma-~ ^r .he
r,°nl,es for R“oarch in Medical Techniques in
°f COmpU,er USage and lB
- wedtsh Lloyd, ”The Future Transportation Environment,” Goteborg, Sweden, 1969.
I
• Hr I Ni'.ii : .r, Tuioff
1 he Swedish Newspaper Publishers Training Board. "Swedish Newspaper Industry—A Futuristic
Vision by the Press," Stockholm, Sweden.
1 he Swedish Savings Banks Association. “'The Future Environment of the Swedish Savings Banks."
Stockholm, Sweden.
The Swedish Shipbuilding Experimental Tank. "The Gaslurbine 'Technique in Merchant Ships"
Goteborg, Sweden, 1972.
Swedish Unilever AB. "Scandinavian Environmental Study." Stockholm, Sweden, 1971.
Ullman, A., "Research and Development in the Field of Materials.” Naval Material Department,
Stockholm, Sweden, 1971.
Volvo AB. "Delphi—An Experiment with a Futuristic Methodology'." Goteborg, Sweden, 1970.
Westerlund, S., "The Delphi-Technique—Swedish Practice," Ekonomen, No. 20 (1968), Stockholm,
Sweden.
Wikstrorn, S., " I he Distribution of Our Every Day Commodities—A Futures Study,” University of
Stockholm, Faculty of Business Administration, Bonniers Publishers. Stockholm, Sweden, 1973.
Ytongbolagen, Hallabrottet, “Product Innovations at Ytong,” Hallabrottet, Sweden, 1968.
Acknowledgment
The editors wish to thank Goran Axelsson and
J
------ Jan
Wisen of the Swedish Agency for .-Xdministralive Develop)iment and Ota Sulc of the Czechoslavak Academy of Sciences, Prague, for their
assistance in comipiling portions of this bibliography.
Related Work
•
Ackoff, Russell L., "Towards a Behavioral Theory of Communication," Management Sciences 4, No. 3
(1958).
------ , “Towards a System of Systems Concepts," Management Sciences 17, No. 1 1 (July 1971).
Adelson. M., el al., "Planning Education for the Future." American Behavioral Scientist, March 1967.
.Adorno, T. W., E. Frenkel-Brunswik. D. J. Levinson, and R. N. Sanford, "'The Authoritation
Personality, Part One," Wiley, New York. 1950.
Allen, Allen D., “Scientific Versus Judicial Fact Finding in the United States," IEEE Transactions on
Systems, Man Cd Cybernetics, SMC-2, No. 4. 1972.
Allen, Vernon L., "Situational Factors in Conformity." Advances tn Social Psychology, in Vol. II by
Berkowitz, Academic Press, New York, 1965.
Arrow, K. J., "Alternative Approaches to the Theory of Choice in Risk-Taking Situations,”
Ecunometnca 19. No. 4 (1951).
Asch, S. E., "Effects of Group Pressure upon the Modification and Distortion of Judgments,”
Readings in Social Psychology, Holt. Rinehart and Winston, New York. 1958.
------ , “Studies of Independence and Conformity: A Minority of one against a Unanimous
Majority," Psychological Monographs 70 (1956).
Back, K. W., "'Time Perspective and Social Attitudes," paper presented at APA annual meeting,
symposium on Human Time Structure, Chicago, September 1965.
Baier, K.., and N. Rescher, Values and the Future, Free Press, New York, 1969.
Bailey, Gerald, Peter Nordlic, and Frank Sistrunk, "Literature Review, Field Studies, and Working
Papers,” (from 1D.A Teleconferencing Studies), Human Sciences Research, October 1963,
Institute for Defense Analyses, Research Paper P-1 13. N FS - AD-480 695.
Bass, B. M., " Authoritatianism or Acquiescence?” Journal of Abnormal and Social Psychology 51. (1955),
pp. 616-23.
Appendix
609
Bavelas, Alex, “Teleconferencing: Background Information,” Institute for Defense Analyses Re
search Pajjcr P-106, 1963.
7 ’
Bell, D„ " Twelve Modes of Prediction,” in Julius Gould (ed.), Penguin Survey of the Social Sciences'
Penguin Books, Baltimore, 1965.
1
,
J
Belnap, Nuel, dn Analysis of Questions, System Development Corporation Technical Memorandum
1287/000/00, Santa Monica, Calif., 1963.
Berelson, Bernard, and Gary Steiner, Human Behavior, Harcourt and Brace, New York, 1964. j | '■
Berger, R. M., J. P. Guilford, and P. R. Christensen, “A Factor-Analytic Study of Planning
Abilities,” Psychological Monographs 71 (Whole No 435), 1957.
“
Beum, Gorlin D., and Everett G. Brundage, “A Method for Analyzing the Sociomatrix,” Socionutn ■
13, No. 2 (May 1950).
' .f
Bonier, R. J., A Study of the Relationship between Time Perspective and Open-Closed Belief Systems, NLS. t
thesis, Michigan State University Library, 1957.
, .{ •
Boocock, Saran S., and E. O. Shild, Simulation Games in Learning, Sa/< Publishing Co., Beverly Hills,
Calif., 1968.
jJii 4L
Bottenberg, R. A., and R. E. Christal, “Grouping Criteria—A Method which Retains Maximum
Predictive Efficiency,” Journal of Experimental Education 36, No. 4 (Summer 1968), pp. 28-34. pi
------ , and J. H. Ward, Jr.. Applied Multiple Linear Regression, PRL-TDR-63-6, AD-413 128, Lackland
AFB, Texas: Personnel Research Laboratory, Aerospace Medical Division, March 1963.
W -A
Bretz. Rudy, “The Selection of Appropriate Communication Media for Instruction: A Guide for
Designers of Air Force Technical Training Programs." Rand Report R-601-PR, February 1971,
pp. 30 ff.
Br221h32’ K ’ On DelCrmining Relativc Values," Zeitschnft fur Operations Research 16 (1972), pp. ..
............
Brown, Judson S., “Gradients of Approach and Avoidance Responses L..J
and TL„L
Their "J
Relation to Levels -■
of Motivations,” Journal of Comparative and Physiological Psychology 41, No. A
6 zioaoy
(1948).
... ,
Caldwell, C. H. Coombs, Schoeffler, and R. M. Thrall, “A Model for Evaluating the Output1 <4
Intelligence Systems,” Naval Research Logistics Quarterly 8 (1961).
I’
u
——, and R. M. Thrall, “Linear Model for Evaluating Complex Systems,” Naval Research Logistics
Quarterly 5 (1958).
|
Campbell, Donald T., and Julian C. Stanley, Experimental and Quasi-Experimental Designs for Research.
Rand McNally, Chicago, 1963.
f A
Cantril, H., “The Prediction of Social Events,” Journal of Abnormal Psychology 33 (1938), pp. 364-89.
------ , "The World in 1952: Some Predictions,” Journal of Abnormal and Social Psychology 38 (1943), pp.
6-47.
Carbonell, Jaime R., "On Man-Computer Interaction: A Model and Some Related Issues,” IEEE
Transactions on Systems Science and Cybernetics, January 1969.
Cavert, C. Edward, "Procedures for the Design of Mediated Instruction,” S ,
University of
Nebraska Project, 1972.
Chapanis, Alphonse, " I he Communication of Factual Information through Various Channels,”
Information Storage and Retrieval 9.
Christal, R. E., “JAN: A Technique for Analyzing Group Judgment,” Journal of Experimental
Education 36, No. 4 (Summer 1968), pp. 25-7.
------ , Officer Grade Requirements Project I. Overview, PRL-TDR-65-15, AD-622 806, Lackland AFB,
Texas: Personnel Research Laboratory, Aerospace Medical Division, September 1965.
- Selecting a Harem—And Other Applications of the Policy-Capturing Model,” Journal of •
Experimental Education 36, No. 4 (Summer 1968), pp. 35-41.
Churchman, C. W., Challenge to Reason, McGraw-Hill, New York, 1968.
------ , The Design of Inquiring Systems, Basic Books, New York, 1971.
------ , Theory of Experimental Inference, Macmillan, New York, 1948.
Clark, Charles H., Brainstorming, Doubleday, Garden City, N. Y., 1958.
Cohen, Arthur R., “Some Implications of Self-Esteem for Social Situations,” in Persuasion and •
Persuasibility, Hovland and Janis (eds.), Yale University Press, New Haven, 1959.
f
C-oleman, J., E. Katz, and H. Menzel, “The Diffusion of an Innovation,” Sociometry 20 (1957), pp. |
253-70.
Collaros, P. A., and L. R. Anderson, “Effect of Perceived Expertness upon Creativity of Members ‘
in Brainstorming Groups,” Journal of Applied Psycnology 53, No. 2 (1969).
"I
k
aiij Xiuriay i urull
Appendix
611
L °York *1964’ and H GuCtZk°W' A S°aa/ PsycholoSy °f GrouP Processes for Decision-Making, Wiley, New
No. 6'(!|969).y and D' L F°rd’ A S1Udy °f Subjec,ive Evaluation Models,” Behavwral Scima 14,
Dclbecq, Andre and Andrew Van de Ven. ".A Group Process Model for Problem Identification
and 1 rogram Planning, Journal of Applied Behavioral Science 7. No. 4 (1971).
Dickson, Paul, Think Tanks, Atheneum, New York, 1971,
Drucker, Peter, Technology, Management, and Society, McGraw-Hill. New York, 1970.
IJunnelle, M. D.. ••Are Meetings Any Good for Solving Problems?”, Personnel Adm,mslration, 1964.
Eckcnrode, R. F., Weighting Multiple Criteria." Management Science 12, No. 3 (1965).
Edwards. A. L., 1 echmcjues of Attitude Scale Construction. Appleton-Century-Crofts, New York, 195
1957.
T?
H e’ Thurstone’ “An Internal Consistency Check for Scale Values Determined by
' the
Method of Successive Intervals," Psychometnka 17. No. 2 (1952).
Emlet, Hany, et al., “Selection of Experts.” 39th ORS.A meeting. May 1971.
Eysenck, H. J "The Validity of Judgments as a Function of the Number of Judges,” Journal of
Experimental Psychology 25, No. 6 (19.39).
J
J
^'irwTn IncCha965and Barry M Rich'nan- Cornf,aral,Vf Management and Economic Progress, Richard
hshcr, Lloyd, “The Behavior of Bayesians in Bunches." The American Statistician, December 1972.
v^arner, W. R... Structure and Uncertainty as Psychological Concepts, Wiley, New York, 1962.
Gerardin, Lucien A "Topological Structural Systems Analysis: An Aid for Selecting Policy and
Actions for C.omplex Sociotechnological Systems,"” -paper presented
'
—
at.................
the IFAC/IFORS
Con(rimer on Systems Approaches to Developing Gountii les, Ak'ieis, Mas 1’173
Goidon, William, "( )|>riational Appi<»a< li to < .leativiiy, !latvanl /tn\inri\ /fri'irw
Tl, N<». «.
(Novrinlx-i Detembri I9.56).
Gullman, L. A., "A Basis foi Scaling Ouahtative Data," Ameruan S<.< tolognal Rei
•firir <), N<>. 2 (19I I).
Hamer, Raymond M., Sherman Kingsbury, and David Glcicher. Uncertainty in
Research, Management
and New Product Development, Reinhold. New York. 1967,
Hall, E. J., "Decisions, Decisions, Decisions.” Psychology Today, November 1971
Jane S Mouton, and Robert R. Blake, "Group Problem Solving Effectiveness under
Conditions of Pooling vs. Interaction." Journal of Social Psychology 59 (1963), pp. 147-57.
11 DC’ A’
Study °f lnleraclion and Consensus in Different Sized Groups.” Amencan Sociological
Review 17, No. 3 (1952).
5
, "Handbook of Small Group Research." Free Press, Glencoe, Illinois, 1962.
Harvey, O. J„ "Conceptual Systems and Attitude Change." in M. Sherif and C. Sherif (eds.)
Altitude, Ego Involvement and Change. Wiley, New York. 1967.
------ (ed.), Experience, Structure and Adaptability. Springer Publishing Co.. 1966.
“Sr,,em ■Struc,ure' Flexibility, and Creativity," in O. J. Harvey'(ed.), Esper,mcr, StrurWre, and
Adaptability, Springer Publishing Co., 1966.
—, D. E. Hunt, and H. M. Schroder, Conceptual Systems and Personality Organization, Wiley, New
I OFK) 1961.
——, and J A Kline, Some Situational and Cognitive Determinants of Role Playing; A Replication and
11967])” eCH RCP N° 15' C°nlra“ N°nr' (07)’ Universily of Colorado, 1965 (cited by Harvey
Havron, M. Dean, and Mike Averill, “Questionnaire and Plan for Survey of Teleconference Needs
among Government Managers" for the Socio-Economic Branch, Department of Communica
tions, Canada, Contract OGR2-0303, November 30. 1972.
Haythorn, William, "The Influence of Individual Members on the Characteristics of Small
Groups, Journal of A bnonnal and Social Psychology 48. No. 2 (1953).
Olat and N. Reseller. "On the Epistemology of the Inexact Sciences." Management Science
6, No. 1 (19o9).
5
Hoffman, L. R.. and G. G. Smith. "Some Factors Affecting the Behaviors of Members of Problem
Solving Groups,” Sociometry 23, No. 3 (1960).
P‘ J” "Thc ^aramorphic Representation of Clinical Judgments," Psychological Bulletin 47
(1960), pp. 116-31.
House, Robert J., "Merging Management and Behavioral Theory: The Interaction between Span
of Control and Group Size," Administrative Science Quarterly 14, No. 3 (September 1969). ,
George- anel /Xndre Delbecq, "Guidelines for Combining the Judgements of Individual
Members in Decision Conferences," Academy of Management Journal (June 1972).
~Tfi7n4Wsardr?Ps a,’d To
A" A"a'J‘SU °! En°,S Obu"n'd Wbm
Judgments, Paper
1 "®S°C,al S>;SpmS ReSCar"h‘ institute, Univetw of Wisconsin, Madison, 1967
raeli, N-, Abnormal Personality and Time, Science Press, New York, (1936).
, "Attitudes to the Decline of the West,” Journal of Social Psychology 4 (1933)
No. I (r|93P3)ESt'mateS Of lhe DiVOrCe Rate for the Years 1935-1975.”
of Social Psycholog), 4,
. I he Psychopathology of Time,” Psychological Review 39, No. 5 (1932).
~25,’ NSo°T(19S30)ClS °f
SOCial Psychology of Futurism." Journal of Abnormal and Social Psychology
u-nM^r°bablCnFutUrC EventS: A Study of Reactions
the Future,” Journal
°J Applied Psychology 16 (1932), pp. 584-88.
J
Janis, living L., "Groupthink," Psychology Today, November 1971.
Johnson, S. C.. "Hierarchial Clustering Schemes,” Psychometnka 32, No. 3 (1967).
JO|964el/FL|deb , L A[t dc la Fo,’jecture'” in «he Pu,unties Senes, edition du Roucher, Monaco,
- 4. (English translation: The Art of Conjecture, Basic Books, New York, 1967 )
oZnR^
A’ 1 Verpky’ “?UDbJeCtive Probability: A Judgment of Representativeness,” ' 4
Oregon Research Institute, Research Bulletin II, No. 2 (1971).
.•».
Kaplan, A.. A. Skogstad. and M. Girschick, “The Prediction of Social
and Technological Evcnti,”
1 ubhi Opinion (fiuirlerly I I, No. I (1950),
K.iMrnb.iuiii. R. ]., "A Pirliininary Study of the I hmrnsioiu of Future
.............. — • '••‘••e 'rime Pcnpective,”
Doctoial Dissertation, University of Southern Califoi nia. Ann Arbor, Mich., Univeniity Micro^'4
films, No. 60 394, 1960.
'
’
1
Katz E., "Communication Research and the Image of Society: Convergence of Two Traditions,”
I
of Society: Convergence of Two Traditions,”
American Journal of Sociology 65 (1960).
K
L
J/
‘‘ExPeriniental Studies of Group Problem Solving and Process,”
and Process,”
Handbook of Social Psychology 2, Addison-Wesley, Reading, Mass., 1954,
Kite Richard W., Paul C. Vity, “Teleconferencing: Effects of Communication Medium, Network *
and Distribution of Resources,” IDA Study S-233.
Kleinmuntz, B„ Formal Representation of Human Judgment, Wiley, New York, 1968.
Kotler. Philip, -A Guide to Gathering Expert Estimates: The Treatment of Unscientific Data,”
Business Horizons, October 1970.
Las.sey, William R., Leadership and Social Change, University Associates Publishers, Iowa City, 1971.
-eavitt, H. J
Some Effects of Certain Communication Patterns
Patterns on Group Performance,” Journal
oj Abnormal and Social Psychology 46, No. 1 (1951).
Eevinson D. J
Levinson,
J., "An Approach to the Theory and Measurement of Ethnocentric Ideology,” Journal
of Psychology 28 (1949), (cited by Rokeach [I960]).
5
J
Lichtenstem,
Lichtenstein. Sarah,
Sarah and J. Robert Newman, “Empirical Scaling of Common Verbal Phrases
Associated with Numerical Probabilities,” Psychometric Science 9, No. 10 (1967).
-ikert, R. A.. "A Technique for the Measurement of Attitudes,"
Attitudes." Archives of Psychology 22 (1932), pp.
{
-i!.
Einstone, H. A., ‘ On Discounting the Future,” Technological Forecasting and Social Change 4 (1973),
pp. 335-38.
, “Planning: Toy or Tool,” IEEE Spectrum 11, No. 4 (April 1974).
Lund F. H„ "The Psychology of Belief,” Journal of Abnonnal Psychology 20, No. 1 (1925)
Information Flow Model of Human Behavior’”
Madden J M "An Application to Job Evaluation of a Policy-Cap: ung Model for Analyzing
Individual and Group Judgment,” Journal of Industrial Psychology 2, No. 2 (1964), pp. 36-42.
adden. J M and M. J. Giorgia, “Identification of Job Requirement Factors by Use of
Simulated Jobs, Personnel Psychology 18, No. 3 (Autumn 1965), pp. 321-31.
nT'^Uu'iT'i’sG?)17” ASSC'S and Liabilities
GrouP Problem Solving,” Psychological Review 7t,
— , Problem Solving and Creativity in Individuals and Groups, Brooks/Cole Press, 1970.
lansfield. Edwin ”The Speed of Responses of Firms to New Technique^,” Quarterly Journal of
Economics 77 (May 1963).
y J
J
Martin, James, Design of Man-Computer Dialogues, Prentice-Hall, Englewood Cliffs, N. J., 1973.
i,
ij
-.if and Murray 'I'uroff
Maslow, A., ”The Further Reaches of Human Nature.” Viking Press, New York 1972
?969. R'Chard
A Dialcc,ical APPr'«ch ■“ Strategic Planning," Munagem^ Samcr, April
MSk,hl’966ePh E" and IrWin Ai,man' S"‘a"
R,'ea,ch' Ho11' Rinehart and Wins'°n- New
u9»).De,era,inan*s'he Predic,io” °'s<x;ial Ev"ns■"Joun,al alAbnonnal
Nlpd?/ AIqO1m "E'talua,i°n of Training in Creative Problem Solving," Joumal of AppM
Psycholog) 4J, No. 3 (June 1959).
J
J
"Influence of Brainstorming and Problem Sequence
uence in
in a Creative Problem Solving Test”
Journal of Applied Psychology 43. No. 6 ( Decernlx.-r 1959)
S
1959).
MVS'tr'
and Gerald E Rabid<;au•"
nndop^l, Wiley, New
I OiK, 1
*
NI(ei9a)) H ’ "Inn0Vati°n’ ln,egralion and Marginality.” American Sociological Review 25, No. 5
J^l’ge? W ’ and M K Starr’
StrUClU" 'Jjf //umnn I)ecii,,,n> Prentice-Hall, Englewood Cliffs, N.
Mitroff, Ian I., ”A Communication Model of Dialectrical Inquiring Systems-A Strategy for
strategic Hanning, Management Science 14. No. 10 (June 1971).
and Kre^Crick
“Dialectical Decision Theory: A Meta-Theory of Decision Making,”
Management Science 19, No. 1 (September 1972).
S
MaUp ?"d Hen? M 'Ik? ''ParticiPalion »nd Power Equalization," Organ,zalwnal Brhav.or
and Human Performance 5, No. 5 (1970).
N^r^>2 \an/d/ R’ i When7\Sr ' Ffa5,hll,ty Qf Anguishing Supervisors' Policies in Evaluation of
Z
by Using Ratings of Simulated Job Incumbents, PRL.TR.64-25, AD-GlO 812, Lackland
m
n’ *^aS: Hersonnel Research Laboratory. Aerospace Medical Division, October 1964.
Newell Allen and Herbert Simon. “Computer Simulation of Human Thinking,” Science, De
cember 22, 1961.
°
Nowakowska, Maria "Perception of Questions and Variability of Answers." Behav.aral Satnc' 18.
No. 2 (March 19/3).
N^en, Donald. Robert Mitchell, and Anthony Stout. Handbook of Staff Development and Human
Relations Training.. Institute for Applied Behavioral Sciences, National Education Association,
Washington, D. C., 1967.
Osborn, A. J-., Applied Imagination. Scribners. New York. 1957.
Peters William S and George Summers, Sial,Mui Analysis for Bus,n,ss Deas,ms. Prentice-Hall,
Englewood Cliffs, N. J, 1968.
1967) C ' and L R BCaCh’ Man 35 311 ,nluilivc Stalistician,” Psychological Bulletin 68, No. 1
Pettigrew, f. F., “The Measurement and Correlates of Category Width
as a Cognitive Variable,”
Journal of Personality 26. No. 4 (1958).
Pfeiffer, J. L., “Preliminary Draft Essays and Discussion Papers on a
Conceptual Approach to
Designing Simulation Gaming Exercises,” Technical Memorandum
I (preliminary draft),
Syracuse, N. Y.: Educational Policy Research Center, October 1968.
Pfeiffer, J. William, and John E. Jones, .4 Handbook of Structured Experiences
for Human Relations
Training, Vols. I, II, and III, University /Xssociates Press, Iowa City, 1972.
>IC/?C//’cJ\R ’^nc
Karl;n'“Reading Rat« and the Information Rate of a Human Channel.”
bell System Technical Journal 3b, No. 2 (1957).
Reid, Alex, “New Directions in Telecommunications Research.” a report prepared for the Sloan
Commission on Cable Communications, June 1971.
UThC InC*u,s‘on of the lability of Unforeseen Occurrences in Decision Analysis,”
itststs transactions on Engineering Management, Vol EM-14. Decemlx-r 1967.
R^erts, A., “Dogmatism and 'Time Perspective: Attitudes Concerning Certainty and Prediction of
the ruture, unpublished doctoral dissertation, University of Denver, 1959.
RmS/iQnmrmanr; 7?°SmatiSrn’ I ime
and Anomie,” Journal of Individual
Psychology 16 (1960), pp. 67-72.
Rogers, E. M., Diffusion of Innovations, Free Press. New York. 1962.
Appendix
61
Rokeach, M„ Beliefs, Attitudes, and Values, Jossey Bass Inc., San Francisco, 1968.
^o’/igSD^nn^Q ?‘Udying IndividuaI Differences in ‘Narrowmindedness,of Personal^
^S'ncw YotkC/7960A/'W’ InVeSl,gal,OnS lnt0 the Nature
Systems and Personality Systems, Bas
—, and B. Fruchter, “A Factorial Study of Dogmatism and Related Concepts,” Journal
Abnormal and Social Psychology 53, No. 3 (1956).
H
J
Mult'dmms,onal Scalmi- VoL 1 (Theory), Vol. II (Applications), Semin;
O" ReSearCh: ThC S,a,C 0'the Ar' in 1968/’ R‘^'ch
RU> E Nn 5
Sam UC!^n’P- A“Consumption Theory in Terms of Revealed Preference,” Economics 15 (1948
UniveXaTok)oa?950
At'emPtS “
Ecmm“
Hitotsubsf
Schroder, H. M M J. Driver, and S. Streufert, Human Informal,.on Proccssmg Individuals and Grov
Functioning in Complex Social Situations, Holt,
|
Holt, Rinehard
Rinehard and
and Winston,
Winston, 1967
1967.
lPar7
Nm
'1'e?^onian’ “Retention of Information Under Conditions Approaching'
Steady State,” Journal of Experimental Psychology 62 (1961).
i
Shenf,
?t
e|d,2M.,
V1” The Ps
Psychology^Social
ychol°gy of Social Norms, Harper Brothers, New York, 1936.
NewYoT^^O0'1^ DelbeCq’ and L* Cununings- Organizational Decision Making, McC
McGraw-HU^
I-^d,sj°ix^
,
I
1U1&, IJJU.
A Methodology for Determining Relevanc
in Complex Decision Making,” IEEE Transactions on Engineering Management, Vol. EM* 12, No. i
March 1965,
Souder, W. E., “The 'Validity of Subjective Probability of Success Forecasts by R&D Managen,’
IEEE Transactions oni Engineering Management, Vol. EM-16, No. 1, February 1969.
5
Steiner, Ivan D., Group
____Productivity,
________ Academic Press, New York, 1972.
p Pre
Process and
Stevens, S. S., “Ratio Scales of
°f Opinion,
°pinion’”” in D- K- Whilla (ed-)- Handbook of Measurement am
Assessment in Behavioral Sciences.
Stolber, Walter B., “The Objective Function
7 : l
r- j
e,
------------------- 1 in ProSram Budgeting, So
Basic Outlines?
d'eilschnft fur die Gesamle Staatsivissenschaft, 127 Band/Z Heft, May 1971.
laylor, C. W., Widening Horizons in Creativity, Wiley, New York, 1964.
——, and R. Ellision, “Biographical Predictors of Scientific Performance,” Science 155, No. 376£
(March 3, 1967).
Thomas, Hugh, “On the Feedback Regulation of Choice Behavior,” International Journal of ManMachine Studies 2 (1970).
J
rh19r59OnC’ L‘ L‘’ and E’ J’ Chave’ The Measurement of Attitude, University of Chicago Press, Chicago
Forgeson, W. G., Theory and Methods of Scaling, Wiley, New York, 1958.
TTiS?^n; A- M., “Monitoring and Storage of Irrelevant Message in Selective Attention,” Journal o,
Verbal Learning Behavior 3 (1964).
J
\
-—, “Selective Attention in Man,” Brain Medical Bulletin 20, No. 12 (1964)
Van de Ven Andrew, and Andri L. Delbecq, "The Comparative Effectiveness of Applied Group
Decision-Making Processes,” Academy of Management Journal (1974).
——, and Andre Dclbecq, “Nominal and Interacting Group Processes for Committee Decisioii
Making Effectiveness, Academy of Management Journal 14, No. 2 (1971).
rir Ge?!freyL1T/l? Arl
A Study in Policy .Making, Basic Books, New York, !965ij
107m Lvan H ’ Ihe Nature of Consciousness," Mathematical Bioscience 7, No. 1/2 (February
J ZU).
W allach, M A., and N. Kogan, “The Roles of Information, Discussion, Consensus in Group-Risk
laking, Experimental Social Psychology 1 (1965).
Ward J. H„ Jr., Hierarchical Grouping to Maximize Payoff, Lackland AFB, Texas: PenonnS
Laboratory, Wright Air Development Division, March 1961.
^Z^ttMarc"^), pp^ti? “ °bjKtiVe FunCtio"’”
'"I
■■
■
Warfield, John N., “An Assault on C.
Complexity,” alBattelle Monograph, No. 3, /\pril 1973.
Webb, E. J., and J. R. Salancik, The Inlm ieu ',
'» ihf Only
1,1 loivn. University of Texas i’ress.
Austin, 1966.
Weblx-r,
Acetal Contexts of Transportation and Conununualion. Working Paper No ■->•>()
.. ... ............ . .............. ................... .......... . ..
,:.la....
^Vtiiti^Ra^h^K5 “ 'Ihe
Computers on Organizations. Praeger. New York.
1970.
Ralph K
Selecttye Inattentton. Psychology Today. November 1971.
\Mison. A G
Research for Regional Planning.” Regional Studies 3 i 1969..
No "’(Sctber
C°nSenSUS
SubJ—
Dis.nbutions,” Ala„asemen, Snmr 15.
‘ 'l" .. .....‘I11' "I
'■-II-'
Reading.
.d
I
Subject Index
abuses of Delphi, 101, see also pitfalls
accuracy, 231-232, 241
accuracy measurement, 238-239
alternatives to Delphi, 60
anonymity, consequences of, 59, 545, 585
applications. 4 (list), 73-225. 158
appropriateness of, 4
business and industry, 76-78, 168-225
of contributory Delphi, 28
corporate. 168-226
cross-impact. 361-365, 370-375, 375-380
fields of, 11 (list)
goal formulation, 262-282
government planning, 75-76, 84-166
to information management, 364-365
on-line conferencing, 362-363
as organizing tool, 361-362
properties of, 4 (list)
recommended, 119 (table)
scenario-building, 466-477
specialized techniques, 383-487
see also examples, uses
appropriateness of Delphi. 4. 23
ARPANET. 501
Arrow’s theorem, 538
audience for results. 69
bias, 246-248
bias by time period, 164-165
biased responses, 160, 230, 231
caveats, 40, 93-94, 158, 223, see also suggestions
certainty, 243-246
characteristics 5, 8 (table), 103, 236
conventional, 5
real-time, 5
cluster analysis. 385, 390-391
comments, collecting and editing, 216
commitment, 59
committees, problems with. 86
committee system, 85
committee work, via computer conferencing,
555-557
communication
comparison of modes, 522
computer-aided, see computerized conferenc
ing
of Delphi results, 70
electronic. 517-534
group, see group communication
research, 520-527
of statistical results, 221 (fig.)
taxonomies, 528-532
types of, 8-9 (table)
vs. transportation, 510, 513-515
comparison of Delphi
to normal group modes, 7-9 (table)
to panel studies, 222-226
computer, as a communications medium, 499
|
computer-based learning, 550-562
computerized conferencing, 489-492, 497-515
advantages, 499-500, 502-503, 504-505
analysis of, 524 (fig.)
applications, 491 (list), 510-515
commercial application, 501-502
cost, 491, 503-504
and Delphi, 490-491
in educational system, 550-562
forecast, 550-562
future of, 509-515
impact on society, 514
origins, 505-509
possible misuses, 560-561
reactions to, 504-505, 508-509
utility, 491
versions available, 500-502
computerizations, time-sharing, 281-282
computer management of society, 563-569
computer misuse, 563-569
computer utilization, 302
computers and Delphi, 487-570
conditional probability estimates, 106-107
conflictual synthetic systems, 30
conference Delphi, defined, 5
confidence, 79
confidence scale, 91-92, 274-276
conflict management and diagnosis, 557-558
consensual systems, 21
consensus. 435, 466
artificial, 100
operationally measured, 108
as opinion stability. 277-281
consensus-oriented ’ -Iphi, 28, 463-486
to facilitate, 42
utility of, 23
consensus range, 105
consistency, 329-333, 470, 472
context, sources of, 65
conventional Delphi, defined, 5
convergence, 162-163, 167, 229
correlation between questions, 163-164
costs for a Delphi, 158
credibility, 160
criticisms, 5, 573-586
cross-impact analysis, 325-382
annotated bibliography, 365-368
applications, 361-365
Bayesian-based theory, 329-335
r»r. >1 •••
■
K*'
II
if
i L1
i
fla
■I
rl
I
I
’ii
■i
■i-i
fl
•
■‘W
Subject index
cot relation coefficient, 344-346
difference equation method, 346-348
discrete event, 325-368
early models, 327-328
event set, 339
example, 357-36!
information theory method. 353-357
KS1M, 369-382
likelihood measure method, 348 349
maximum information added, 351-353
non-Bayesian theory, 341-357
resolution of inconsistencies. 333-335
scenarios, 335-336
via simulation, 369-382
see also KSIM
deception, 585-586
definiteness, 241
Delphi
advantages, 158
as analysis tool, 100-101
applicability. 59
caveats, see caveats
characteristics, see characteristics
classical, 505
cost, 158
definition, 3, 38, 489
design considerations, see design considerations
as educational device, 100-101
effort involved. 213 (fig.)
effectiveness, 115-118
evaluation, see evaluation
examples, see examples
flow, 39 (fig.)
future for, 487-570, 496
goals, 96
graphically aided, 40, 403
as heterogenistic tool, 494-495
impact of monitor, 57
information flow, 214 (fig.)
interpretation of results, 70
manpower needs, 212
mathematically treated, see group estimation
modifications, 59, 60, 62, 103, see also cross
impact, computer conferencing, policy
Delphi
objectives, 125-126
panels, see panel
performance, 312-315
philosophy, see philosophy
pictorially aided, 435
pitfalls, see pitfalls
potential improvements, 35
problems, see problems
procedure, see study design
Subject Index
questionnaire design, see
questionnaire design
reasons for using, 1 14
refinements. 103
reliability, 116
results, see results
roles of, 76, 86
statements, 232-233
suggestions, see suggestions
underlying assumptions. 239-240
uses, see uses
empirical, 115, 118
of feedback, 270-272, 276-277
of impact of study, 159
by panelists, 115-118
performance, 291-327
reliability, 116
replicability, 115
summary, 320-321
see also validity
evaluation criterion for Delphi, 54
evaluation matrix, 105
example, 113 (fig.)
vs. face-to-face discussion, 291-321
design considerations, 64-70, 120-121
context. 65
communication of results, 70
interpretation and summation of
responses, 70
leadership roles, 545-546
orchestrating interaction, 69-70
panel creation, see panel, panel selection
questionnaire design, see
questionnaire design
stimulating response, (38—69
time, 65-67
4
design considerations, see also study design
design-monitor team, 93
effort required of, 213 (fig.)
desirability scale, 90, 136 (table), 472
dialectical inquiring system, 29-33
contrasted to adversary procedure, 32
see also Hegelian IS
Dialectical Policy Inquirer, 31-32
genetic counseling services, 100
glass, 171
Hegelian, 96-97
home computer uses, 175, 182
housing, 170
images of future, 436-439
Kantian, 95-96
land-use planning, 97
leisure and recreation, 169
market opportunities, 66-67
market research. 170
medical, 78—81, 174-175
metropolitan transit, 375-380
national priorities, 95-96
newsprint, 169
differential salience, 416-417
discounting the future, 574-578
DISCUSS, 500 501
dispersion, 229-230
dogmatism
effects on Delphi. 288-290
dominance judgement, 409
dyadic, 408
effectiveness of Delphi. 117 (table)
encouraging involvement, 115, 118
facilitating communication, 115, 118
electronic media communication, 517-534,
539-540
EMISARI, 500, 506-507
empirical science, 21
error measurement, median group error, 294
estimation
desirable features of, 240-248
multidimensional, 399-400, 402-431
group, see group estimation
individual, see individual estimation
evaluation. 89-90. 114-119. 159, 227-322
<omparativc. 116 (table)
event probability distribution, 469 (fig.),
475 (fig.)
evolution of Delphi, 10-11
examples
aircraft competition, 370-375
business, 174-175
chemical, 171
civil defense policy, 96-97
communications, 171
computer, 169
county government, 96
cross impact, 357-361
drug abuse, 124-159
education, 174-175
exploring human relationships, 52
forecast government employment, 99
I
I
!
personnel management, 1 70
pharmaceutical, 171
philosophical orientation of, 76, 77
planning, 81-83
plastics, 195-209
pollution sources. 111 (fig.)
quality of life, 387-401
reaction to retarded, 41
regional planning, 112-114
social change, 170
steel and ferroalloy, 210-226
telephone, 169
6
water rescurces imanagement, 102-123
see also uses, applications
experimental consensual systems, 21
expertise
effect on group performance, 295-297 •
1
illusory, 581-582
role of, 30, 84
vs. non-experts, 112, 144-145
(table)
extrapolative scenario, 469
J
face-to-face discussion
computer-aided, 502
vs. Delphi, 291-321
failure of Delphi, reasons for, 6
factor analysis, 396-398
fallacies, 584-585
'
'll
.i .
feasibility scale, 90-91, 135 (table), 472 5-- J
feedback, effects of, 270-272, 276-277
forced choice, 510
forecasting vs. fact-finding, 299-300
I
formal symbolic systems, 24
format of Delphi, see stud\ . sign
FORUM, 517
foundations of Delphi, 17-36
Lockean basis, 22
future prospects of Delphi, 191-192, 496, 49&570
fuzzy set theory, 478, 494
’ Ml
goals, defined, 263
goals, Delphi, 263-267
w
group
. communication,
—--- , —
535-549
_ _ .w
large widespread groups, 548-549
steps in feedback session, 540-544
using portable equipment, 546-547
’ jOS
via electronic media, 517-534, 539-540 '
group estimation, definition, 236
group estimation theory, 236-261
axiomatic approach, 257-259
error theory, 249-251
probabilistic approach, 251-257
score, 237-239
see also, estimation
group feedback session, 540-544
group ordering problems, 538
■
habitus mentalis, 4 8-49
Hegelian Inquiring System, 29-33
characteristic questions, 19
characteristics, 29
suitability, 31
heterogeneity, 494
heuristic process, 482
importance of, in forecasting, 581-582
history of Delphi 10
1
I
'- p!
■
Subject Index
lusty, 240
HYPER TUTOR, 501
idiomergent, 49-51
vs. industrial, 51 (tables), 54 (table)
importance scale, 91, 137 (table)
indices, sec scales
individual estimation, 240-247
INDSCAL. 414 426
inductive icprrsenlation. 21
uiltii malion
llou m Delphi, 214 (fig.)
from a dialectic viewpoint. 30
information content
of Leibnizian IS, 24
of Lockean IS, 21
■ n-house vs. consultants. 190
■ novation, 512
inquiring systems (IS). 19-35. 43-44. 566 567
definition, 21
differentiated, 18-19
goal orientation of. 29
Hegelian, 29-33
Kantian, 25-29
Leibnizian, 23-25
list of, 15, 19
Lock&an, 21-23
Merleau-Ponty. 43-44
Singerian. 33-35
mitt action matrix, 371 (fig.), 376 (fig.)
interaction modes. 57-64
affairs, 62
lumparison, 58 (table)
episodes, 60-61
events, 61-62
experiences, 59-60
occurrences. 63-64
transactions, 57-59
interquartile spread, 163 (fig.), 216
interrelated events. 106, see also cross
impact analysis
interval scale, importance of, 272
justifications of Delphi, 10
Kantian inquiry
examples of, 28
Kantian Inquiring System, 25-29
characteristic questions, 19
characteristics, 26
suitability, 29
KSIM, 369-382
computer output, 377-380
■ examples, 370-375. 375-380
interaction matrix, 371 (fig.), 376 (fig.)
interactions. 376-377
^nathematics. 372-373
Subject Index
619
policy testint/, 377-3110
vatiablcs. 370. 375
Law of Comparative Judgment, 273
learning curve, 469 (fig.)
Leibnizian Inquiring System, 23-25
characteristic questions. 19
characteristics, 23
suitabihiv, 25
i’
leiiri of invit.ition. *>3 ‘)|
clustering, 473-476
learning loops, 482-484
policy generation, 480-482
simulation. 481
utility, 470
normative systems, 464
NUCLEUS, 550, 558
numlx-r of rounds, 88, 94, 212, 229, 320
stopping criterion. 281
l iken m ,ile. 272
l.iHkr.in lli«|iill Illg Syslein, 21 23
basis of Delphi. 22
characteristic questions. 19
characteristics, 21
methodological example, 22
problems. 22
strengths. 23
suitability. 23
weaknesses. 23
long-range planning via computer confereucing, 555 -557
MAILBOX. 501-502. 507-508
management, aided by computer conferencing,
512-513
materials. 69
MDSCAL. 411
measure of polarization. 92
median group error (MOE). 294
Merleau-Ponty Inquiring System. 43-44
applications. 43
methodology. 262-287
misusing Delphi results in business. 189-190
models
of Delphi. 25
empirical. 21
inquiring system. 21
motivation. 69
multidimensional scaling, 385, 402-431
advantages. 414
annotated bibliography, 426-431
definition, 402
example. 410-414, 418-426
INDSCAL, 414-426
ordinary, 409-414
preference analysis, 422-423
two-way. 409- 4 14
uniqueness property. 419
use in Delphi. 403, 426
multi-model synthetic systems. 27
multiple studies. 190
normalization, sec multidimensional scaling
normative scenario. 469
normative system-building, 463-486 (fig. 465,
185.
I-
il
li
I
i
!
I
I
A;
objrc lives, defined, 263
opinion stability measure, 277-281
optimism-pessimism consistency, 230, 231, 584
ORACLE. 501, 550-562
ordinal scale, 272
overselling, 584-585
panel
compared to Delphi. 222-226 (fig. 224)
contributions required of, 213 (fig.)
creation. 68
mix, 68, 102
respondent’s objectives, 133 (table)
self-definition, 57
size, see panel size
subdivision, 103
panel selection, 94, 127, 210-211, 582-583
panel size, 86
effect on performance. 292-295, 315-318, 320
participation statistics, 130 (table)
participator)- democracy, 493
PARTY LINE. 500, 506
payment for Delphi by corporations, 187-188
payments. 69
perception. 414-426, 576-577
performance evaluation, 291-327
phases of Delphi, 5-6
philosophical modes, 18-35
philosophy. 15-71
pictorial aids. 435
pitfalls. 571-586
deception, 585-586
discounting the future, 574-578
illusory- expertise, 581-582
optimism-pessimism bias, 584
overselling, 584-585
prediction urge, 578
simplification urge, 579-581
sloppy execution, 582-584
PLA TO, 500-501
policy Delphi, 84-101
as precursor to committees, 86
examples. 95-100, 124-159
guidelines. 93-94
measure of polarization, 92
mechanics of, 87-95
participants, 88-89
phases of, 88
problems, 92. 100-101
rating scales, 89
role of, 86-87, 100
sample questions, 97-98
size, 86
uses, 94-95
|
I
T
/J
-If
I
•
policymaking, 463-464, 480-482
polic y question, 75, 84
prediction urge, 578
PROBE. 170
problem identification, 477-480
1
problems. 6, 22, 100-101, 189-190
biased responses, 160, 166-167
criticisms. 6-7, 573-586
dogmatic drive for conformity, 582
reasons for failure, 6
group ordering, 538
see also pitfalls
probability estimates
■:4
difficulty with, 107
guide for. 107-108
problem solving via computer conferencing,
510-511
proprietary problems, 191
1 ■1
■
I
I
>
I
1
quality of life, 495
difficulty in measuring, 582
factor analysis, 396-398
models, 388-390, 389 (fig.)
rankings. 392-393 (tables)
results. 391-396
question design. ' .■■
■ JI
JI
541, 543 (fig.)
questionnaire design, 93, 127-134, 172-174,
196. 198, 232-233
examples. 106 (fig.), 154-155, 198 (fig.), 200201 (fig.), 204 (fig.), 441-442, 448-462
mailing delays, 129
sample questions, 97, 162
time for completion, 129, 132
for trend extrapolation, 219 (fig.)
|
I
I
I
realism, 241-243
reality construction, 37-71
real-time Delphi, definition, 5
regression analysis vs. subjective judgment, 395396
reliability, 116
respondents, see panel
response distribution, 230
response rate, 132
results of Delphi, 40, 134-158, 437-439
create an audience for, 69
interpretation of, 70, 583
used as data, 188
' 'I
I •I
'I
I
1)2(1
>
y risk. analysis by Delphi, 7 78
roles of Delphi, 76
round one. 212 216
design, 212-214
information package. 104
results. 214-216
rounds, number of. 88. 94. 212. 229. 320
rounds, stopping criterion. 281
round three. 220-222
round two. 216-220
design. 216-219
feedback. 104, 105
results. 220
rule of the triangle. 330
scales. 89. 90-92 i table i. 1 -13/. '218 1 fiy .
262. 467, 47)
confidence. 91-92. 218 fig.
desirability. 90. 136 i table >
feasibility. 90-91, 135 (table i
importance. 91. 137 itablei
see also multi-dimensional scaling
scaling techniques, comparison. 272-273
scenario-building logic. 464-4 70
scenario-building technology. 470-477
scenarios. 335-336. 386. 469
definition. 470
examples. 550-561. 563-569
use of. in Delphi. 79-80
self-rating. 129. 131 (table). 233 234. 296
significance of. 310-312. 320
simplification urge. 579-581
Singerian Delphi. 35
Singerian Inquiring System. 33-35
characteristic question. 19
distinctive features. 34
main features. 33
potential for Delphi. 35
strengths, 35
weaknesses, 35
sloppiness ill Delphi. 582 581
so< i.il i lion <•. i <7
i Vi
MM ioiuetiv. 121
specialized techniques. 383 487.
als'. cross
impact analysis, policy Delphi, computer conferencing
statistical methods, re Lockcan IS. 22
statistical summaries. 109 (fig.). 221 ifig.i. 138.
444-447 (fig.)
stimulating response. 68 69
strengths of Delphi. 23. 586
.study design. 126-127. 172-174. 182 (table).
210. 390-391. 436 437. see
design
considerations
Study sophistication. I tit)
Subject Index
subjective probability. 580
suggestions (or Delphi, 64-70, 226
introduce ambiguities, 44—15
recommendations. 120-121
introduce uhat if items. 46
see n/s" caveats, design considerations
syncretic scenario. 464
synergistic thinking. 435
technology.- assessment. 29
teleconferencing. 517-520
teleological. 33
time. 129. 132
time needed by participants. 306-307
time domain. 65- 67
trend extrapolation. 237
trend extrapolation form. 219 (fig.)
truth content of Delphi. 24
types of Delphi. 5
conventional, 5
real time, 5
uncertainty. 229. 243-246
usefulness of Delphi. 54
uses of Delphi. 46-47. 51. 121. 125
administrative planning. 82
budget estimation. 89
combine opinions. 160
as educational process. 94
educational area. 82
elicit a hot list. 83
encourage participation. 46
establishing priorities. 161
estimating historical data. 78-79
examining the past. 82-83
filter out noise. 83
health-care planning. 81
investigate past performance. 95
probe insights. 47
problem identification, 81
ivgion.il planning. HI 82
t isk analvsis. 77 78
»'< <//(•« examples, apphi alions
using Delphi Jesuits in business. 188 189
utility. 538
validity. 18
as consensus. 22
comparison to real data. 79
Lockean. 21
philosophical positions. 19-20
see also evaluation
validity scale. 218 (fig.)
variance, as function of rounds, 299
variations in Delphi, 59. 60. 62
11
■
•yI
I
I
Ii
i
k
1jfl
1 -r
i
.■,]i
Position: 4641 (1 views)