Stability and agreement criteria for the termination of Delphi studies

doi:10.1016/0040-1625(79)90007-6

Technological Forecasting and Social Change

Volume 13, Issue 1, January 1979, Pages 83-90

https://doi.org/10.1016/0040-1625(79)90007-6 Get rights and content

Abstract

This paper addresses the question of the methodology of testing for stability between successive Delphi rounds and the necessity of doing so prior to the analysis of the level or type of consensus. Following a discussion of a hierarchical set of a criteria for determining the termination of such studies, the paper reviews some of the more common treatments of the subject in the literature. It is finally proposed that a χ² test be used to test the stability of responses between rounds. The method is demonstrated by a numerical example.

References (9)

J.Morley English et al.
The Prediction of Air Travel and Aircraft Technology to the Year 2000 Using the Delphi Method
Transport. Res.
(1976)
J.P. Martino
The Longnormality of Delphi Estimates
Technol. Forecast
(1970)
M.E. Schoeman et al.
Using the Delphi Method to Assess Community Health Needs
Technol. Forecast. Soc. Change
(1977)
J.S. Dajani et al.
Delphic Predictions and Cross-Impact Simulations
Am. Soc. Civ. Eng. J. Urb. Plann. Devel.
(1975)

There are more references available in the full text version of this article.

Cited by (223)

Identifying core measures to be used in mental health research with care experienced young people: A Delphi study
2024, Children and Youth Services Review
Care experienced young people, including those who are neurodivergent, experience worse mental health outcomes than their non-care counterparts. Understanding the mental health of this population, including how it changes over time, is critically important to the development of more effective, supportive services. However, one of the challenges to this is the lack of consensus around a set of suitable mental health measures that can be used with care experienced young people, including those who are neurodivergent.
The primary aim of this study was to develop consensus around a core set of measures that can be used by researchers when investigating the mental health of care experienced young people, including with those who are neurodivergent. Additional research questions included: (1) what are the strengths and limitations of these measures?; (2) what is the value and drawbacks of attempting to develop a set of core measures?; (3) what do young people with care experience think about the measures?
This study used a mixed methods approach. The primary method was a classical Delphi design where consensus from a panel of experts (n = 18) was sought in relation to the recommendation of 7 mental health measures. These measures covered 4 domains: general mental health, quality of life/related quality of life, anxiety and depression, suicide and self-harm. Alongside the Delphi, an advisory group of care experienced young people (n = 6) was asked to examine the suitability/ appropriateness of the collection of measures.
There were a number of strengths and limitations of the measures highlighted by the panel of experts. Given these, only the SDQ and the RCADS-25 were recommended for routine use with care experienced young people. Moreover, the panel recommended the use of the SDQ for care experienced young people who are neurodivergent. In addition, the advisory group suggested that there would need to be significant changes to the wording and content of all measures to improve their suitability. Lastly, a tension exists relating to the utility of recommending a core set and the potential restrictions this may impose.
Evaluation in design science: A framework to support project studies in the context of University Research Centres
2024, Evaluation and Program Planning
This paper proposes a discussion of the evaluation of an artefact developed under the Design Science paradigm using the Delphi method. It evaluates the Ecosystem framework of University Research Centres in Project Studies, considering a set of criteria pre-established in the literature. The Delphi method is an evaluation implemented in an electronic platform involving twenty-one participants, among whom were academics, practitioners, and PhD candidates in the field of project management. It reached consensus and stability in two rounds: the results indicate a consensus among the participants in the applicability, novelty, simplicity, completeness, fidelity to modelled phenomena, consistency and internal coherence, scalability, flexibility, interest, elegance, and reusability criteria. Usability was the only criterion that did not attain the predefined percentage of consensus among the participants (70%). Given the framework’s characteristics, Delphi participants indicated the need to produce complementary guidelines for its implementation.
Consensus guidelines, Delphi methods, and evidence around anaesthetic technique for endoscopic retrograde cholangiopancreatography
2023, British Journal of Anaesthesia
Consensus guidelines on the anaesthetic management of endoscopic retrograde cholangiopancreatography (ERCP) have recently been published. The rigorous synthesis of expert opinion is invaluable when there are limited data, and these guidelines are a significant step forward. This review both guides practice and identifies important research questions. We challenge those working in this field to collaborate and produce the evidence for whether monitored anaesthesia care (MAC) is associated with a lower incidence of adverse events and better outcomes than general anaesthesia for ERCP.
Assessing the effectiveness of potential protected areas and OECMs in conserving biodiversity against subsurface resource extraction impacts
2023, Biological Conservation
The direct and indirect impacts of resource extraction activities are an increasingly important planning and management challenge for organizations responsible for protected areas and other effective area-based conservation measures (OECMs). The development of subsurface resources in protected areas or OECMs can have significant implications for the ways in which such areas are established and their overall effectiveness in conserving biodiversity. In this article, we introduce a decision-support tool, developed through expert consensus and real-world application, that can be used to assist organizations to evaluate the effectiveness of conservation mechanisms (laws, regulations, and associated policies) and other management tools from potential subsurface resource extraction activities. The tool provides a novel and comprehensive method to determine whether a potential protected area or OECM is effective in managing all subsurface activities that may have an impact on biodiversity. While developed in Canada, the tool has universal applicability for conservation managers globally working to achieve standards of protection for biodiversity. We conclude by discussing the ways in which Parties to the Convention on Biological Diversity (CBD) can consider resource extraction rights and impacts in national efforts to effectively achieve the goals and targets of the recently adopted Kunming-Montreal Global Biodiversity Framework (GBF).
Formal evaluation of construction safety performance metrics and a case for a balanced approach
2023, Journal of Safety Research
Introduction: Measuring safety performance is crucial to making informed decisions that improve construction safety management. Traditional approaches to construction safety performance measurement primarily focus on injury and fatality rates, but researchers have recently proposed and tested alternative metrics such as safety leading indicators and safety climate assessments. Although researchers tend to extol the benefits of alternative metrics, they are studied in isolation and the potential weaknesses are rarely discussed, leaving a critical gap in knowledge. Method: To address this limitation, this study aimed to evaluate existing safety performance against a set of pre-determined criteria and explore how multiple metrics may be used together to optimize strengths and offset weaknesses. For a well-rounded evaluation, the study included three evidence-based assessment criteria (i.e., the extent to which the metric is predictive, objective, and valid) and three subjective criteria (i.e., the extent to which each metric is clear, functional, and important). The evidence-based criteria were evaluated using a structured review of existing empirical evidence in literature, while the subjective criteria were evaluated using expert opinion solicited through the Delphi method. Results: The results revealed that no construction safety performance measurement metric is strong in all evaluation criteria, but many weaknesses may be addressed through research and development. It was also demonstrated that combining multiple complementary metrics may result in a more complete evaluation of the safety systems because multiple metrics offset respective strengths and weaknesses. Practical Applications: The study provides a holistic understanding of construction safety measurement that may guide safety professionals in their selection of metrics and assist researchers who seek more reliable dependent variables for intervention testing and safety performance trending.
A framework for the design of pediatric healthcare environment using the Delphi technique
2023, Ain Shams Engineering Journal
Citation Excerpt :
The standard deviation (SD) is less than 1.5 [31]. According to Dajani et al. [32], the Coefficient of variation (CV) is another method of judging agreements. If the CV is in between 0 and 0.5, a good degree of consensus has been achieved and a second Delphi round is not needed; if the CV is in between 0.5 and 0.8, a consensus is less satisfactory and another round may be required; and if the CV is 0.8, it has a poor level of consensus [33].
By investigating specific therapeutic goals and design considerations, this paper aims to develop a framework for the design of a pediatric healthcare-built environment. Growing body of research indicates that the characteristics of pediatric healthcare facilities differ from those of adult healthcare facilities, therefore the findings of adult healthcare facility design cannot be generalized to pediatric healthcare facility design. Moreover, the studies specifically focusing on the built environment in pediatric healthcare are also sparse. Literature studies were conducted to extract therapeutic goals and design considerations. Two rounds of the Delphi technique were used to extract the Indian experts’ opinions, and to make a consensus among the experts on each goal and design considerations. SPPS software was used to calculate statistical output i.e., mean, standard deviation (SD), Interquartile range (IQR), coefficient of variation (CV), and boxplots. Results were drawn with the help of statistical data and boxplots. Finally, eight therapeutic goals and 33 design considerations emerged which may contribute towards designing pediatric healthcare environments. The design considerations will aid architects and facility planners in making design decisions for new or renovated pediatric healthcare facilities.

View all citing articles on Scopus

^∗: Jarir S. Dajani is Associate Professor, Program of Infrastructure Planning and Management, Department of Civil Engineering, Stanford University, Stanford, California.

View full text

Stability and agreement criteria for the termination of Delphi studies

Abstract

Transport. Res.

Technol. Forecast

Technol. Forecast. Soc. Change

Delphic Predictions and Cross-Impact Simulations

Am. Soc. Civ. Eng. J. Urb. Plann. Devel.