Summary
The purpose of the work is to offer a methodology for the application of the Delphi method in pedagogical investigations.
The author starts from the bibliographic analysis in relation to this topic and argues it with a practical example carried out in the researcher's own master's thesis, which allows a better understanding of the ideas presented.
In addition, it is conceived within a set of related topics in this regard, which will make up an electronic book about statistics applied to scientific-pedagogical research.
By using hyperlinks, the job articulates your solution with the Windows electronic Excel tab.
Summary
The work has as purpose to offer a methodology for the application of the method Delphi in the pedagogic investigations.
The author leaves of the bibliographical analysis in connection with this topic and he argues him with a practical example carried out in the master thesis of the own investigator, what facilitates a better understanding of the exposed ideas.
In addition, it is conceived inside a group of topics tune in this respect that will conform an electronic book about the statistic applied to the scientific-pedagogic investigation.
By means of the hyperlinks employment, the work articulates its solution with the electronic tabulator Excel of the Windows.
Introduction
Many times, when reviewing different theses of scientific, academic, and diploma work, it has been observed that in some of them reference is made to the so-called Delphi Method; But when analyzing the work, there are sometimes inaccuracies: in the way of applying said method, the way in which the experts were selected, and even in basing the analysis only on one round of consultation.
That is why this work aims to clarify all the doubts that may arise in our professors and researchers in relation to the topic discussed, through the application of a practical example, taken from the author's master's thesis in which this method is applied..
Before developing the topic, it is clarified that these articles make up an electronic book of the author, so there are hyperlinks that link content presented on other topics and that may be of interest to your query. For this, the interested party must only go to the given hyperlink and double click.
1. What is the Delphi method?
The Delphi method consists in the systematic use of the intuitive judgment of a group of experts to obtain a consensus of informed opinions (Valdés, 1999), (Moráguez, 2001). It is essential that these opinions are not permeated or influenced by the criteria of some experts.
This method is more effective if anonymity, controlled feedback, and group statistical response are guaranteed.
The method can be applied:
- As a forecast of the behavior of known variables; that is, to evaluate the behavior of a known variable and thus possible forms of behavior can be inferred. In the perspective determination of the composition of a system, eg: in the event that the elements of the system to be studied are not known, or they have never been applied to the object of study and the sample is oriented to the determination of the possible structure of the system or model to be applied.
This last case constitutes the example we illustrate.
2. What are the advantages of the delphi method?
This author considers, like Zayas (1998), Campistrous (1998), Valdés (1999), Moráguez (2001), that the advantages of the method are given in that:
- It allows the formation of a criterion with a higher degree of objectivity. The consensus reached on the basis of the criteria is very reliable. The task of decisions, based on the criteria of experts, obtained by the latter has a high probability of being efficient. value decision alternatives. Avoid conflicts between experts by being anonymous, (which is an essential requirement to guarantee the success of the method) and creates a favorable climate for creativity. The expert feels fully involved in solving the problem and facilitates its implantation. Of this the principle of voluntariness of the expert in participating in the investigation is important. It guarantees freedom of opinion (for being anonymous and confidential). No expert should know that their peer is being asked for opinions.
3. What are the disadvantages of the delphi method?
Its most significant disadvantages are given that:
- Its application is very laborious and time consuming, since it requires a minimum of two turns to obtain the necessary consensus.It is expensive compared to others, since it requires the use of: expert time, sheets, printers, telephone, mail… It requires good communications to save time on searching and receiving responses. It must be carried out by a group of analyzes: experts as such. Subjective criteria are issued, so the process may be loaded with subjectivity, subject to external influences.
Hence the need to apply several rounds, seek varied analysis techniques to obtain a consensus and statistical tests to determine its degree of reliability and relevance.
4. How are the experts selected and what instruments are applied?
For the practical application of the method it is necessary to consider two fundamental aspects methodologically: selection of the group of experts to survey and the preparation of the questionnaire or questionnaires. But first of all: Who can be considered experts?
An individual is defined as an expert, a group of people or organizations capable of offering conclusive evaluations of a problem in question and making recommendations regarding its fundamental moments with maximum competence (Valdés, 1999) (Moráguez, 2001).
From this definition it is inferred, as a basic requirement for the selection of an expert, that he have experience in the subject to consult, given his years of work (praxis), and that they can be complemented with: theoretical knowledge acquired through the different forms of improvement, and academic or scientific degree achieved in relation to the subject, among others.
5. How is the delphi method developed?
The explanation will be based on the way the author did it in his master's thesis (it can be consulted in the library).
A "pool of potential experts" or their list was created, based on the following characteristics: experience, competence, creativity, willingness to participate in the survey, capacity for analysis and thought, collectivist and self-critical interest.
Taking into account the above characteristics, instrument 1 was applied, which was only used with potential experts (Annex 1).
Thus, a population of 50 probable candidates from all over the country was considered, according to the author's estimation and consultations with other feasible experts, of whom they considered they might be (the decision to do so or not remained anonymous with these possible experts).
When determining the competition coefficient, 30 people were obtained who were part of the pool of experts to be considered in this subject from the competition coefficient (K), in which:
The expert's coefficient of competence (K) is determined as:
- Kc: knowledge coefficient on the subject that is asked for an opinion. This coefficient is self-assessed according to the value of the scale (Annex 1, question 1). This value, proposed by the possible expert, is multiplied by 0.1 and a score is obtained, eg: If the possible expert dialed 8, it is multiplied by 0.1 and 0.8 is obtained; then Kc = 0.8. Ka: argumentation coefficient. This coefficient is self-rated at high (A), medium (M) or low (B) as the degree of influence of the following sources: theoretical analyzes carried out by the possible expert, his experience obtained, works by national authors, works by foreign authors, their own knowledge of the state of the problem abroad and their intuition (Annex 1, Table 2). Let's see table 2 of this annex:
| Sources of argumentation | Degree of influences from each of the sources | ||
| A (high) | M (medium) | B (low) | |
| Theoretical analyzes carried out by you. | (0.4) | (0.3) | (0.2) |
| Your experience gained. | (0.5) | (0.4) | (0.2) |
| Works of national authors. | (0.025) | (0.025) | (0.025) |
| 0.025 | 0.024 | 0.022 | |
| Works of foreign authors. | (0.025) | (0.025) | (0.025) |
| 0.025 | 0.024 | 0.022 | |
| Your own knowledge of the status of the problem abroad. | (0.025) | (0.025) | (0.025) |
| 0.025 | 0.024 | 0.022 | |
| Your intuition | (0.025) | (0.025) | (0.025) |
| 0.025 | 0.024 | 0.022 |
Source: Campistrous, 1998.
When observing the table, in the aspect: sources of argumentation, the revised proposals (Campistrous, 1998), (Valdés, 1999)…, do not distinguish between the categories of High, Medium and Low, based on the indicator: works with national authors (note the values that appear in the upper part in parentheses and not marked in bold, from the indicator of works by national authors), assigning the same score to each item; reason why its differentiation is proposed from the values that appear below the bar in bold, so that each of the categories of the argumentation sources are well differentiated.
According to the points obtained, this coefficient receives the value of 0.796, (sum of each item of this indicator, see in Annex 1, Table 2, values marked with an 'X'). Then the competence coefficient of this expert will be given by: K = ½ (0.8 + 0.796) = 0.798 ≈ 0.8, which in this case is high, because:
- Yes £ 0.8K £ 1; then the competition coefficient is high Yes 0.5 £ K <0.8: medium competition coefficient Yes K <0.5: low competition coefficient
It is important to clarify that the values of each of the items in the second table of Annex 1 are of interest to the researcher, so they should not appear in it.
For the purposes of this survey, it was determined to exclude 25 people as potential experts for achieving a low coefficient of competence (many were highly experienced Technical and Vocational Education teachers, but lacked other requirements). So there were 25 experts.
6. What is the Torgerson Model?
There are different techniques and models for assessing consensus for experts, which can be studied in the Brochure of the Management Techniques Group (Getedi) (Zayas, 1990)
and other manuals, which for obvious reasons of scope of this work will not be analyzed; therefore, it is considered convenient to explain a very useful mathematical method and model, when the scales used in the instruments applied to the experts are ordinal: Torgerson Mathematical Model, which is a variant of the Thurstone method (Moráguez, 2001).
It has already been analyzed that one of the disadvantages of the method (Delphi) lies in the subjectivity of the criteria issued, so to try to solve this problem, it is decided to use this mathematical model, which allows, not only to assign a scale value to each indicator, but determine limits between each category and, in this way, you can obtain the real limits (assigned by a real number), between the ordinal categories and their corresponding interval scale (real numbers), between each of the ranges that make up the evaluation criteria given by the experts, and in this way it is possible to know precisely what the real limits of each category are; that is, up to what real values can the variable be considered to be indispensable, very useful, etc.
The Torgerson model tries to objectify the criteria of the experts or other personnel surveyed, by converting the ordinal scale into an interval scale (from qualitative to quantitative).
To achieve objectivity, based on subjective criteria, the model relies on two laws of social psychology: Law of Comparative Trials and the Law of Categorical Trials (Campistruos, 1998), (Moráguez, 2001). It is also supported by the following assumptions:
1) Each object (indicator) corresponds to the subjective dimension of a normally distributed random variable, whose mean, m, is the scale value of that object. All variances are the same.
2) Each category limit corresponds to the subjective dimension of a normally distributed random variable, whose mean, t, is the scale value of that limit. All variables are the same.
3) The random variables that represent both objects and limits are independent. One variable cannot contain values of another.
4) Decision rule: an object a belongs to the k-th category when its scale value x is between the values of the order limits k-1 and k. This rule leaves the border between each of the assumed categories for the indicators well defined. (Campistrous, op. Cit.), (Moráguez, op. Cit).
This model makes it possible to convert ordinal judgments, issued by independent experts, about indicators, into an instrument that expresses their relative position on a continuum; that is, it allows the ordinal scales to be taken to the interval scale (real numbers) and in this way to know the limits, in real values, in which each category is located; for example:
Excellent, Very Good…; or 5, 4, 3…
7. Methodology to apply the Torgerson Mathematical Model
1) Operationalization of the variables to be used
It is important to clarify that when requesting information from the experts about estimating the possible categorization of each of the proposed indicators or variables, it is necessary that the researcher leave the concepts that he assumes for each variable well defined, as well as establish their operationalization, which it is no more than arranging the different parameters and indicators that allow measuring the variable in the subject (Tena, 1996), to enable a better understanding and evaluation of the judgments issued by the experts.
To better understand the method, the proposal of this author for his master's thesis was taken as a reference, which by the method of experts determined the 24 indicators, distributed in six dimensions, suggested to evaluate the external efficiency or educational impact of schools polytechnics (Moráguez, 2001). The Academic Effect dimension, made up of six indicators, was taken as the benchmark for analysis.
We remind the reader that the words that are underlined and in blue are hyperlinks created so that the same by doing control + click can go directly to the annex, table, etc.
Annex 2 shows how the relevance of each indicator to measure the proposed variables was, for the second round of experts, in the Academic Effect dimension.
In the columns the ordinal categories appear to measure the variables given in: Indispensable (I), Very Useful (MU), Useful (U), Perhaps Serves (Q) and Does Not Serve (N). Note that each indicator was assigned a consecutive number (rows) to facilitate the compilation and statistical work of the same, which for obvious reasons will not be detailed in this work.
2) Compilation of absolute frequency
We remind the interested party that by clicking on the marked hyperlinks you can develop the calculation in Excel of this same example.
As a result of the compilation of the instruments applied to the experts, a table (Annex 2, Table I) is made in Excel, and it contains the criteria of the experts in relation to the indicators corresponding to the Educational Effect dimension. The columns reflect the totals for each category.
3) Determination of the accumulated frequency
Each indicator is determined by its accumulated frequency, thus: indicator 1 (Annex 2, Table I) 21 experts considered it Indispensable and 4 Very Useful, so in the accumulated frequency (Annex 2, Table II) that appears in the Very Useful item is 25 (box C14); therefore the 21 are included, who see it as Indispensable (box B14), and the 4 Very Useful. In an analogous way each one was found and the table according to Annex 2 was completed.
4) Determination of the accumulated relative frequency or accumulated probability
The cumulative probability matrix is determined with four decimal places (Annex II, Table III), which results from dividing each accumulated by the number of the sample, in this case 25; for example, if we divide 21 (value in cell B14) by 25, the cumulative probability 0.8400 (cell B23) results. This table is thus completed.
It is important to note how from the category in which probability 1 is repeated, which in this case is the Very Useful category, it is not necessary to complete the following columns, because the maximum probability has already been accumulated, which means that this indicator It is considered at least Very Useful. To better understand the next step, only the last column has been removed: No Use to measure variable (N).
5) Calculation of the cut-off points and scale of the indicators (Annex 2, Table IV)
5.1 Determination of Inverse Standard Normal Values
a) Manual method to find the values
To determine the inverse standard normal values of the accumulated probabilities of each indicator, the values are located in the tables in Annex 3 (sheet 1 to 3, which is nothing more than the table of the standard distribution), according to the associated probability and proceed as follows:
Search in Annex 3 for the closest value of the Normal Standard curve of the accumulated probability 0.8400 (Table III, Annex 2). It is clarified that in Excel this inverse standard function is called because once the value of the accumulated probability is known, the standardized values of the table must be searched.
In Annex 3, sheet 2 the closest value of the accumulated probability appears (it is marked) of 0, 8389 and corresponds the inverse standard value of 0.99; value that is placed in box B32 of table IV of annex 2. Proceed in the same way for the rest of the values of the accumulated probabilities up to box E28.
It is important to know that when the accumulated probability is equal to 1, the inverse standard value of 3.5 corresponds to it and that this value becomes asymptotic from 3.49, so in practice it is taken equal to 3.5, for work with both ends equal; that is, if, on the contrary, the accumulated probability is equal to 0, then the inverse standard value is assumed to be equal to -3.5 (Annex 3, sheet 3 and sheet 1 respectively); Observe in the table (Annex III Sheet 1) that the probabilities, starting from z = -3.5, take extremely small values (infinitesimal), so it can be considered that for values less than -3.5, the cumulative probability is assumed equal to 0 and it is also an easy resource to associate the maximum and minimum probabilities to the same absolute value (3.5); Then if Z = -3.5, the probability for this value = 0 (the minimum);on the contrary, if Z = 3.5, the probability associated with this value is 1 (the maximum).
It is appropriate to clarify to the reader that the maximum and minimum values of the normal function many authors differ in them; for example: Murray (1961) takes maximum value 3.99 and minimum -3.99; Freud (1977) assumes 3.09 and -3.09; Douglas (1996) takes ± 3.99; Devore (2000) assumes ± 3.49.
But as all these values are asymptotic from ± 3, 5, this is why this author assumes it this way, as it is even easier to remember, This way you can find the values of this function manually and fill the table IV (Annex 2). b) Automatic method to find the values of the inverse standard function These values are found automatically through Microsoft Excel in Windows. In order to automatically determine each one of said values, you must place it in the box where the value of the Inverse Standard Normal Function will be inserted (Box B39 of sheet 1 of Excel, which in the example of this work corresponds to the box B32, from Annex 2.
Once located in said box, you must specify in the calculation bar, look for the Inverse Standard Function corresponding to the accumulated probability 0.8400 (box B31 of Table III, annex 2), but you must specify that when the accumulated probability equals 0, the inverse function takes the value -3.5 (value marked in gray in Sheet 1, annex 3); on the contrary, if the accumulated probability takes the value of 1, then its Inverse Standard Function will assume the value of 3, 50 (last value of Sheet 3, annex 3). This is programmed in the Excel formula bar: = YES (B23 = 0, -3.5, YES (B23 = 1,3.5, (STANDARD DISTR.INV (B23))))
Note that the box referenced is B23 (Table III, Annex 2), because it is the first box to which the Inverse Standard Function of the cumulative probability must be determined.
This is how this function is programmed (in Excel) that allows determining the inverse values of each associated probability, so that when this probability is equal to zero, then the computer assumes it as a true logical response and returns the value -3.50, and when it is probability 1, return the logical value of 3.50.
You can see this by clicking on the Excel calculation bar by clicking on the following hyperlink:
Call exercise in Excel It is clarified that according to the methodology proposed by the model, the last column (s) whose values are equal to 3.5 are eliminated, because when the last one is determined cut-off point, will indicate that all values greater than that value will correspond to the eliminated variable (s) (budget number 4 of the method); the latter will be better understood in the following section. For this reason, only four columns appear with scales I, MU, U and Q. (Annex 2, Table IV); It is insisted that for reasons of better understanding the next step, the columns of U and Q have not been eliminated, (it can be done) since the maximum value of the Normal Distribution reaches 3, 5 (probability 1,000).5.2 Search for the cut points Once each image is determined by the Inverse Standard Normal Curve,each column and each row are added (Annex 2, Table IV); the rows are averaged: the sums of the rows are divided by four, which is the number of columns that correspond to each scale, and the result is assigned to the Average column (P, box G32). If columns U and Q had been removed, then it would be divided by two (remaining column numbers: I and MU). We proceed in the same way with the columns where the scales of I, MU, U and Q appear, but the result of the sum of the column is divided by the number of indicators (in this case 6) and the average of each is found. column (Annex 2, Table IV, row 39).box G32). If columns U and Q had been removed, then it would be divided by two (remaining column numbers: I and MU). We proceed in the same way with the columns where the scales of I, MU, U and Q appear, but the result of the sum of the column is divided by the number of indicators (in this case 6) and the average of each is found. column (Annex 2, Table IV, row 39).box G32). If columns U and Q had been removed, then it would be divided by two (remaining column numbers: I and MU). We proceed in the same way with the columns where the scales of I, MU, U and Q appear, but the result of the sum of the column is divided by the number of indicators (in this case 6) and the average of each is found. column (Annex 2, Table IV, row 39).
The values that result from the previous operations are called cut points, and they determine the values of the interval in which the qualitative variables will be included (I, MU, U and Q) (see numerical ray determined with the values of the intervals at the bottom of Annex 2).
Now the values of the cut-off points are averaged, which are only the averages of each category (column), so the average is obtained, called the limit value N.
To find out what range the variable really is in, the average value of each row P is subtracted from the limit value N. For example: for the first row or indicator No. 1 (1), the limit value N is subtracted (average average) (2.79) the average value of this row P (2.87, box G32) (Annex 2, Table IV) and its result appears in column NP (-0.08, box H32). Since this value is below 0.66 (box B39), which is the cut-off point or upper limit for the category of Indispensable (I).
Therefore, the experts considered that this category was essential for the indicator (1).
Similarly, each range of the rest of the indicators is determined, the results of which can be seen in the table in the aforementioned annex.
8. Determination of the consensus level of the first round.
The level of consensus (C) is determined by the expression:; where:
Where…
In the event that there is no consensus among the experts C <75%, another consultation round should be made, making the necessary modifications and clarifications that allow consensus to be reached on the new modifications and adaptations of the new proposal by these experts.
As observed in Annex 2, Table IV, the 25 experts considered that the six proposed indicators, to evaluate the educational effect, were indispensable (I), with a high degree of relevance; since there were no negative votes (Vn = 0); the total votes were 25, which coincided with the positive ones (Vt = 25), then: C = (1 - 0/25) • 100 = 100%. In this way, the rest of the 24 indicators in the six dimensions proposed were determined.
By way of conclusion, it can be expressed that the Delphi method or expert method is very valuable for decision-making, and that it is currently widely used in leading companies as a method of management and general research, due to the participation that is granted to a whole series of experts, which allows obtaining informed opinions with a high level of competence, to finally reach consensus on the issues raised.
One of the most important elements of this method lies in the careful selection of possible experts from a stock exchange or relationship, of which it is assumed that they could be, with the application of the instruments already analyzed and that guarantee anonymity: that each expert does not know who their equal is, in order to avoid halo effects, among others.
In general, when it comes to evaluating or determining the relevance of indicators expressed on an ordinal scale, the Torgerson mathematical model should be applied, (already explained its use in this work) modification of the Thurstone model, which allows to take ordinal scales to scale. of interval and thus to know in what numerical interval are each of said qualitative categories to know the degree of relevance of the indicators, methodological variants, and in general the proposals made.
This is not the only method that exists to compile the results of expert opinions; on the contrary, this method can, and should be, combined with others that help to obtain a better consensus of the formulations made.
File corresponding to Annexes 1 and 2: Expert Consultation (.doc) and Example of how to apply the Torgerson mathematical model of the Delphi method (.xls)
Appendix 1
Consult Experts
(Delphi method)
You have been selected as a possible expert to be consulted regarding the degree of relevance of a set of indicators that presumably should serve to assess the external efficiency or educational impact of industrial polytechnic schools in the province of Holguín, in view of the research He is carrying out as a thesis for a master's degree in Planning, Administration and Supervision of Educational Systems by the Latin American and Caribbean Pedagogical Institute (IPLAC).
It is necessary, before carrying out the corresponding consultation as part of the empirical research method: “consultation with experts”, to determine your coefficient of competence in this matter in order to reinforce the validity of the result of the consultation to be carried out. For another reason, please answer the following questions as objectively as possible. Thank you!
1. Mark with a cross (X), in the following table, the value that corresponds to the degree of knowledge that you possess on the topic “determination of indicators to evaluate the quality of education”. (Consider that the scale presented to you is ascending, that is, the knowledge on the referred subject grows from 0 to 10).
Table 1
| 0 | one | two | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 |
| X |
Source: (Campistrous, 1998; 19).
2. Carry out a self-assessment of the degree of influence that each of the sources, presented below, has had on your knowledge and criteria on determining indicators to assess the quality of education. To do this, mark with a cross (X), as appropriate, in: A (al-to), M (medium or B (low).
Table 2
| Sources of argumentation | Degree of influences from each of the sources | ||
| A (high) | M (medium) | B (low) | |
| Theoretical analyzes carried out by you. | (0.4) | (0.3) X | (0.2) |
| Your experience gained. | (0.5) | (0.4) X | (0.2) |
| Works of national authors. | (0.025) | (0.024) x | (0.022) |
| Works of foreign authors. | (0.025) | (0.024) | (0.022) x |
| Your own knowledge of the status of the problem abroad. | (0.025) x | (0.024) | (0.022) |
| Your intuition | (0.025) x | (0.024) | (0.022) |
Appendix 2
Correspondence between qualitative categories and numerical ray
Notice that the cutoff points determine how far the range limits go for each category. In this case, it was found that all values less than or equal to 0.66 fall into the Indispensable category and the rest to very useful.
Standard distribution table (Annex 3 Sheet 1)
Source: Excel 2000 Inverse Standard Function
Standard Distribution Table Annex 3 Sheet 2
Standard Distribution Table Annex 3 Sheet 3
Example of how to apply the Torgerson mathematical model to give the final evaluation of a group of students. A group of students wants their final evaluation in a signature, taking into account all the qualitative evaluations carried out in the semester and for this, this mathematical model is applied.
Bibliography
1. CAMPISTROUS PÉREZ, L. Indicators and educational research / Luis Campistrous, Celia Rizo Cabrera. - Havana, August 23, 1998. - 23p. - (Unpublished material).
2. FRIEDRICH, W. Methods of Marxist-Leninist social research. Edit. Social Sciences, Havana, 1988. 325 p.
3. MORÁGUEZ IGLESIAS, A. Proposal of indicators to evaluate the external efficiency of industrial polytechnic schools in the province of Holguín. - 2001. - 105 p. Thesis to choose the title of Master in Planning, Administration and Supervision of Educational Systems. - IPLAC: Latin American and Caribbean Pedagogical Institute. Havana, 2001.
4. TENA SUCK, E., A. Experimental research manual: thesis preparation. - Edit. Plaza and Valdés, Mexico, 1996. 82 p.
5. THURSTONE, LL The Method of paired Comparisson for Social Values. In: J. Abn. Soc. Psychol. No. 21, 1927. 14 p.
6. Theory of Attitude Measurement. In: Psychol No. 36, 1929. 14 p.
7. TORGERSON, WS Multidimensional Acalling: I. Theory and Method (I Theory and method). In: Psychometrika. No. 17, 1952. 12 p.
8. Multidimensionalk Scalling of Similitary In: Psychometrika. No 30, 1965. 15 p.
9. ZAYAS MIRANDA, E. The process of decisions and problem solving. GETEDI Management Techniques Group. Series: Management issues. No 004, ISTH, Holguín, 1990. 70 p.
This method was created at the RAND CORPORATION in the USA in the 1940s, by TJ Cordon and Olaf Helmer and was published in 1964 (Zayas, 1990; 30).
