Methods of expert evaluation - System theory and system analysis

Methods of expert evaluation

In the future, on the ideas of the BCG matrices, three-dimensional matrices [47] are proposed, the axes of which form complex indicators: the attractiveness of the market for the organization, the competitive position of the enterprise, the competitiveness of the product.

Expert estimates are a group of methods used to evaluate complex systems at a qualitative level. The term expert comes from the Latin word expert, meaning "experienced."

When using expert assessments, it is usually assumed that the opinion of an expert group is more reliable than the opinion of an individual expert. In some theoretical studies it is noted that this assumption is not obvious, but it is also asserted that, if certain requirements are met, in some cases, for some problems (see below), group estimates can be made more reliable than individual ones. Therefore, it is important in the organization of expert surveys to introduce certain rules and use the appropriate methods for obtaining and processing expert estimates.

Algorithm for organization of expert surveys and evaluation processing. A lot of work has been devoted to the study of the features and possibilities of applying expert assessments.

They cover:

■ the problems of forming expert groups, including requirements for experts, the size of the group, the issues of training experts, assessing their competence;

■ forms of expert interviews (various kinds of questionnaires, interviews, mixed forms of interviews) and methods of organizing the interview (including questionnaire techniques, brainstorming, business games, etc.);

■ approaches to evaluation (ranging, rationing, various types of ordering, including methods of preferences, paired comparisons, etc.);

■ methods for processing expert assessments;

■ ways to determine the consistency of expert opinions, the reliability of expert assessments (including statistical methods for estimating variance, probabilities for a given range of changes in estimates, Kendall's ranking correlation, Spearman coefficient, concordance coefficient, etc.) and methods for increasing the consistency of estimates by means of appropriate ways to process the results of an expert survey.

The algorithm for organization of expert surveys is shown in Fig. 2.16. An overview of the forms and methods of obtaining and processing expert assessments can be found, for example, in [11, 12, 14, 48, etc.].

In particular, B. G. Litvak [48] on the basis of generalization and research of the types of measurement scales and relations, considers features of proximity measures of various kinds (on non-metrical and vector relations, structural, Euclidean), characterizes principles and methods based on the choice various ways of ordering and preference relations (including ranking and hyper-ordering methods, Cherchmen > Ackoff, Thurstone, the mixed alternative method von Neumann - Morgenstern, the principle of discarding the alternatives Arrow, median search algorithms Kemeni, metrized rankings, Pareto, methods for determining preferences on sets of multidimensional alternatives, etc.).

The most common expert measurement procedures include [14, 48]: ranging, pair comparison, multiple comparisons, direct evaluation, sequential comparison, Thurston's method, Cherchman - Ackoff, the von Neumann method - Morgenstern.

Fig. 2.16

The expediency of applying this or that method is determined by the nature of the problem being analyzed, the information used.

If only qualitative assessments of objects are justified on certain qualitative grounds, then methods of ranking, pair and multiple comparison are used. If the nature of the information being analyzed is such that it is advisable to obtain numerical estimates of objects, then one can use this or that method, from direct numerical evaluations to the more subtle methods of Thurstone and von Neumann - Morgenstern

Methods of expert evaluation have different qualities, but lead in general to close results. The practice of applying these methods has shown that the most effective use of various methods for solving the same problem is the most effective. A comparative analysis of the results raises the validity of the conclusions drawn. It should be borne in mind that the method that requires minimum costs is ranking, and the most laborious is the method of sequential comparison ( Cherchmena - Akoffa ). The paired comparison method without additional processing does not give a complete ordering of objects.

When carrying out sociological measurements that can be considered as a kind of expert evaluations (especially in the case of organizing a selective sociological study), usually use qualitative scales of various kinds that are associated with quantitative estimates of the degree of significance ( very important & quot ;, important , "it is more important than not", etc.), or the qualitative attribute introduced in the question is evaluated and the question is formulated in the form of an assertion, and when answering it they are asked to express the degree of agreement or disagreement with the statement (in the form of "completely agree", "agree", "disagree", "strongly disagree" or "yes", "yes rather than no", "rather no than yes & quot ;, no etc).

In this case, appropriate methods of processing the results can be applied. For example, when using the Likert scale, the questions asked by a group of people should be assessed on a five-point scale (5 points - "totally agree", 4 points - "agree", 3 points - "neutral answer", 2 points - "do not agree" , 1 point - "completely disagree") and it is recommended to apply the method of summary evaluations during processing. The scale analysis of Guttman reduces to constructing scales of the ordinal level of measurement, representing single-scale scales formed on the basis of the originally used ranked scale by eliminating questions or factors that are unrelated to the measured characteristic. When applying the semantic the differential developed by C. Oggood to measure the meaning of concepts and words and the differentiation of the emotional side of the value of the concept being evaluated, graphic methods that help determine the distribution profile of installations are used as intermediate processing methods.

Rating matching methods (consensus technique). These methods are used to process individual peer reviews. The methods have many variants, differing in the ways with which the general estimates are obtained from individual estimates.

In this case, various methods of matching estimates are also used:

1) the simplest, based on obtaining the average probability

(where n is the number of participating experts) or the weighted-average probability value

(where k i is the weights attributed to the evaluation of each expert);

2) special methods for measuring the measurement and increasing the consistency coefficients (or consistency coefficients) of expert opinions;

3) methods based on the selection of an expert group with a high coefficient of consistency of opinions. For example, a method based on converting the first three ranks of a discrete scale into a continuous one with subsequent normalization of this new scale, reflecting the opinions of the selected experts.

Often the methods of the theory of rank correlation are used in processing the materials of the collective peer review. To quantify the degree of consistency of expert opinions, the concordance coefficient W is applied, which allows one to assess whether the series of preferences constructed by each expert are consistent among themselves. Its value is in the range 0 ≤ W ≤ 1; W = 0, means the exact opposite, and W = 1 - full match of ranking results. Practically, the validity is considered good if W = 0.7 or 0.8.

The small value of the concordance coefficient, indicating a weak consensus among the experts, is a consequence of the following reasons: in the whole set of experts there is really no common opinion; Within the set of experts under consideration, there are groups with high consensus of opinions, but the generalized opinions of such groups are opposite.

For clarity of the representation of the degree of consistency of the opinions of any two experts A and B is the coefficient of pair rank correlation ρ, it takes the values ​​-1 & lt; ρ & lt; +1. The value ρ = +1 corresponds to the complete coincidence of the ratings in the ranks of the two experts (the full consistency of the opinions of the two experts), and the value ρ = -1 corresponds to two mutually opposite results of ranking the importance of the properties (the opinion of one expert is contrary to the opinion of the other).

As one of the methods for increasing the consistency of expert assessments, the Delphic Oracle method is used, or the Delphi method, based on the ideas of which a number of modifications have been formed.

Methods such as "Delphi". The Delphi method, or the delphic oracle method, was originally proposed About. Helmer and his colleagues [106] as an iterative procedure for "brainstorming", which would reduce the influence of psychological factors in conducting meetings and improve the objectivity of the results. However, almost at the same time, "Delphi" -procedures became a means of increasing the objectivity of expert surveys using quantitative estimates in a comparative analysis of the constituent "goal trees" and when developing scenarios & quot ;. The main means of increasing the objectivity of results when applying the Delphi method - use feedback, familiarize the experts with the results of the previous round of the survey and take these results into account when assessing the importance of expert opinions.

In specific techniques that implement the "Delphi" -procedure, this idea is used to varying degrees. So, in a simplified form, a sequence of cycles of iterations is organized during the brainstorming. In a more complex version, a program of consecutive individual interviews is developed using questionnaire methods that exclude contacts between experts, but provide for familiarizing them with each other's views between rounds.

In advanced versions, the "Delphi" -procedure is a program of successive individual surveys using questionnaire methods. Questionnaires from the tour to the tour are being refined. Experts are assigned weighting factors of the significance of their opinions (competence coefficients), calculated on the basis of previous surveys, also refined from tour to round and taken into account when obtaining generalized survey results. To reduce such factors as suggestion or adaptability to the majority opinion, sometimes it is required that experts substantiate their point of view, but this does not always lead to the desired result, but, on the contrary, may enhance the adaptability effect or the Oedipus effect considered below.

Due to time-consuming processing of results and significant time-consuming, the original recommendations of the "Delphi" not always possible to implement in practice.

Recently, the "Delphi" -procedure in one form or another usually accompanies any other methods of modeling systems - the method of the "target tree", the morphological, network, and so on. In particular, a very promising idea of ​​the development of expert assessment methods, proposed by В. M. Glushkov , is to combine a focused multistage survey with the "scan" problems in time, which becomes quite realizable when using a computer.

To improve the effectiveness of surveys and the activation of experts sometimes combine the "Delphi" -procedure with elements of the business game: the expert is invited to carry out a self-assessment, putting himself at the place of the designer who is actually tasked with the execution of the project, or to the place of an employee of the management apparatus, systems of organizational management, etc.

Features and disadvantages of methods of expert assessments. The choice of approaches and methods depends on the specific tasks and conditions of the examination. However, there are some common problems that need to be understood when conducting any expert surveys. Briefly describe them.

The possibility of using expert judgments, the justification of their objectivity is usually based on the fact that an unknown characteristic of the phenomenon being studied is treated as a random variable, the reflection of the distribution law of which is the individual evaluation of an expert expert about the reliability and significance of an event. It is assumed that the true value of the investigated characteristic lies within the range of expert estimates (where - representative sample) , received from the group of experts, and that the generalized collective opinion is reliable.

However, in some theoretical studies this assumption is questioned.

For example, it is proposed to divide the problems, for the solution of which expert judgments are applied, into two classes. The first class includes problems that are sufficiently well provided with information and for which the principle of the "good gauge" can be used, considering the expert as the custodian of a large amount of information, and the group opinion of experts is close to the true one. The second class includes problems for which knowledge is insufficient to assure the fairness of the above assumptions, experts can not be considered as "good meters", and care must be taken to process the results of the expert survey, since in this case the opinion of one (single) expert giving more attention than others to the study of a poorly understood problem may turn out to be the most significant, and with formal processing it will be lost. In this regard, the problems of the second class, in general, should be applied qualitative processing of the results. The use of averaging methods (valid for good meters ) in this case can lead to significant errors.

The tasks of collective decision-making on the formation of goals, the improvement of methods and forms of management can usually be attributed to the first class. In order to improve the objectivity of the results, it is advisable to identify contradictory and "rare" opinions and subject them to a more thorough analysis.

Another feature that should be borne in mind when applying expert assessments is as follows. Even in the case of solving problems related to the first class, one should not forget that expert assessments carry not only the narrowly subjective features inherent in individual experts, but also collective-subjective Features that do not disappear when processing survey results (and when applying "Delphi" -procedures and methods to improve the consistency of expert opinions may even increase).

For a more popular explanation of this feature, assuming that one of the varieties of the expert poll is a vote, we give the opinion of one of the heroes Guy de Maupassant {{ * Maupassant, Guy de. Paulie. collect. op. - T. 1. - M., 1958. - P. 259-260. : "You will probably agree with me that ingenious people are rare, is not it? But let us be generous and assume that there are five people in France. Let's add, with the same generosity, two hundred highly talented people, a thousand others, also talented, each in his field and ten thousand people, one way or another outstanding. Here is the General Staff of eleven thousand two hundred and five minds. Behind him goes the army of mediocrities, followed by the whole mass of fools. And since mediocrities and fools always make up the vast majority, it is unthinkable to imagine that they could elect a reasonable government. " And further, emotionally strengthening his point of view, Maupassant gives such assessments of the situation: "... the only force that can be measured - this is the one with the least to be reckoned with: the senseless power of the majority. ... The ignorant majority will always prevail over a genius, over science, over all the accumulated knowledge ... and proposes introducing corrections into the voting system based on the introduction of a kind of "competence coefficients" experts.

One way to eliminate the drawbacks associated with the feature in question is that when applying expert surveys to make decisions in organizational systems, special attention should be paid to the formation of an expert group and to methods for processing the survey results, highlighting and taking into account rare and contradictory opinions, and the resulting average estimates should be looked at as some "public point of view", depending on the level of scientific and technical knowledge of society relative to the subject of the study Ania or decision. In this case, of course, such a "public point of view" can change as the system develops and our ideas about it, and expert surveys need to be repeated. Such a way of obtaining information about a complex problem, characterized by a high degree of uncertainty, should become a kind of mechanism in a complex system, i.e. it is necessary to create a regular system of work with experts.

There is one more feature of the method of expert assessments, which drew attention to A. M. Gendin, calling it the Oedipus effect.

It is that the expert leader in the organization of an expert poll in the form of "Delphi" -procedures with an oral discussion of the results of the evaluation between the survey rounds can gradually "withdraw" group of experts in the desired direction.

It should also be noted that the use of the classical frequency approach to the assessment of probability in the conduct of expert surveys is difficult, and sometimes impossible (due to the impossibility of proving the representativeness of the sample). Therefore, at present, research is being conducted on the nature of the probability of peer evaluation, based on the theory of Zade's fuzzy sets, on the representation of the expert assessment as the degree of confirmation of the hypothesis or as the probability of achieving the goal (the latter direction develops on the basis of the information approach expounded in the next chapter). >

The considered features of expert assessments lead to the need to develop methods for organizing complex examinations that help, dismembering large uncertainty on a part, introducing evaluation criteria and applying different forms of interview, to obtain more objective and reliable estimates.

Methods and models for organizing complex examinations, such as the method of solving matrices, methods that take into account several criteria and their weight coefficients, methods based on the information approach, etc., are being developed in search of means for increasing the objectivity of evaluations. These methods will be considered in Ch. 6.

thematic pictures

Also We Can Offer!

Other services that we offer

If you don’t see the necessary subject, paper type, or topic in our list of available services and examples, don’t worry! We have a number of other academic disciplines to suit the needs of anyone who visits this website looking for help.

How to ...

We made your life easier with putting together a big number of articles and guidelines on how to plan and write different types of assignments (Essay, Research Paper, Dissertation etc)