# The method of complicated expert procedure, proposed...

## The method of complicated expert procedure, proposed in the PATTERN technique

The methodology of PATTERN is discussed in Ch. 5. It distinguishes three groups of evaluation criteria:

■ relative importance;

■ mutual utility;

■ status and development time ( state-time period ).

These criteria groups were used to evaluate the components of the target tree and evaluation of the implementation of projects that implement the objectives. But later modifications of these types of criteria were developed and applied.

In Ch. 5 these groups of methods are considered in the PATTERN variant. Here we draw attention to the fact that due to which the system of criteria proposed in PATTERN allows us to increase the objectivity of evaluations, and we will give variants of its development.

The estimation of relative importance taking into account several criteria and their weighting factors is carried out by the method of rationing. Several criteria (coefficients) of relative importance are used and their weights are taken into account. A matrix is ​​constructed for the correspondence of the elements subordinate to the node and the criteria adopted for their evaluation (see Table 5.1 in Chapter 5).

The procedure provides for checking the fulfillment of the two normalization conditions (5.1) and (5.2) in order to achieve uniformity of results: rationing of the weight coefficients of the criteria and estimates of the relative importance of the evaluated components for each criterion . The resulting estimates of the relative importance of the jth element of the i-th level are defined as follows in accordance with (5.3):

(6.1)

After the processing of the estimates - the validity of the resulting results is checked according to (5.4)

The following designations are accepted: - the name of the criterion; - the weight of the criterion; - the relative weight of the element being evaluated (by the appropriate criterion); - coefficient of relative importance of the j th element of the i-th level.

In a multi-level hierarchical structure, the procedure is repeated at each level for each node in the hierarchical structure. The evaluation procedure starts from the top level of the structure, then moves to the next higher level, where the evaluation is performed sequentially for each of the nodes, then for the third one with a sequential evaluation for each of the nodes, and so on. In order to obtain normalized estimates of the relative importance of the element of the lowest or any element of the intermediate levels of the hierarchical structure, it is necessary to multiply the estimate of the element rij of the level of interest by analogous estimates of all the elements-nodes that are between this element and the vertex of the hierarchy.

The evaluation procedure is usually organized in the form of two or three rounds. After each round, the experts get acquainted with the results of evaluations of colleagues. Those experts whose opinions differ significantly from others can be given the opportunity to substantiate their point of view, which can increase the objectivity of assessments, encourage some experts to reconsider their estimates.

Automated procedures are being developed to perform the calculated operations.

An example of data entry for one node of an estimated hierarchical structure is given in Ch. 5. When developing an automated procedure, naturally, it is necessary to provide for the considered conditions for the normalization and multiplication of the initial estimates of the underlying elements on the estimates of the component that unites them. Examples of automated procedures are given in the manual of the authors [4].

In the following, the method proposed in PATTERN developed. When organizing the evaluation procedure, it was suggested to take into account the competence coefficients of experts, to be refined at each round of the survey, taking into account their proximity to average estimates. To take into account the competence coefficients, a procedure similar to the one given was applied, only in the matrix experts and their competence coefficients were placed instead of the criteria.

The sequence of processing the results of the assessments, taking into account both the weight coefficients of the criteria, and the competence coefficients of the experts, can be any: first you can process the estimates in the sequence outlined above, and it is possible, in the second round, to take into account the experts' competence coefficients and process each criterion in sequence , and then - combine the estimates by different criteria.

With a large difference in expert competence factors, more complex algorithms are used to process the estimates, taking into account the consensus of experts, and on their basis either expert opinions having low consistency of opinions with others should be excluded, or special attention should be paid to these opinions as to rare .

The above measures, allowing to increase the objectivity of the assessment, can be supplemented with the allocation of expert competence spheres by levels and nodes of the hierarchical structure.

Procedures that take into account the weight coefficients of the criteria and the coefficients of competence of experts allow obtaining more objective estimates in comparison with simple averaging of experts' opinions. However, one of the main drawbacks of expert assessments is still retained: averaging, application of iterations, and the introduction of the dependence of the experts' competence coefficients on the average score strengthens the collective-subjective opinions.

Therefore, along with the identification of contradictory and rare opinions (opinions of experts with a low coherence coefficient with others), it is useful to use (as recommended in the generalized methodology in Chapter 5, paragraph 5.4) indirect quantitative estimates and graphical representations that allow compare these assessments and assessments of experts. It is advisable to take into account not the total values ​​of Gu-, but the intermediate estimates q x s - x for each criterion.

The results can be displayed on the display or printer either in the form of histograms, or estimates of different experts and indirect quantitative estimates can be compared in one plane, which is more convenient for identifying conflicting estimates.

The group of assessments of mutual utility, highlighted in the system of assessments of the PATTERN methodology, allows us to clarify estimates of relative importance. However, a full search of all components of the corresponding level of the target tree is very laborious. Therefore, various options for reducing the full enumeration of the compared components were proposed. In various developments in our country, the evaluation of mutual utility was sometimes interpreted as an assessment of interrelationship without determining the numerical coefficient of the binding force.

Example

For example, in the development of automated control systems, the choice of the subsystems was made taking into account their interconnection, and the structure of the functional part of the ACS for the planned period included subsystems that, with respect to others, received lower ratings of relative importance, but related to subsystems that received high estimates of relative importance; and the development of subsystems not related to the most significant ones was postponed to the next planning period.

Estimates of the state and development terms ("state - term") in the variant of the PATTERN technique are difficult to apply in real conditions. A preliminary careful justification of the resource costs for the life cycle stages of the project is required (see Table 5.2). As the project progresses, the initial estimates may change. In practice, the criterion in question is sometimes interpreted as an estimate of the effectiveness of the use of economic resources.

At the same time, for large, costly projects, the application of evaluation ideas by the criterion "state-time" is useful.

In the following, modifications and modifications of the groups and types of criteria proposed in the PATTERN methodology were developed and applied.

Increasing the objectivity of estimates using the ideas of this technique is enhanced by:

■ the dismemberment of large uncertainty over the levels of the hierarchical structure;

■ the introduction of groups of criteria and the breakdown of uncertainty by groups of criteria;

■ introducing types of criteria of relative importance and taking into account the weight coefficients of the criteria;

■ the ability to identify areas of expertise for different criteria.

[...]

[...]

[...]

[...]

[...]

[...]

[...]

[...]

[...]

[...]

[...]

[...]

[...]

[...]

## Other services that we offer

If you don’t see the necessary subject, paper type, or topic in our list of available services and examples, don’t worry! We have a number of other academic disciplines to suit the needs of anyone who visits this website looking for help.

## How to ...

We made your life easier with putting together a big number of articles and guidelines on how to plan and write different types of assignments (Essay, Research Paper, Dissertation etc)