Purpose The candidate patterns collected need to be evaluated in this stage. The following table explains each criteria and sub-criteria as well as what each of the three qualitative score means i terms of the criteria. The objective of the evaluation is to give guidance as to the suitability of candidate patterns both in terms of the domain knowledge and the methodological correctness. Even if quantitative measurements can be attributed to certain sub-criteria, comparable scores are not possible. It is also the case that differing organisations will different emphasis on the various criteria, creating implicit or explicit weighting of the criteria.
The only case where a direct comparison will be needed will be when two or more candidate patterns propose differing solutions the same problem. Whenever such a situation occurs, the domain experts should give weights signifying relative importance, to the criteria used in the suitability table.
 
 

Criteria	Sub-criteria	High Value	Medium Value	Low Value
Usefulness
	Degree of triviality  The degree to which the pattern addresses a problem which is of little importance to the business because the problem or solution is obvious.	The pattern is concerned with issues that are or most likely will be of concern to other ESI companies.	While the pattern deals with a pertinent problem, the solution is already well known.	The pattern is concerned with a problem which does warrant the creation of a pattern since it is so trivial with the proposed solution being obvious to domain experts.
	Grade of implementability  Extent that pattern is thought to be practical and implementable. Is change compatible with business strategy. Have trade-offs been taken into account [Pfeffer 97].	The pattern is useful in that it prescribes practical, easy to understand and implement solutions.	The pattern may be of some use despite some practical problems in implementation and some difficulty in understanding the solution.	The pattern is not usable. The solution is impractical and difficult to understand. The pattern only proposes "paper-based" change rather than real change
	Degree of confidentiality	The pattern does not disclose any confidential business information.	Some information may be able to be used by (potential) competitors.	The pattern discloses sensitive business information.
Quality
	Degree of complexity  The number of variables and the number of relationships between variables.	The pattern addresses only a few manageable main concepts and ideas.	The pattern is complex but may still be useful in that the complexity is needed.	The large number of factors that affect the implementation of the solution minimises the chances that the solution can be implemented.
	Addition of Value  The local and global benefits accruing to the business with the implementation  [Yilmaztürk et al 98]	The consequences of a successful implementation are great value to the part of the business directly affected as well as the whole business	The local and global benefits are unclear, difficult to determine or marginal.	There are no local or global benefits or there is a conflict between these so that in total no value is added
	Level of genericity  Abstraction level of the problem that the pattern addresses [UMIST-UP1 98]	The pattern addresses an ESI problem that is general enough to be applicable in other companies.	The pattern addresses a problem that is so general that it does not apply only to the ESI sector.	The pattern addresses a problem that is only relevant to the business in which it was discovered.
	Grade of understandability  Visualisable and identifiable according to [UMIST-UP1 98]	The pattern is easy for decision-makers, domain experts and those to be affected, to comprehend.	The pattern is only partially und-erstandable to decision-makers, domain experts and those to be affected.	The pattern is incomprehensible to  stakeholders
	External compatibility  The extent to which the pattern could be used by other companies	The pattern has taken into account differences in national, business and organisational cultures and ways of working amongst identified future external users of the pattern. This entails a full explanation of the underlying conditions on which the patterns are based.	The pattern has partially takes into account differences in national, business and organisational cultures and ways of working amongst identified future external users of the pattern. There is a partial explanation of the underlying conditions on which the patterns are based.	The pattern does not take into account differences in national, business and organisational cultures and ways of working amongst identified future external users of the pattern. There is no explanation of the underlying conditions or context on which the patterns are based.
Cost
	Level of experience in their use	The pattern has been implemented within the business.	The pattern has been partially or sporadically used.	The pattern has never been implemented.
	Economic feasibility of the proposed solutions	Proposed solution is relatively easy to implement. Organisational support exists in terms of sufficient resources as well as managerial support. The solution is politically and socially acceptable .	Proposed solution is difficult but feasible to implement. Organisational support is lukewarm. Resources are available but may not be sufficient. There may exist political and social difficulties in making the pattern feasible.	Proposed solution is not feasible. Organisational support will be difficult to obtain. The resources will not be made available. The existing difficult social and/or political climate would make an implementation impossible.

 Proposed actions

Proposed actions are the actions that may be taken as a result of the the pattern recieving scorrs according to the criteria above. The may be one of the following:

1. Acceptance

If the candidate pattern is accepted as a whole, then the following two stages may be skipped over.
 
2. Corrective actions needed

If the candidate pattern does not measure up to the criteria described in the previous section, then it will need to be adjusted according to the description in section 0.
 
3. Further evaluation needed

The candidate pattern is not (totally or partially) clear to the evaluator. The source needs to be examined or consulted with for further information
 
4. Rejection

Corrective actions

The following section presents each criteria again. These corrective actions give guidelines about how to deal with candidate patterns when the evaluation of suitability reveals criteria which have not satisfactorily been reached.

The "current state" is a description of situation when the lowest possible score is obtained for the criteria. The "desired state" is the description of situation when the highest possible score is obtained. "Action" explains what needs to be done so as to move from the current state to the desired state.
 

Degree of triviality

• Current State: The pattern is concerned with a problem which does warrant the creation of a pattern since it is so trivial with the proposed solution being obvious to domain experts.
• Desired State: The pattern is concerned with issues that are, or most likely will be of concern to other ESI companies.

• Action: Investigate the pattern in more detail to determine if there are aspects that are not as trivial that need further clarification or if this is not possible abandon this candidate pattern.

Grade of implementability

• Current State: The pattern is not usable. The solution is impractical and difficult to understand

• Desired State: The pattern is useful in that it prescribes practical, easy to understand and implement solutions.

• Action: It may be necessary to find an alternative solution that is more usable. This would entail a trade off between practicality and the achievement of a satisfactory solution

Degree of confidentiality

• Current State: The pattern discloses sensitive business information.

• Desired State: The pattern does not disclose any confidential business information.

• Action: If it is not possible to remove or generalise the confidential information, then abandon further work on the pattern

Degree of complexity

• Current State: The large number of factors that affect the implementation of the solution minimises the chances that the solution can be implemented.

• Desired State: The pattern addresses only a few manageable main concepts and ideas.

• Action: The pattern needs to be decomposed into smaller components. These together will most likely constitute a hierarchy or network of patterns.

Level of Genericity

• Current State: The pattern addresses a problem that is so general that it does not apply only to the ESI sector.

• Desired State: The pattern addresses an ESI problem that is general enough to be applicable in other companies.

• Action: The pattern may need to be specialised into a group of more detailed patterns that are applicable to the ESI sector in terms of the experiences in the ELEKTRA project. It may however, be so that domain is such that the pattern is applicable both for the ESI sector and more generally.

Grade of understandability

• Current State: The pattern is incomprehensible to decision makers, domain experts and those to be affected.

• Desired State: The pattern is easy for decision-makers, domain experts and those to be affected, to comprehend.

• Action: Use the concepts/terms of the decision-makers, domain experts and those to be affected, by involving these people directly in the formulation of the pattern.

External compatibility

• Current State: The pattern does not take into account differences in national, business and organisational cultures and ways of working amongst identified future external users of the pattern. There is no explanation of the underlying conditions or context on which the patterns are based.

• Desired State: The pattern has taken into account differences in national, business and organisational cultures and ways of working amongst identified future external users of the pattern. This entails a full explanation of the underlying conditions on which the patterns are based.

• Action: Since one of the aims of creating patterns is to reuse them throughout the ESI sector in Europe, the patterns should be made compatible to these differing environments. Concepts and terms that are likely to be taken for granted need to be explained. Reference to national laws and regulations must be explained. Customs and behaviour, which may be of a national or organisational character, should either be included and explained, or excluded, if possible. By following these guidelines, the patterns will be more comprehendable and useful to the intended users.

Level of experience in their (partial) use

• Current State: The pattern has never been used.

• Desired State: There has been a great deal of experience within the business in the (implicit or explicit) use of this pattern.

• Action: While the desired state is preferable, it is possible that the pattern has been used only implicitly. Another situation is that the pattern has been used only in terms of decomposed parts of it. The candidate pattern is a composition of these, or is a generalisation.

Economic feasibility of the proposed solutions

• Current State: Proposed solution is not likely to be feasible. Organisational support will be difficult to obtain. The necessary resources will not be made available. The existing difficult social and/or political climate would make an implementation impossible.

• Desired State: Proposed solution is relatively easy to implement. Organisational support exists in terms of sufficient resources as well as managerial support. The solution is politically and socially acceptable .

• Action: Management needs to be informed of the rationale and consequences of implementation of the pattern. If social or political problems exist it may be necessary to either reduce the impact of the pattern or abandon it in favour of internal harmony.