The Performance Function

This function is called for comparing the observed and inferred outputs for a data sample. It has two main arguments: the output number to be tested, and the data set reference, which can be either a file reference or a pointer on an array. The performance depends on the output classification flag.

The function writes a result file containing an array, with as many lines as there are items in the data set. The number of columns depends on the data file and on the output nature (defuzzification type and classification option).

If the data file has an output column, the FIS performance ith respect to a given data set is estimated by three complementary indices: maximum error, performance index and coverage index (which depends on the blank threshold).

• Max error: It is computed as the maximum absolute value, over the whole set of examples, between the observed output, $y_i$ for the $ith$ example, and the the one inferred by the system, $\widehat{y_i}$.

• Coverage index: Some data items may match rules with a low degree. A data item is called blank if its cumulated matching degree on the rule base is lower than a given threshold. This threshold is a user parameter, with a default value of 0.2. Blank items do not appear in the performance results. Their total number is used to define a coverage index, equal to $$CI = 1 - \frac{I}{n}$$. I is the number of blank items and n the total number of items in the data file.

  Note: The matching degree highly depends on the rule premise conjunction operaror (min or product). The coverage index can decrease very fast if the number of variables is high, and the blank threshold must be chosen accordingly.

• Performance index: It depends upon the output type. For a crisp output using the classification flag, it corresponds to the number of misclassified items. Otherwise it is parameterized using the SetErrorIndex function, and can be one of PI, RMSE ou MAE (see glossary in section 1. This performance index is returned by the function.

Example:

Figure 6 displays the result file that corresponds to a FIS having conjunctive rules, a 3 MF fuzzy output, and the classification option.
In that case, the file has 9 columns for each example: observed value, inferred value after defuzzification, alarm flag (0 if everything is OK, 3 columns MF1, MF2, MF3 with the example output membership degree to each of them, current error, Bl flag (0 for a non blank example, 1 for a blank example), and cumulated error.

Figure 6: Inference result file - 3 MF fuzzy output, classification option

General Format

The first line of the result file is a column header. The possible labels are defined in the file fis.h:

• "OBS": The observed output, part of the data set
• "INF": The inferred output
• "Al": Alarm risen while inferring (see below)
• "ClINF": Inferred class label, for crisp output and classification flag
• "CLAl": Alarm risen while inferring (see below)
• "Err": Difference between inferred and observed output
• "Bl": Blank flag, 1 if this item is considered as blank, o otherwise
• "CErr2": square cumulated error over the previous examples.

WARNING: The cumulated error does not take into account the blank examples.

The first column holds the observed output, if it is available in the data file, as inference can also be done without an observed output.
The second one gives, when the observed output value is available, the difference between observed and inferred values.
A variable number of columns follows, depending on the output nature.

Output	Classif	Defuzzification	Field #	Available fields
crisp	yes	sugeno	4	infered v.	Alarm	infered class	Alarm
crisp	no	sugeno	2	inferred v.	Alarm
crisp		MaxCrisp	2	inferred v.	Alarm
fuzzy	yes	sugeno/area	n+2	infered v.	Alarm	$\mu_1,\mu_2\ldots\mu_n$
fuzzy	no	sugeno/area	2	infered v.	Alarm
fuzzy	yes	MeanMax	n+2	infered v.	Alarm	$\mu_1,\mu_2\ldots\mu_n$
fuzzy	no	MeanMax	2	infered v.	Alarm

Note: $\mu_1,\mu_2\ldots\mu_n$ are available if the example activates at least one rule. For a fuzzy output defuzzified with sugeno ou area operators, they are equal to the inferred output value membership degrees to MF $1,2\ldots n$. For a fuzzy output defuzzified with a MeanMax operator, or for a crisp output obtained through a MaxCrisp operator, they represent the matching degree of all possible outputs: MF values for a fuzzy output, or real values for a crisp output.

The column Bl is 1 if the item is inactive, 0 otherwise. Finally, the last column gives the current value of the performance index.

Available alarm values:

Integer values which depend on the defuzzification operator:

• NOTHING (Value 0): All types. Everything is normal.

• NO_ACTIVE_RULE (Value 1): All types. The example does not fire any rule of the rule base.

• AMBIGUITY (Value 2): SugenoClassif (Crisp output) and MaxCrisp. The difference between the two main classes is less than a threshold (default value AMBIGU_THRES = 0.1).

• NON_CONNEX_AREA (Value 3): WeArea - Set when the area defined by the fired fuzzy sets (threshold set to MIN_THRES = 0.1 by default) is non connex.

• NON_CONNEX_SEGMENT (Value 4): MeanMax - Set when the max corresponds to two fuzzy sets (with a tolerance threshold set by default to EQUALITY_THRES = 0.1) and the resulting segment is non connex.

The performance function also computes the coverage index, which depends on the minimum matching degree given as an argument.

General file format for implicative rules

If the output on which the performance is calculated is implicative, new columns are introduced in the perf file, in the following order.

• $k$ columns $MF_1$, ...$MF_k$ right after column OBS. k is the number of output fuzzy sets. The value written in each column is the membership degree of the observed output to the corresponding fuzzy set.
• $k$ columns $MF_1$, ...$MF_k$ right after column Al. k is the number of output fuzzy sets. The value written in each column is the membership degree of the inferred output to the corresponding fuzzy set.
• MINK, which contains the inferred possibility distribution minimum kernel value.
• MAXK, which contains the inferred possibility distribution maximum kernel value.
• MINS, which contains the inferred possibility distribution minimum support value.
• MAXS, which contains the inferred possibility distribution maximum support value.
• MATCH, which gives the matching degree between the observed output and the inferred possibility distribution.