Logical rules extracted from data

Look at datasets to find more results obtained using different classifiers.

Medical:  Appendictis | Breast cancer (Wisconsin) | Cleveland heart disease | Diabetes | Hepatitis | Hypothyroid | Ljubljana cancer | Statlog Heart |

Other:  Ionosphere | Iris flowers | Mushrooms | Monk 1 | Monk 2 | Monk 3 | Satellite image dataset (Statlog version) | NASA Shuttle | Sonar | Vovel |

Confusion matrices: column labels refer to the true class, row labels to the assigned class, for medical data healthy cases are first.

Appendicitis.

106 vectors, 8 attributes, two classes (88 acute +18 other),
obtained from Shalom Weiss.
Attribute names: WBC1, MNEP, MNEA, MBAP, MBAA, HNEP, HNEA

Rules found using PVM
Accuracy 89.6% in leave-one-out, 91.5% overall

C1: MNEA > 6600  OR  MBPA > 11
C2: ELSE

Rules found using C-MLP2LN, no optimization
Accuracy 89.6% in leave-one-out, 91.5% overall

C1: MNEA > 6650 OR MBPA > 12
C2: ELSE

Second neuron gets 3 more cases correctly using 2 rules, but we treat it as noise rather than an interesting rare case.
Using L-units another set of rules is generated with the overall 89.6% accuracy (11 errors).

C1: WBC1 > 8400 OR MBPA >= 42
C2: ELSE

C4.5 generates 3 rules with overall 91.5% accuracy. It may also generate 7 rules for 97.2% accuraccy but this is strong overfitting, with each rule classifying only 1-2 cases.

Summary of accuracy (%) and references

S.M. Weiss, I. Kapouleas, "An empirical comparison of pattern recognition, neural nets and machine learning classification methods", in: J.W. Shavlik and T.G. Dietterich, Readings in Machine Learning, Morgan Kauffman Publ, CA 1990
H.J. Hamilton, N. Shan, N. Cercone, RIAC: a rule induction algorithm based on approximate classification, Tech. Rep. CS 96-06, Regina University 1996.
Duch W, Adamczak R, Grąbczewski K, A new methodology of extraction, optimization and application of crisp and fuzzy logical rules. IEEE Transactions on Neural Networks 12 (2001) 277-306

Wisconsin breast cancer.

From UCI repository, 699 cases, 9 attributes (1-10 integer values),
two classes, 458 benign (65.5%) & 241 malignant (34.5%).
For 16 instances one attribute is missing.

Attributes: from original database remove F0, id. number (warining: in some papers original feature numbers are given).
F1: Clump Thickness 1 - 10
F2: Uniformity of Cell Size 1 - 10
F3: Uniformity of Cell Shape 1 - 10
F4: Marginal Adhesion 1 - 10
F5: Single Epithelial Cell Size 1 - 10
F6: Bare Nuclei 1 - 10
F7: Bland Chromatin 1 - 10
F8: Normal Nucleoli 1 - 10
F9: Mitoses 1 - 10

C-MLP2LN results:

Rules S1: Single rule: IF f2 = [1,2] then benign else malignant

            Original class.
Calculated
        1      417     12
        2      41     229

Accuracy: 646 correct (92.42%), 53 errors; Sensitivity=0.9720, Specificity=0.8481

Rules S2: 5 rules for malignant, overall accuracy of 96%.

3 benign cases wrongly classified as malignant and 25 malignant cases wrongly classified as benign.

Rules S3: 4 malignant rules, overall accuracy of 97.7%, confusion matrix

3 benign cases wrongly classified as malignant and 25 malignant cases wrongly classified as benign.

Rules S4: Optimized rules: 1 benign vector classified as malignant (rule 1 and rule 5, the same vector).
ELSE condition makes 6 errors, giving 99.00% overall accuracy:

Other solutions: 100% reliable rules rejecting 51 cases (7.3%) of all vectors.
For malignant class these rules are:

For the benign cases rules are: NOT (R5 OR R6 OR R7 OR R8), where:

Summary of results (rules discovered for the whole data set).
 

Duch W, Adamczak R, Grąbczewski K, Żal G, Hybrid neural-global minimization method of logical rule extraction. Journal of Advanced Computational Intelligence 3 (5): 348-356.
Duch W, Adamczak R, Grąbczewski K, A new methodology of extraction, optimization and application of crisp and fuzzy logical rules. IEEE Transactions on Neural Networks 12 (2001) 277-306
H.J. Hamilton, N. Shan, N. Cercone, RIAC: a rule induction algorithm based on approximate classification, Tech. Rep. CS 96-06, Regina University 1996.

Papers on a smaller (569 cases) Wisconsin breast cancer dataset are on the O.L. Mangasarian page.

Cancer (Ljubljana data)

From UCI repository (restricted):  286 instances, 201 no-recurrence-events (70.3%), 85 recurrence-events (29.7%);
9 attributes, between 2-13 values each, 9 missing values

Rules found using PVM: 70% for training, 30% for test
Accuracy 77.4% train, 77.1% test

C1: Involved Nodes > 0 & Degree_malig = 3
C2: ELSE

C-MLP2LN more accurate rules: 78% overall accuracy
R1: deg_malig=3 & breast=left & node_caps=yes
R2: (deg_malig=3 OR breast=left) & NOT inv_nodes=[0,2] & NOT age=[50,59]

Michalski,R.S., Mozetic,I., Hong,J., & Lavrac,N. (1986). The Multi-Purpose Incremental Learning System AQ15 and its Testing Application to Three Medical Domains.  In Proceedings of the Fifth National Conference on Artificial Intelligence, 1041-1045, Philadelphia, PA: Morgan Kaufmann.

Clark,P. & Niblett,T. (1987). Induction in Noisy Domains.  In: Progress in Machine Learning (from the Proceedings of the 2nd European Working Session on Learning), 11-30, Bled,  Yugoslavia: Sigma Press.

CART & PVM 77.4% train, 77.1% test; S.M. Weiss, I. Kapouleas. An empirical comparison of pattern recognition, neural nets and machine learning classification methods, in: J.W. Shavlik and T.G. Dietterich, Readings in Machine Learning, Morgan Kauffman Publ, CA 1990

Duch W, Adamczak R, Grąbczewski K (1997) Extraction of crisp logical rules using constrained backpropagation networks, International Conference on Artificial Neural Networks (ICNN'97), Houston, 9-12.6.1997, pp. 2384-2389
Duch W, Adamczak R, Grąbczewski K, A new methodology of extraction, optimization and application of crisp and fuzzy logical rules. IEEE Transactions on Neural Networks 12 (2001) 277-306

Hepatitis.

From UCI repository, 155 vectors, 19 attributes, 13 binary, other integer, class is first.
Two classes, 32 die (20.6%), 123 live (79.4%)
Missing values (here F1=class): F4(1), F6(1), F7(1), F8(1), F9(10), F10(11), F11(5), F12(5), F13(5), F14(5), F15(6), F16(29), F17(4), F18(16), F19(67)

C-MLP2LN rule, overall accuracy 88.4%, using F2=age, F13=Ascites, F15=bilirubin, F20=histology,

R1: age > 52 & bilirubin > 3.5
R2: histology=yes & ascites=no & age = [30,51]

C-MLP2LN, lignuistic variables from L-units, overall accuracy 96.1%, looks good but uses F19=protime which has missing values in almost half of the cases.

age >= 30 & sex=male & antivirals=no & protime <= 50

Duch W, Adamczak R, Grąbczewski K, A new methodology of extraction, optimization and application of crisp and fuzzy logical rules. IEEE Transactions on Neural Networks 12 (2001) 277-306

Cleveland heart disease.

From UCI repository, 303 cases, 13 attributes (4 cont, 9 nominal), many missing values.
2 (no, yes) or 5 classes (no, degree 1, 2, 3, 4).
Class distribution: 164 (54.1%) no, 55+36+35+13 yes (45.9%) with disease degree 1-4.
 
C-MLP2LN simplified rules 85.5% overall accuracy. Rules for healthy class:

R1:  (thal=0 OR thal=1) &  ca=0.0     (88.5%)
R2:  (thal=0 OR ca=0.0) & cp NOT 2 (85.2%)
ELSE  sick  (89.2%)

Statlog Heart disease.

13 attributes (extracted from 75), no missing values.
270=150+120 observations selected from the 303 cases (Cleveland Heart).

Results without risk matrix

Duch W, Adamczak R, Grąbczewski K, A new methodology of extraction, optimization and application of crisp and fuzzy logical rules. IEEE Transactions on Neural Networks 12 (2001) 277-306

Diabetes.

From UCI repository, dataset "Pima Indian diabetes":
2 classes, 8 attributes, 768 instances, 500 (65.1%) healthy, 268 (34.9%) diabetes.

F2 is "Plasma glucose concentration (2 hours oral glucose tolerance) test"
F6 is "Body mass index (weight in kg/(height in m)^2)"

1 rule from SSV, overall accuracy 74.9%, Sensitivity=45.5, Spec.=90.6

IF F#2 > 144.5 then diabetes, else healthy

Rule from C-MLP2LN with L-units, overall accuracy 75%

IF ( F2<=151 AND F6<=47 ) THEN healthy, else diabetes

2 rules from SSV, overall accuracy 76.2%, Sensitivity=60.8, Spec.=84.4

IF F#2 > 144.5 OR (F#2 > 123.5 AND F#6 > 32.55) then diabetes, else healthy

Estimation of accuracy (4 leaves in SSV): average of 10 runs, each 10xCV, accuracy 75.2 ±0.6

Results from crossvalidation.

Duch W, Adamczak R, Grąbczewski K, A new methodology of extraction, optimization and application of crisp and fuzzy logical rules. IEEE Transactions on Neural Networks 12 (2001) 277-306

Hypothyroid.

Thyroid, From UCI repository, dataset "ann-train.data":
3772 learning and 3428 testing examples;
Training: 93+191+3488 or 2.47%, 5.06%, 92.47%
Test: 73+177+3178 or 2.13%, 5.16%, 92.71%
21 attributes (15 binary, 6 continuous); 3 classes

C-MLP2LN rules (all values of continuous features are multiplied here by 1000)

Initial rules:

primary hypothyroid:  TSH>6.1  &  FTI <65
compensated       :  TSH > 6 & TT4<149 & On_Tyroxin=FALSE & FTI>64 & surgery=False
ELSE normal

Optimized more accurate rules: 4 errors on the training set (99.89%), 22 errors on the test set (99.36%)

primary hypothyroid:  TSH>30.48  &  FTI <64.27  (97.06%)
primary hypothyroid:  TSH=[6.02,29.53]  &  FTI <64.27 & T3< 23.22 (100%)
compensated            :  TSH > 6.02 & FTI>[64.27,186.71] & TT4=[50, 150.5) & On_Tyroxin=no & surgery=no  (98.96%)
no hypothyroid         :  ELSE   (100%)

3 crisp logical rules using TSH, FTI, T3, on_thyroxine, thyroid_surgery, TT4  give 99.3% of accuracy on the test set.
Duch W, Adamczak R, Grąbczewski K, A new methodology of extraction, optimization and application of crisp and fuzzy logical rules. IEEE Transactions on Neural Networks 12 (2001) 277-306

Iris flowers

150 vectors, 50 in each class: setosa, virginica, versicolor
PL=x3=Petal Length;  PW=x4=Petal Width

PVM Rules: accuracy 98% in leave-one-out and overall
 

C-MLP2LN rules:

7 errors, overall 95.3% accuracy

Higher accuracy: overall 98%

100% reliable rules reject 11 vectors, 8 virginica and 3 versicolor:

Summary:

References:
S.M. Weiss, I. Kapouleas, "An empirical comparison of pattern recognition, neural nets and machine learning classification methods", in: J.W. Shavlik and T.G. Dietterich, Readings in Machine Learning, Morgan Kauffman Publ, CA 1990
N. Kasabov, Connectionist methods for fuzzy rules extraction, reasoning and adaptation. In: Proc. of the Int. Conf. on Fuzzy Systems, Neural Networks and Soft Computing, Iizuka, Japan, World Scientific 1996, pp. 74-77
Duch W, Adamczak R, Grąbczewski K, A new methodology of extraction, optimization and application of crisp and fuzzy logical rules. IEEE Transactions on Neural Networks 12 (2001) 277-306
C. Browne, I. Duntsch, G. Gediga, IRIS revisited: A comparison of discriminant and enhanced rough set data analysis. In: L. Polkowski and A. Skowron, eds. Rough sets in knowledge discovery, vol. 2. Physica Verlag, Heidelberg, 1998, pp. 345-368
D. Nauck, U. Nauck and R. Kruse, Generating Classification Rules with the Neuro-Fuzzy System NEFCLASS. Proc. Biennial Conf. of the North American Fuzzy Information Processing Society (NAFIPS'96), Berkeley, 1996
S.K. Halgamuge and M. Glesner, Neural networks in designing fuzzy systems for real world applications. Fuzzy Sets and Systems 65:1-12, 1994

Mushrooms

8124 instances, 4208 (51.8%) edible and 3916 (48.2%) poisonous;
22 attributes (all symbolic): cap shape (6, e.g.. bell, conical,flat...), cap surface (4), cap color (10), bruises (2), odor (9), gill attachment (4), gill spacing (3), gill size (2), gill color (12), stalk shape (2), stalk root (7, many missing values), surface above the ring (4), surface below the ring (4), color above the ring (9), color below the ring (9), veil type (2), veil color (4), ring number (3), spore print color (9), population (6), habitat (7).
Together 118 logical input values.
2480 missing values for attribute 11

Spanish version of these results!

C-MLP2LN rules:

Disjunctive rules for poisonous mushrooms, from most general to most specific:

Alternative R4' rule:  population=clustered.AND.cap_color=white

These rule involve 6 attributes (out of 22).  Rule 1 may be replaced by:

odor = creosote.OR.fishy.OR.foul.OR.musty.OR.pungent.OR.spicy

Re1: odor=(almond.OR.anise.OR.none).AND.spore-print-color=NOT.green

Other methods:

References:

Duch W, Adamczak R, Grabczewski K (1996) Extraction of logical rules from training data using backpropagation networks, in: Proc. of the The 1st Online Workshop on Soft Computing, 19-30.Aug.1996, pp. 25-30, available on-line at: http://www.bioele.nuee.nagoya-u.ac.jp/wsc1/
Duch W, Adamczak R, Grabczewski K, Ishikawa M, Ueda H, Extraction of crisp logical rules using constrained backpropagation networks - comparison of two new approaches, in: Proc. of the European Symposium on Artificial Neural Networks (ESANN'97), Bruge, Belgium 16-18.4.1997, pp. 109-114
Duch W, Adamczak R, Grąbczewski K, A new methodology of extraction, optimization and application of crisp and fuzzy logical rules. IEEE Transactions on Neural Networks 12 (2001) 277-306
Schlimmer,J.S. (1987). Concept Acquisition Through Representational Adjustment (Technical Report 87-19), Doctoral disseration, Department of Information and Computer Science, University of California, Irvine.
Iba,W., Wogulis,J., & Langley,P. (1988).  Trading off Simplicity and Coverage in Incremental Concept Learning. In Proceedings of  the 5th International Conference on Machine Learning, 73-79,  Ann Arbor, Michigan: Morgan Kaufmann.

Monk 1

Original rule is: head shape = body shape OR jacket color = red

C-MLP2LN:
100% accuracy with 4 rules + 2 exception, 14 atomic formulae.

Other systems: see the original paper:
S. Thrun, J. Bala, E. Bloedorn, I. Bratko, B. Cestnik, J. Cheng, K. De Jong, S. Dzeroski, R. Hamann, K. Kaufman, S. Keller, I. Kononenko, J. Kreuziger, R.S. Michalski, T. Mitchell, P. Pachowicz, B. Roger, H. Vafaie, W. Van de Velde, W. Wenzel, J. Wnek, and J. Zhang.
The MONK's problems: A performance comparison of different learning algorithms. Technical Report CMU-CS-91-197, Carnegie Mellon University, Computer Science Department, Pittsburgh, PA, 1991.

Monk 2

Original rule: exactly two of the six features have their first values

C-MLP2LN:
100% accuracy with 16 rules and 8 exceptions, 132 atomic formulae.
Other systems: see the Thrun et al. original paper: The MONK's problems

Monk 3

Original rule:

NOT (body shape = octagon OR jacket color = blue) OR (holding = sward AND jacket color = green)
was corrupted by 5% noise.

C-MLP2LN:
100% accuracy with 33 atomic formulae.
Other systems: see the Thrun et al. original paper: The MONK's problems
Duch W, Adamczak R, Grąbczewski K, A new methodology of extraction, optimization and application of crisp and fuzzy logical rules. IEEE Transactions on Neural Networks 12 (2001) 277-306

Comparison of results:

NASA Shuttle

Training set 43500, test set 14500, 9 attributes, 7 classes
Approximately 80% of the data belongs to class 1.

Rules obtained from FSM, without optimization:

Rules obtained from FSM, without optimization:

17 optimized FSM rules make only 3 errors on the training set (99.99\% accuracy), leaving 8 vectors unclassified, and no errors on the test set but leaving 9 vectors unclassified (99.94\%). After Gaussian fuzzification of inputs (very small, 0.05\%) only 3 errors and 5 unclassified vectors are obtained for the training and 3 vectors are unclassified and 1 error is made (with the probability of correct class for this case close to 50\%) for the test set.

32 rules from SSV gave even better results: 100\% correct on the training and only 1 error on the test set.

Satellite image dataset (STATLOG version)

Training 4435 test 2000 cases, 36 semi-continous [0 to 255] attributes (= 4 spectral bands x 9 pixels in neighbourhood) and 6 decision classes: 1,2,3,4,5 and 7 (class 6 has been removed because of doubts about the validity of this class).

Duch W, Adamczak R, Grąbczewski K, A new methodology of extraction, optimization and application of crisp and fuzzy logical rules. IEEE Transactions on Neural Networks 12 (2001) 277-306

Ionosphere

200 training, 150 test cases, 34 continuous attributes, 2 classes

N. Shang, L. Breiman, ICONIP'96, p.133
David Aha: k-NN+C4+IB3 (Aha \& Kibler, IJCAI-1989), IB3 parameter settings: 70% and 80% for acceptance and dropping respectively.
RIAC, C4.5 from: H.J. Hamilton, N. Shan, N. Cercone, RIAC: a rule induction algorithm based on approximate classification, Tech. Rep. CS 96-06, Regina University 1996.

Sonar

208 cases, 60 continuous attributes, 2 classes
From the CMU benchmark repository

Vovel

528 training, 462 test cases, 10 continous attributes, 11 classes
From the CMU benchmark repository

N. Shang, L. Breiman, ICONIP'96, p.133, made 10xCv instead of using the test set.

Other Data

DNA-Primate splice-junction gene sequence

Links to other Duch-Lab projects, many talks and to other papers on various subjects.

Maintained by Wlodzislaw Duch.