Title: What Do Machine Learning Models Actually Learn from the Kepler Object of Interest Catalogue? Separating Label Leakage from Label-Set Choice
Authors: Dang Duc Thanh, Pham Hoang Bao Chau, Nguyen Minh Duc, Trinh Thanh Do
Volume: 10
Issue: 9
Pages: 237-247
Publication Date: 2026/09/28
Abstract:
Machine learning studies of the Kepler Object of Interest (KOI) catalogue report classification accuracies ranging from below .80 to above .98. The spread is usually read as a difference in method. Two other things differ across these studies and are rarely stated: which of the catalogue's columns the model was given, and which disposition labels it was asked to separate. Several columns are not astrophysical measurements at all - the four Robovetter false-positive flags, the disposition score koi_score, and the pipeline disposition koi_pdisposition each encode the outcome of NASA's automated vetting, which is also the source of the class label. And the CANDIDATE disposition, which marks a signal whose vetting is unfinished rather than a class of object, is included by some studies and excluded by others. This paper separates the two. Three classifiers - random forest, XGBoost, and a one-dimensional convolutional network - were first trained on an identical 36-feature set drawn from the cumulative KOI table (N = 9,564) with every vetting-derived column removed, with class imbalance addressed by SMOTE inside the training partition. On an independent test partition (n = 1,913), random forest and XGBoost each attained an accuracy of .79 (macro F1 = .75 and .76) and the convolutional network .75 (macro F1 = .73); one-versus-rest ROC analysis for the CONFIRMED label returned AUC values of .966 for XGBoost, .961 for random forest and .959 for the convolutional network. Errors were concentrated almost entirely in the CANDIDATE label, whose recall under XGBoost was .55 against .85 for CONFIRMED and .88 for FALSE POSITIVE. The XGBoost configuration was then re-estimated across three feature sets and two label sets on the same partition, seed and hyperparameters. Restoring the four false-positive flags to the three-class problem raises accuracy from .793 to .907; restricting the same 36 features to the binary CONFIRMED-versus-FALSE-POSITIVE problem raises it to .939; and the cell that matches the design of the highest published catalogue-based figure reaches .994, above the 98% that study reported. Averaged over the two orders in which the changes can be made, 11.7 of those 20.1 accuracy points are attributable to the label set and 8.5 to the vetting columns. Accuracy on the KOI catalogue is therefore not interpretable unless both the feature set and the label set are disclosed.