Analysis and Visualization of Classifier Performance: Comparison under Imprecise Class and Cost Distributions

Applications of inductive learning algorithms to real world data mining problems have shown repeatedly that using accuracy to compare classifiers is not adequate because the underlying assumptions rarely hold.  We present a method for the comparison of classifier performance that is robust to imprecise class distributions and misclassification costs.  The ROC convex hull method combines techniques from ROC analysis, decision analysis and computational geometry, and adapts them to the particulars of analyzing learned classifiers.  The method is efficient and incremental, minimizes the management of classifier performance data, and allows for clear visual comparisons and sensitivity analyses.