On 7/12/11 5:34 AM, Javier Perez Florido wrote:
I'd like to know how a ROC curve is built in Weka for classifiers such
as Decision Trees, Multilayer Perceptrons and Random Forests. I've tried
to find such information on the web, but no success. I don't know if
some parameter during training is varied to get such curve (a different
model is built/trained for each point).
But it seems that the curve is built using the following procedure
regardless of the classifier (two class problem): predictions are
collected from the classifier for each test instance.This set of
predictions are then sorted in descending order of probability assigned
to the positive class. The ROC curve is generated by stepping down this
list (starting from the top) and counting how many TPs and FPs are
contained in the list up to that point. The threshold at each point is
just the probability of the true class at that point in the sorted list.
That is, a single model is built.
Which of these two possibilities is used by Weka?
The second method is the one that Weka uses to generate an ROC curve.