Hi Eibe,

As follow-up, does the RandomCommittee MetaClassifier (or any of the ensemble methods) compute a variance?

Thank you,

Peter

On Jan 1, 2018 11:02 PM, "Eibe Frank" <eibe@waikato.ac.nz> wrote:

The Explorer, the KnowledgeFlow, and the CLI do not have the option to perform repeated k-fold cross-validation. Only the Experimenter does, and it uses averaging: if you run k-fold cross-valiation, repeated r times, it will collect r * k separate results and compute the average and standard deviation from these r *k results.

 

Cheers,

Eibe

 

From: Edward Wiskers
Sent: Tuesday, 2 January 2018 4:52 PM


To: Weka machine learning workbench list.
Subject: Re: [Wekalist] Confusion matrix in the Explorer forcross-validationmode

 

Thank Eibe. 

 

How about performing k-fold cross-validation for N times (say 10 times performing cross-validation) is the resulting confusion matrix is the sum of all the 100 confusion matrix? (same concept you described previously)

 

Edward 

 

On 2 Jan 2018 11:47 a.m., "Eibe Frank" <eibe@waikato.ac.nz> wrote:

The Explorer uses pooling to compute performance statistics. The command-line interface and the KnowledgeFlow use the same approach. This makes it easier to draw ROC curves, etc.

 

Due to this approach, the confusion matrix is computed from the pooled results for the k test sets in a k-fold cross-validation.

 

One disadvantage of this approach is that it does not produce estimates of variance. The Experimenter uses averaging rather than pooling and it will give you estimates of variance.

 

Cheers,

Eibe

 

From: Edward Wiskers
Sent: Monday, 1 January 2018 1:40 PM
To: Weka machine learning workbench list.
Subject: Re: [Wekalist] Confusion matrix in the Explorer for cross-validationmode

 

Thanks Peter. 

 

Why sum not average?

 

 

 

Edward 

 

On 1 Jan 2018 7:03 a.m., "Peter Reutemann" <fracpete@waikato.ac.nz> wrote:

> In the Explorer, is the confusion matrix computed as the sum or average of
> the 10 confusion matrices (ie, sum or average of the confusion matrices of
> each of the 10 folds)?

Sum - from the 10 test folds.

Cheers, Peter
--
Peter Reutemann
Dept. of Computer Science
University of Waikato, NZ
+64 (7) 858-5174
http://www.cms.waikato.ac.nz/~fracpete/
http://www.data-mining.co.nz/
_______________________________________________
Wekalist mailing list
Send posts to: Wekalist@list.waikato.ac.nz
List info and subscription status: https://list.waikato.ac.nz/mailman/listinfo/wekalist
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html

 


_______________________________________________
Wekalist mailing list
Send posts to: Wekalist@list.waikato.ac.nz
List info and subscription status: https://list.waikato.ac.nz/mailman/listinfo/wekalist
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html

 


_______________________________________________
Wekalist mailing list
Send posts to: Wekalist@list.waikato.ac.nz
List info and subscription status: https://list.waikato.ac.nz/mailman/listinfo/wekalist
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html