On 9/03/2016, at 6:30 AM, Amir Hadanny
1.I'm using the Experimenter running 10 iterations of 5 different algorithms on 10
Fold cross validation on my data set. Is there any way to compare the AUC's in a
non-parametric way (rather than the T-test). for example : De-Long method or sign-rank?
I recieved a reviewer comment which specified the AUCs on my data probably do not have
normal ditrubations so T-Test shouldn't be used.
The Experimenter only implements the t-test (more precisely, it implements two versions:
the standard t-test, and the corrected resampled t-test [the latter is the default]). You
will need to export the data and perform the test using some other tool.
2. I was advised to use a Penelized Logistic
Regression (I.E LASSO). Is there anyway to do so in Weka? I know there is the Ridge
parameter in Weka's logistic regression , but is it good enough? is the only way to do
that will be R? I'm afraid of this option - since then i won't be able to compare
the AUCs I got from Weka's experimenters and the R's LASSO results…
Logistic in WEKA only supports an L_2 penalty on the weights (i.e., ridge regression).
Lasso regression corresponds to an L_1 penalty.
Using the RPlugin for WEKA 3.7, you can use (almost all) all the R regression and
classification algorithms in the MLR package by applying the MLRClassifier in WEKA, so you
can directly compare algorithms from R and WEKA. The schemes implemented in MLR are listed
You could use the classif.glmnet or classif.cvglmnet in MLRClassifier.
Also, the LibLINEAR package for WEKA 3.7 has an option for building logistic regression
models with an L_1 penalty:
Set type of solver (default: 1)
for multi-class classification
0 -- L2-regularized logistic regression (primal)
1 -- L2-regularized L2-loss support vector classification (dual)
2 -- L2-regularized L2-loss support vector classification (primal)
3 -- L2-regularized L1-loss support vector classification (dual)
4 -- support vector classification by Crammer and Singer
5 -- L1-regularized L2-loss support vector classification
—> 6 -- L1-regularized logistic regression