This is an interesting question and there is a very recent paper that addresses exactly
Bergmeir, C., Hyndman, R. J., & Koo, B. (2018). A note on the validity of
cross-validation for evaluating autoregressive time series prediction. Computational
Statistics & Data Analysis, 120, 70-83.
Assuming you set "Time stamp” to “<None>", and do not use overlay data or
periodic attributes, the autoregressive model class considered in this paper is exactly
the one used by time series forecasting in WEKA.
The paper shows that under some assumptions using k-fold CV is suitable for evaluating
autoregressive time series models. One of these assumptions is that the time series is
stationary (and ergodic).
If your time series is not stationary, you may be able to turn it into a stationary one.
Neither CVParameterSelection, GridSearch, or MultiSearch currently support a percentage
split evaluation that preserves the order of the data, so k-fold CV is currently the only
option for parameter tuning of learning algorithms in WEKA's time series forecasting
On 1/08/2018, at 9:20 AM, lumio
Am I right that CVparameterSelection is not right choice for search of model
parameters in case of time series forecast ? Because what I observed is
small rmse of train set and big of test set. What is best practice in this
PS: How Auto-Weka manages this ? Is it appropriate for time series ?
Sent from: http://weka.8497.n7.nabble.com/
Wekalist mailing list
Send posts to: Wekalist(a)list.waikato.ac.nz
List info and subscription status: https://list.waikato.ac.nz/mailman/listinfo/wekalist
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html