kfold.pl -- k-fold cross-validation

This module contains predicates for estimating prediction accuracy using k-fold cross validation or a random train/test split. A ``fold'' in this context refers to a set of interpretations, several cases can be included in a given interpretation. Common testing strategies like leave-one-university-out in WebKB or leave-one-research-group-out in UW-CSE are naturally obtained by setting k to the number of interpretations in the domain. The module contains support for stratified k-fold (where strata are specified by a user-defined predicate) and prepartitioned k-fold where folds are read from file.

It is recommended that any empirical evaluation of kLog is run using predicates in this module since they all save experimental results on disk in a structured form (see below).

Usage
See predicates kfold/4, random_split/4, fixed_split/6, prepartitioned_kfold/4, stratified_kfold/5.
Results
Results are saved in a structured way in the file system. kfold/4 and random_split/4 create a results directory named after a SHA-1 hash of: (1) current klog flags, (2) domain traits, (3) train/test specification, e.g.
PWD/results/e789ba9abfa5e1ae77ec0f481dab9971f343df35

where PWD is the current working directory. This ensures that different experiments run with different parameters (such as kernel hyperparameters, regularization, etc) or background knowledge will be kept separate and can coexist for subsequent analysis (e.g. model selection).

For k-fold cross-valitation, rhe results directory is structured into k sub-directories, one for each fold. These contain the following files:

auc.log
hash_key.log      text used to generate the hash key
output.log        predicted margins for each case in the form target margin # case term
output.yyy        same after stripping the case term
output.yyy.pr     recall-precision curve
output.yyy.roc    ROC curve
output.yyy.spr    precision points calculated at 100 recall points between 0 and 1.
test.txt          list of test interpretations in this fold
train.txt         list of training interpretations in this fold

@tbd Prediction on attributes (multiclass, regression)

@credit AUCCalculator by Jesse Davis and Mark Goadrich

author
- Kurt De Grave, Paolo Frasconi
stratified_kfold(+Signature, +K, +Models, +FeatureGenerator, +StratumFunctor) is det
Perform a stratified k-fold cross validation. The target Signature defines the learning task(s). K is the numnber of folds. Models is a (list of) model(s) to be trained and FeatureGenerator defines the used kernel. StratumFunctor is the name of a user-declared Prolog predicate of the form stratum(+Interpretation,-Stratum); it should unify Stratum (e.g. the category for classification) with the stratum of the given Interpretation.
prepartitioned_kfold(+Signature, +Models, +FeatureGenerator, +DefinitionFile) is det
Perform a repeated kfold cross-validation reading folds from a file containing facts of the form test_int(Trial,Fold,IntId). Each trial is saved into a separate subfolder (which in turns contains as many folders as folds). This is useful for comparing against other methods while keeping the same partitions, so that paired statistical tests make sense.
random_split(+Sig:atom, +Fraction:number, +Models:{atom}, +FGenerator:atom) is det
Train and test Model(s) on a given Fraction of existing cases (rest used for test). TrainFraction should be either a fraction in (0.0,1.0) as a float or in (0,100) as an integer.
fixed_split(+Sig:atom, +Comment:atom, +Train:[atom], +Test:[list], +Models:{atom}, +FGenerator:atom) is det
Do train/test on a given fixed split specified by Train and Test. Comment can be used to describe the split briefly (will go into the results folder name).
kfold(+Signature, +K, +Models, +FeatureGenerator) is det
Perform a k-fold cross validation. Identical to stratified_kfold/5 except no stratum predicate is used.