What can be improved?
Caveat currently uses the train dataset (and val) for "evaluation" aka testing.
The rationale is that this matches the intended use case.
However would also like to test for the ability of caveat to:
- discover unseen sequences / imagine new valid sequences that are correct
- work with smaller datasets
- recover zero samples from biased down samples, such as missing short activities
Broadly we want to specify how we partition out train, val and test sets. If at all.
For all cases suggest adding a library of Samplers to be specified in the a new data_params config group, eg:
data_params:
data_path: ...
test_sampler: Random
val_sampler: Random
test: 0.2
val: 0.2
test_strategy: inclusive
suggested samplers:
- Unbiased
- BiasedShortActs
- BiasedEvenActs
suggested strategies:
- exclude: train only
- inclusive: combine all
- val_inclusive: combine val and test
Version
v0
What can be improved?
Caveat currently uses the train dataset (and val) for "evaluation" aka testing.
The rationale is that this matches the intended use case.
However would also like to test for the ability of caveat to:
Broadly we want to specify how we partition out train, val and test sets. If at all.
For all cases suggest adding a library of Samplers to be specified in the a new data_params config group, eg:
suggested samplers:
suggested strategies:
Version
v0