There are two clashing ways to use dropout in model parts - on the output or the input.
@varisd suggested going with the first option (each model part is responsible for dropping out its outputs), but this requires a greater deal of work since the majority of the codebase is done the other way (except for transformer components).
I now prefer the first option. We could even introduce a new decorator which would apply dropout on a returned value from the decorated method, provided that the class has the dropout_keep_prob field.
There are two clashing ways to use dropout in model parts - on the output or the input.
@varisd suggested going with the first option (each model part is responsible for dropping out its outputs), but this requires a greater deal of work since the majority of the codebase is done the other way (except for transformer components).
I now prefer the first option. We could even introduce a new decorator which would apply dropout on a returned value from the decorated method, provided that the class has the
dropout_keep_probfield.