Skip to content

Dropout builds its mask on the host and uploads it every forward #20

Description

@CanReader

src/nn/dropout.rs:41. rng::bernoulli runs on the CPU and the mask goes up with
.to(device) on every call. In a transformer that's one full activation sized H2D
transfer per dropout, per layer, per step, sitting right on the critical path.

kernels.cu already has kernel_dropout_forward with a curand seed
(kernels.cu:1601), it's just never declared in ffi.rs.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions