Skip to content

Smaller items worth cleaning up (tracking) #30

Description

@CanReader

Collecting the small stuff so it doesn't get lost. Each line is independent, split
any of them out if it turns into real work.

Where What
src/nn/linear.rs:33 Linear::no_bias builds a full Linear and throws the bias away, so it still pulls out_features samples from the RNG. Harmless today (checked, seeded weights match), but it ties the RNG stream to an allocation we discard.
src/nn/embedding.rs:60 Embedding::forward does v as usize on an f32, and a negative value saturates to 0 and silently reads row 0.
src/data/loader.rs:26 Batch::labels() has the same as usize saturation.
src/tensor/ops/reduce.rs:105 variance() divides by data.len() with no guard for an empty tensor.
src/rng.rs:28 uniform(n, lo, hi) calls gen_range(lo..hi), which panics on an empty range. Reachable through kaiming_uniform with fan_in == 0.
src/tensor/ops/mod.rs:41 PARALLEL_THRESHOLD is a flat element count with no regard for how expensive f is, so gelu (an erff per element) stays serial below 32k while neg goes parallel above it.
src/serialize/safetensors.rs:70 The whole file is built in one Vec before fs::write, so peak memory is 2x the model.
src/serialize/safetensors.rs:143 used.contains(&effective) inside a loop over all entries is O(n^2) in tensor count.
src/serialize/checkpoint.rs:93 Values go out one write_all(&[u8; 4]) at a time and come back the same way. Fine through BufWriter, but a bulk cast would be a big constant factor win on large models.
cuda/kernels.cu:180 LAUNCH_ELEMENTWISE casts size_t n to int for the block count, silently wrong past 2^31 elements. Same cast in fastnn_cuda_sum, _max, _mse_loss, _binary_cross_entropy.
cuda/kernels.cu:679 fastnn_cuda_sum with n == 0 computes blocks == 0 and launches an invalid grid.
src/autograd/engine.rs The graph and every intermediate gradient stay alive for the whole reverse pass, since order holds a clone of every node. Peak memory is activations plus gradients at the same time, where PyTorch frees each node's saved tensors as it retires them. Worth at least a note in the module docs.

Metadata

Metadata

Assignees

No one assigned

    Labels

    P3Hygiene

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions