CLI binary?

[danya02]

I've been looking for a utility to do something like this. The Go library is packaged as a CLI app, but to install it I'd need to download the Go toolchain, which I don't have. Meanwhile, I do have a Rust toolchain available, so I'd like it if I could just cargo install this.

I think a CLI wrapper shouldn't take too much effort. If you don't have the time for it, I could try doing that and submit a PR.

[unsektor]

FYI: i have prepared PoC version of implementation of CLI application here: https://github.com/unsektor/rs-imohash/pull/2/files .
It works, but threading logic must be generalized and placed in lib.rs file instead of main.rs.

Working with threads in Rust is not trivial. Implementation of good generalized solution requires some time.
Here is an example of threads synchronization how to read data from threads on fly https://github.com/brmmm3/scandir-rs/blob/6b9f4d145e1486a94d273b8c0001d417dda79f32/pyscandir/src/walk.rs#L228-L275

[unsektor]

offtopic:

Also there is similar functionality written in Python https://github.com/unsektor/py-imohash . It uses THIS Rust library as underlying, but mutli-threading is powered by native Python, it has known performance restrictions because of the GIL.
In other words, it makes job, but not as fast as possible.

[danya02]

I'm not entirely sure what the multi-threading is needed for? If reading from different files, then you can fairly trivially distribute the list to a pool of workers, and then use the global stdout lock (implicitly used in println! and friends) to make sure that the output rows don't intermingle. Just say that the output order is not defined and you have to sort it yourself if you need to.

If it's for reading from different parts of the same file, then your access pattern for the file stops being linear and the spinning-rust drive (the full contents of which is what I wanted to hash with this) will have to do a lot of seeking back-and-forth to service the threads' requests.

[unsektor]

Yes, multi-threading is needed to handle different files at the same time, to decrease total processing time of hash computation.

You are right, text (stream) is universal interface, so performing the result output to stdout should be enough in case when program interface is the only CLI (i/o are text streams).

I'm talking about a bit lower API level, where the interaction interface is Rust native structures (e.g. synchronized stack/queue as input and a vector of tuples (hash, file path) as output). This is mainly necessary for a direct function invocation (for example, in a deduplicating application) or even via FFI. In these examples, it is strange to use a library function in which communication is carried out through a text stream with lots of synchronization I/O locks, at least by direct/reverse text conversation of formatted output (for example, {hash} {file_path}) for each result entry, am I wrong?

Summarizing, in my opinion, when developing a solution in this "generalized" way, two goals are achieved: the native rust interface and the CLI interface built on top of the previous one.

[danya02]

e.g. synchronized stack/queue as input and a vector of tuples (hash, file path) as output

I'm not sure that this interface is necessarily a good idea. To my mind, an algorithm-based library like a hash should just accept data and return hashes (just as it does now), with optional conveniences for reading a file.

What you seem to be suggesting involves embedding an entire multithreading manager (like rayon, or else something simpler for the specific use-case) into this crate. That seems like a bit of an overkill: in particular, it makes that part of the crate very much nostd-incompatible, and it bloats dependent binaries regardless of whether they end up using this.

I'm pretty sure that your idea can be easily implemented with rayon with:

    let file_paths = vec![
        "file1.txt",
        "file2.txt",
        "file3.txt",
    ];

    let hashes: Vec<(_, _)> = file_paths
        .par_iter()
        .map(|&file_path| {
            let h = Hasher::new();
            (file_path, h.sum_file(file_path).unwrap())
        })
        .collect_into_vec();

so there isn't much need to embed that into this crate, just write that in your API consumer (like the CLI app) instead.

---

Anyway, I'd prefer seeing an async or sans-io version of the Hasher struct, so that you could use this with something that doesn't implement Read+Seek. That way, you could avoid blocking on it like you have to now, and you could get the parallelism you want on a single async-based thread (assuming the storage medium is slow enough for that to matter, of course).