Does DAR have a built-in way to output progress?

[AlexanderTheGrey]

Is there a way for DAR to output the archive creation progress percentage (similar to verbose xz output), and then separately have a log of all files processed (similar to --index-file from tar), or do I need to use pv? If so, what's the proper usage with pv (a lot of times progress information is difficult to consolidate when piping to pv)?

[Edrusb]

I understand you would like to view a percentage of completion indicator while dar is processing a backup/archive?

For xz, this is feasible, because first the overall data to compress is very easy to obtain and from the beginning (this is the file size as obtained by the stat() system call). For dar, this is different, because of filtering mechanisms, it would require scanning the whole directory to backup to know which file will be under backup and which one will be skipped and sum the file sizes of only concerned files. Such initial scanning process could take a long time (with a lot of small files it could double the processing time) and would not be itslef possible to include in the displayed percentage of accomplishment! Worse, it could give a wrong result if the filesystem is living with files and directories to be appended, removed, truncated, added during this initial scan processing (while dar is able to handle a live filesystem).

Today, there the -vf option can give you raw indication of where the backup process is, but it assumes you known the directory tree structure under backup... not really what you would expect, right?

If this is to get the file listing while they are processed, you can get it using -vt option alone (no other -v* option added). You then just have to check the exit code of dar to know how the operation ended, and check dar's standard output (which can be redirected to a file if you prefer)

[AlexanderTheGrey]

Thanks for the response. xz has the same problem: its progress indicator can't show meaningful completion percentage in pipes because it doesn't know the total input size upfront, just like dar can't pre-count filtered files without scanning everything first. It's a trade-off between showing accurate progress percentage and the overhead of an upfront directory scan (which comes with its own issues).

What about showing the progress of a single file (especially relevant for large files)? I was thinking about outputting a single line from the current file being processed (since I'm only concerned about monitoring progress on the terminal), which is then replaced with the next one using \r. Maybe per-file percentages can be built-in (similar to rsync --progress), which can be used alongside the separate log file option that lists all processed files?

[Edrusb]

Yes, that's more realistic to do a per-file progression. Though here, to do that, there are some pitfalls: dar is the CLI part while libdar library is the engine under the hood and should stay CLI independent.

So in terms of implementation there is either

1/ the API must be extended for dar to be able to fetch this percentage information. Second, as dar process is calling libdar, it must be a second thread added on dar side of the API to fetch the current percentage info and displaying it from time to time. Also the libdar API must update this information from time to time.

Or

2/ rely on user_interaction object passed during the API call to perform the operation and let libdar (which is expected to be CLI independent) to send back the percentage formatted message (including the \r char to overwrite current line) that would in case of dar go to its stdout or stderr depending on the context (in case of webdar, gdar or other application using libdar this would be a problem as \r would not have any special meaning)

the 2/ approach is much easier to implement but breaks the CLI-independency of libdar and would only provide the expected result for dar (not for gdar or webdar for example). The second is better from design point of view, but more complicated to implement.

Another issue, due to the way the implementation is structured in libdar, is the use of "generic_file" class: this is a pure virtual class that provide the interface to all file-like operations (open/read/write/truncate/seek/close). This way you have a compressor class that compresses to another given generic_file the data it has to write and compresses from this other generic_file the data he has to read, same for ciphering, escape sequences, sparse files, slicing (= sar), down to plain normal files. So far, the data backup is a simple call to copy a generic_file content to another one. On the archive side this generic_file is a stack of sparse-file/compression/escape/encryption/slicing while on the filesystem side this is just a plain file implementation of a generic_file. Adding the percentage information during a copy may impact a lot of classes in the library (must take time to see that in details) and is independent for the decision to use 1/ or 2/ as explained above.

All this for further reference and for you to get a high level view of the constraints brought by the feature you request