add option to `flux job attach` to ignore existing output

[grondo]

A use case for flux job attach to a running job is for users to occasionally attach and interact with a job that has a cli type interface. However, currently flux job attach always emits all previous output that is in the eventlog, and this can get annoying if the job has been running for a long time with a lot of output.

It would therefore be useful to add an option to flux job attach to only display the last bit of output. The command would likely still need to process all the data since I don't think there's a way to "tail" an eventlog from the KVS, but it would only actually display the most recent bit. Actually implementing this could be slightly challenging since I'm not sure if there's a way to tell when you've "caught up" when watching an eventlog.

[garlick]

I think @chu11 had some thoughts earlier on this? The KVS sees an eventlog as a series of blobs appended to a key, so maybe if there were someway to watch starting from a blob offset at the end?

[grondo]

That would be most excellent.

[chu11]

I think @chu11 had some thoughts earlier on this?

We had a prior discussion in #5035. (quote from that issue "B/c I'm imagining the hours and hours of stdout that someone may not want to output.").

I think there are two issues, one is to "tail" the output and the other is to ignore previous output. They are similar but different. I think the latter is much easier to output. @grondo suggested a simplish first round implementation could be to just ignore all output w/ timestamps before "now". That's a very costly implementation b/c we're streaming all data, but would make the output to the user nicer. I should note that such an implementation would (should?) be a lot easier now that we have FLUX_KVS_STREAM implemented (#6523) (side note, i'm not sure if job-info watch streams the data).

The KVS sees an eventlog as a series of blobs appended to a key, so maybe if there were someway to watch starting from a blob offset at the end?

yeah, this is from a larger set of issues generally related to "KVS data size" (#6292). Where we can "jump" offsets. But it requires us knowing data sizes of the blobs b/c we otherwise don't know how to jump (rfc flux-framework/rfc#449 as a starting point ... and i have a lingering branch starting on that work in flux-core).

[grondo]

We had a prior discussion in #5035.

Ugh, sorry. This seems to be a duplicate issue!

[grondo]

Closing as duplicate of #5035.