Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
54 changes: 43 additions & 11 deletions docs/manifests.md
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,7 @@ spec §5's MUST, which a list of rows prov re-lays-out plainly falls under.
| `files` | yes | The rows. Absent or null means an empty directory. |
| `verbatim` | no | `true` claims every file under `root`, documents included (§3.1). Absent means `false`; any value but a boolean is refused. |
| `exclude` | no | Patterns for files under `root` the manifest does not claim (§3.2). A list of strings; a pattern outside the syntax is refused. |
| `held` | no | Who keeps the files when not every device has them (§3.5): `by`, a name, required; `at`, an address, optional. Anything else is refused. |

Each row:

Expand Down Expand Up @@ -123,6 +124,19 @@ directory then rewrites one line instead of ten thousand.
- A nested manifest's directory belongs to its own node, so two manifests can
never claim one file.

Completeness is what makes the interesting question answerable. A covered file is
not a document, so §8's orphan walk cannot see it, and it is not a link, so the
census cannot either. Without a claim over the whole directory, a photograph
could vanish and no prov surface would ever say so.

**Hashes are optional, per row.** A manifest with no hashes is an *inventory* —
what is supposed to be here. One with hashes is that plus a fixity baseline.
Hashing reads every file, at mint and at every refresh, which is a real cost over
an archive and a reasonable one to decline for a directory of scans while still
wanting checksums everywhere else (`attach --manifest --no-hash`). A refresh
**preserves the mode**: an inventory that quietly began recording digests would
be claiming a guarantee nobody asked for.

### 3.1 Verbatim: claiming the documents too

Some directories are one artifact, not an archive: a built web app, whose
Expand Down Expand Up @@ -214,18 +228,35 @@ the same either way, and a backend that cannot read in parts reads the file
whole, as before. This holds for every checksum prov records over a payload:
manifest rows, attachment sidecars, inline rows, and `check`.

Completeness is what makes the interesting question answerable. A covered file is
not a document, so §8's orphan walk cannot see it, and it is not a link, so the
census cannot either. Without a claim over the whole directory, a photograph
could vanish and no prov surface would ever say so.
### 3.5 Held: files kept on another machine

**Hashes are optional, per row.** A manifest with no hashes is an *inventory* —
what is supposed to be here. One with hashes is that plus a fixity baseline.
Hashing reads every file, at mint and at every refresh, which is a real cost over
an archive and a reasonable one to decline for a directory of scans while still
wanting checksums everywhere else (`attach --manifest --no-hash`). A refresh
**preserves the mode**: an inventory that quietly began recording digests would
be claiming a guarantee nobody asked for.
An archive too large for every device stays on the machine that keeps it,
while its workspace — the description and the manifest — travels. On a
device without the files, every row would read as missing and the root as a
broken link. `held` says that is expected:

```yaml
root: photo/
held:
by: Harris NAS # the keeper, named for a person
at: https://nas.example:8765/ # where to ask it for a file; optional
```

- **Where `root` is a directory, nothing changes.** That is the keeper, or a
full copy, and the manifest is checked, refreshed and verified exactly as
an unheld one.
- **Where it is not, the rows are held elsewhere.** `check` says nothing
about them; `manifest` reports *held by Harris NAS; not on this device*;
`--verify` verifies nothing and says so, exiting non-zero; and `--update`
and `--add` refuse, because rebuilding the rows from a directory that is
not there would empty the record.
- **Without `held`, an absent root is a broken link, as ever.** A manifest
must say its files are elsewhere before their absence stops counting.

Neither `by` nor `at` is a secret: who may ask the keeper is the keeper's to
decide. `manifest TARGET --held-by NAME [--held-at ADDRESS]` sets it, and
`--not-held` clears it; either rewrites the manifest and re-pins the node.
Set it where the files are, before the manifest travels.

## 4. The chain

Expand Down Expand Up @@ -383,6 +414,7 @@ Each verb is one change set: moved, written and removed together.
| `manifest TARGET --update` | rebuild the rows from the directory as it is now, keeping each row's fields, re-stamping the node |
| `manifest TARGET --add FILE...` | append rows for the named files only, checksumming just those; every other row stays, its file present or not (§5) |
| `manifest TARGET --verify` | re-read every listed file and compare its checksum |
| `manifest TARGET --held-by NAME [--held-at ADDRESS]` | say who keeps the files, so a device without them reads them as held elsewhere (§3.5); `--not-held` clears it |
| `gather ATTACHMENT... --into DIR [--title T] [--discard FIELD] [--dry-run]` | attachments of one index into one manifest over `DIR` (§7) |
| `scatter TARGET [--into INDEX] [--discard FIELD] [--dry-run]` | a manifest back into one attachment per file (§7) |

Expand Down
41 changes: 41 additions & 0 deletions prov-cli/src/check.rs
Original file line number Diff line number Diff line change
Expand Up @@ -359,11 +359,30 @@ pub(crate) fn cmd_manifest(
update: bool,
verify: bool,
add: &[PathBuf],
held: Option<Option<prov::Held>>,
) -> CmdResult {
let mut session = Session::open()?;
let target_rel = ws_rel(&session.ctx, target)?;
let node = resolve_manifest_node(&session.ws, &target_rel)?;

if let Some(held) = held {
let said = match &held {
Some(held) => format!("held by {}", held.by),
None => "not held".to_string(),
};
let wrote = block_on(session.ws.set_manifest_held(&node, held))?;
session.commit()?;
let status = block_on(session.ws.manifest_status(&node))?
.ok_or_else(|| format!("{} declares no manifest", node.display()))?;
if wrote {
eprintln!("{}: now {said}", status.manifest.display());
} else {
eprintln!("{}: already {said}", status.manifest.display());
}
println!("{}", status.manifest.display());
return Ok(ExitCode::SUCCESS);
}

if !add.is_empty() {
let paths = add
.iter()
Expand Down Expand Up @@ -401,6 +420,16 @@ pub(crate) fn cmd_manifest(
}

if verify {
let status = block_on(session.ws.manifest_status(&node))?
.ok_or_else(|| format!("{} declares no manifest", node.display()))?;
if let (true, Some(held)) = (status.elsewhere, &status.held) {
eprintln!(
"{}: nothing verified — its files are held by {} and are not on this device",
status.manifest.display(),
held.by
);
return Ok(ExitCode::FAILURE);
}
let findings = block_on(session.ws.verify_manifest(&node))?;
for finding in &findings {
println!("{finding}");
Expand All @@ -427,6 +456,18 @@ pub(crate) fn cmd_manifest(
", no checksums (an inventory)"
}
);
if let Some(held) = &status.held {
let at = held
.at
.as_deref()
.map(|at| format!(", at {at}"))
.unwrap_or_default();
if status.elsewhere {
eprintln!(" held by {}{at}; not on this device", held.by);
} else {
eprintln!(" held by {}{at}", held.by);
}
}
for path in &status.missing {
eprintln!(" missing: {}", path.display());
}
Expand Down
11 changes: 11 additions & 0 deletions prov-cli/src/cli.rs
Original file line number Diff line number Diff line change
Expand Up @@ -648,6 +648,17 @@ pub(crate) enum Command {
/// a document, or one already listed.
#[arg(long, value_name = "FILE", num_args = 1..)]
add: Vec<PathBuf>,
/// Say that NAME keeps the files, so a device that carries the
/// manifest without them reads them as held elsewhere rather than
/// missing. Set it where the files are, before the manifest travels.
#[arg(long, value_name = "NAME", conflicts_with_all = ["update", "verify", "add", "not_held"])]
held_by: Option<String>,
/// Where a device asks the keeper for a file. Not a secret.
#[arg(long, value_name = "ADDRESS", requires = "held_by")]
held_at: Option<String>,
/// Say that every device carrying the manifest has its files again.
#[arg(long, conflicts_with_all = ["update", "verify", "add"])]
not_held: bool,
},
/// Gather attachments into one manifest: their files move into a new
/// directory, their sidecars go, and one node takes the first one's place
Expand Down
12 changes: 11 additions & 1 deletion prov-cli/src/main.rs
Original file line number Diff line number Diff line change
Expand Up @@ -170,7 +170,17 @@ fn main() -> ExitCode {
update,
verify,
add,
} => check::cmd_manifest(&target, update, verify, &add),
held_by,
held_at,
not_held,
} => {
let held = match held_by {
Some(by) => Some(Some(prov::Held { by, at: held_at })),
None if not_held => Some(None),
None => None,
};
check::cmd_manifest(&target, update, verify, &add, held)
}
Command::Gather {
cards,
into,
Expand Down
121 changes: 121 additions & 0 deletions prov-graph/src/manifest.rs
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,13 @@
//! drift. An excluded directory is not walked at all. See
//! [`Manifest::excludes`] for the syntax.
//!
//! And a manifest may say its files are kept somewhere else: `held:` names the
//! machine that holds them (`by`) and, optionally, where to ask it for one
//! (`at`). On a device without the covered directory, a held manifest's rows
//! are elsewhere rather than missing — a photo archive on a NAS, described in a
//! library every device carries, whose files only the NAS has. Where the
//! directory is present nothing changes. See [`Held`].
//!
//! This module is the model and its serialization, which is all the read core
//! needs; the verbs that build, refresh and verify one are `prov`'s.

Expand All @@ -85,6 +92,9 @@ pub const VERBATIM_KEY: &str = "verbatim";
/// not claim — see [`Manifest::excludes`].
pub const EXCLUDE_KEY: &str = "exclude";

/// The manifest key saying who keeps the covered files — see [`Held`].
pub const HELD_KEY: &str = "held";

/// The manifest key holding the rows.
pub const FILES_KEY: &str = "files";

Expand Down Expand Up @@ -113,6 +123,67 @@ pub struct ManifestEntry {
pub fields: Mapping,
}

/// Who keeps a manifest's files, when they are not on every device that
/// carries the manifest ([`HELD_KEY`]).
///
/// Written as `held: {by: Harris NAS, at: https://nas.example/}`. Neither is a
/// secret: who may ask is the keeper's to decide, never settled by knowing the
/// address.
///
/// It changes what an absent root means and nothing else. Without it, a
/// covered directory that is not there is a broken link and every row is
/// missing; with it, the rows are held elsewhere, `check` says nothing about
/// them, and the verbs that would rebuild the rows from the directory refuse
/// rather than empty the manifest. Where the directory is present — on the
/// keeper itself — the manifest is checked exactly as an unheld one.
#[derive(Debug, Clone, PartialEq, Eq, Default)]
pub struct Held {
/// The keeper, named for a person: `Harris NAS`.
pub by: String,
/// Where a device asks the keeper for a file, if it can be asked at all.
pub at: Option<String>,
}

impl Held {
fn from_value(value: &Value) -> Result<Self> {
let refuse = || {
Error::Structure(format!(
"manifest `{HELD_KEY}` must be `{{by: <name>, at: <address>}}`"
))
};
let map = value.as_mapping().ok_or_else(refuse)?;
let mut by = None;
let mut at = None;
for (key, value) in map {
match key.as_str() {
"by" => by = Some(value.as_str().ok_or_else(refuse)?),
"at" => match value {
Value::Null => {}
value => at = Some(value.as_str().ok_or_else(refuse)?.to_string()),
},
_ => return Err(refuse()),
}
}
let by = by
.map(str::trim)
.filter(|by| !by.is_empty())
.ok_or_else(refuse)?;
Ok(Held {
by: by.to_string(),
at,
})
}

fn to_value(&self) -> Value {
let mut map = Mapping::new();
map.insert("by".into(), Value::String(self.by.clone()));
if let Some(at) = &self.at {
map.insert("at".into(), Value::String(at.clone()));
}
Value::Mapping(map)
}
}

/// A parsed manifest document: the directory it claims, and the files it says
/// are in it.
#[derive(Debug, Clone, PartialEq, Default)]
Expand All @@ -129,6 +200,8 @@ pub struct Manifest {
/// Patterns for files under `root` that the manifest does not claim
/// ([`EXCLUDE_KEY`]), as written. See [`excludes`](Self::excludes).
pub exclude: Vec<String>,
/// Who keeps the files, when not every device has them ([`HELD_KEY`]).
pub held: Option<Held>,
}

impl Manifest {
Expand Down Expand Up @@ -177,6 +250,10 @@ impl Manifest {
patterns
}
};
let held = match meta.get(HELD_KEY) {
None | Some(Value::Null) => None,
Some(value) => Some(Held::from_value(value)?),
};
let rows = match meta.get(FILES_KEY) {
// A manifest over an empty directory has no rows at all, which is a
// legitimate state (`files:` written as null, or absent).
Expand Down Expand Up @@ -218,6 +295,7 @@ impl Manifest {
files,
verbatim,
exclude,
held,
})
}

Expand All @@ -234,6 +312,9 @@ impl Manifest {
if self.verbatim {
map.insert(VERBATIM_KEY.into(), Value::Bool(true));
}
if let Some(held) = &self.held {
map.insert(HELD_KEY.into(), held.to_value());
}
if !self.exclude.is_empty() {
map.insert(
EXCLUDE_KEY.into(),
Expand Down Expand Up @@ -699,6 +780,46 @@ mod tests {
assert_eq!(parse(&text(&excluding)).unwrap(), excluding);
}

#[test]
fn held_round_trips_and_is_left_out_when_absent() {
let text = |m: &Manifest| {
crate::meta::serialize_mapping(
&m.to_mapping("Photo — manifest"),
crate::MetaFormat::Yaml,
)
.unwrap()
};
let plain = parse("root: photo/\n").unwrap();
assert_eq!(plain.held, None);
assert!(!text(&plain).contains("held"), "{}", text(&plain));
let held =
parse("root: photo/\nheld:\n by: Harris NAS\n at: https://nas.example/\n").unwrap();
assert_eq!(
held.held,
Some(Held {
by: "Harris NAS".into(),
at: Some("https://nas.example/".into()),
})
);
assert_eq!(parse(&text(&held)).unwrap(), held);
let nameless = parse("root: photo/\nheld: {by: Harris NAS}\n").unwrap();
assert_eq!(nameless.held.unwrap().at, None);
}

#[test]
fn a_held_it_cannot_read_is_refused() {
for bad in [
"held: Harris NAS\n",
"held: {at: https://nas.example/}\n",
"held: {by: ' '}\n",
"held: {by: 3}\n",
"held: {by: NAS, where: attic}\n",
] {
let text = format!("root: photo/\n{bad}");
assert!(parse(&text).is_err(), "{bad}");
}
}

#[test]
fn an_exclude_it_cannot_read_is_refused() {
for bad in [
Expand Down
1 change: 1 addition & 0 deletions prov/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -307,6 +307,7 @@ pub use mutate::{
PlannedIndex, RegroupOptions, Reordered, Reparented, ScatterPlan,
};
pub use prov_exports::ExportSpec;
pub use prov_graph::manifest::Held;
pub use prov_views::ViewSpec;
pub use provenance::{Actor, Confirmation, Confirmations, Generated, Tier};
pub use remedy::{Fix, Remedy, RemedyKind, Warrant};
Expand Down
Loading
Loading