From e5ac1271f15f60c76a678d1ce628626b94a3a3b6 Mon Sep 17 00:00:00 2001 From: Matt McKay Date: Mon, 7 Sep 2026 18:55:29 +1000 Subject: [PATCH] manifest-schema.yml: say what the validator honours today MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The `schema` block header said the validator reads the raw file "honouring `delimiter` ... and `read_as` (positional workbook reads)". `read_raw()` in `builders/_validate.py` honours `delimiter` only: it consults `schema.format`, `schema.delimiter` (falling back to a top-level `delimiter`) and the `datetime64` column names, passes exactly `sep` and `parse_dates` to `read_csv`, and raises `NotImplementedError` for any format but `csv`. The identifier `read_as` does not appear in the module. No CSV manifest carries `read_as`, so nothing misreads today — the sentence promised a reader that does not exist yet. Reworded to describe the validator as shipped, and to say what `read_as` is for: the contract for workbook range reads once those are built, which is what leaves the 13 non-CSV manifests on the conformance pass alone. Also, beyond the issue's scope but in the same family — comments that state a count and have drifted from the corpus they describe: `source.doi` said "in use by 16 of 33 manifests" and `source.version` "in use by 10". Counted with PyYAML over the 44 sidecars on main: 17 and 21. Comment-only; no field changes. `scripts/validate_datasets.py` still reports 44 manifest(s): 44 pass, 0 fail under both legs of the workflow matrix (pandas 2.3.3 and 3.0.5), and the `--builders` layer is green. Closes #137. Co-Authored-By: Claude Opus 5 (1M context) --- manifest-schema.yml | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-) diff --git a/manifest-schema.yml b/manifest-schema.yml index 791d988..faf7f39 100644 --- a/manifest-schema.yml +++ b/manifest-schema.yml @@ -76,12 +76,12 @@ source: url: https://data.worldbank.org/indicator/NY.GDP.MKTP.KD.ZG # Dataset DOI where the source issues one, else null. Prefer a DOI that # resolves to the DATA deposit; an article DOI is better than nothing, but say - # which it is. In use by 16 of 33 manifests. + # which it is. In use by 17 of 44 manifests. doi: null # The upstream's own version identifier, quoted as the source states it — an # edition ("Maddison Project Database 2020"), a vintage stamp, or a file # header. Null where the source is genuinely unversioned; say so rather than - # inventing a version. In use by 10. + # inventing a version. In use by 21. version: null citation: > World Bank, World Development Indicators, series NY.GDP.MKTP.KD.ZG @@ -191,10 +191,13 @@ integrity: # #121 (nulls), #122 (dtypes) — and are recorded beside the fields below. # # What the validator reads: the RAW file as written, with pandas defaults — -# honouring `delimiter` (countries.csv is semicolon-separated) and `read_as` -# (positional workbook reads), with no `index_col`, and with `parse_dates` on -# every column declared `datetime64`. The manifest describes the bytes on -# disk, not the frame a particular lecture builds from them. +# honouring `delimiter` (countries.csv is semicolon-separated), with no +# `index_col`, and with `parse_dates` on every column declared `datetime64`. +# CSV only: `read_raw()` raises NotImplementedError on any other format, so the +# 13 non-CSV manifests get the conformance pass alone, and `read_as` +# (positional workbook reads) is the contract for those reads once they are +# built, not a kwarg the shipped validator passes. The manifest describes the +# bytes on disk, not the frame a particular lecture builds from them. schema: format: csv