Group coproducts by source dataset, and stream exports as raw bytes#191
Merged
Conversation
buildInterningTables sorted the keys of activityMap, then looked every activity back up in that same map with a partial 'M.!'. Zip the map's ascending association list instead: the key order is identical, the activity comes along for free, and the partial lookup is gone.
An activity UUID is a hash of the activity name and location, so it is not an identity. Agribalyse 4.0 truncates a SimaPro "Process name" to 80 characters and reuses it verbatim across unrelated blocks: three distinct lorry processes hash to one UUID, and dbActivityProductsIndex then reported each one's reference product as a coproduct of the others. Carry the source's own block identifier on the activity (activityNativeId, SimaPro's "Process identifier") and key the products index on the pair. Formats with no such field keep grouping by UUID alone, exactly as before.
ILCD names a process by a single dataset UUID, one process per file. The writer used the activity UUID for that, so an activity with two reference products would have written both to one file. Rather than lose a product, checkILCDExportable refused the whole export — which a SimaPro CSV triggers routinely, since two unrelated blocks can share a name and therefore an activity UUID. Mint a dataset UUID per (activity, product) pair instead, and only when the activity UUID is actually shared. That condition is what keeps parse . write a fixed point: the parser reads the dataset UUID back as the activity UUID, and every re-imported process is single-output, so a second export reproduces the first byte for byte. Deriving unconditionally would not, since UUID5(UUID5(a,p),p) /= UUID5(a,p).
Every writer that emits a zip folded addEntryToArchive over its entries. That function deletes any entry sharing the new path before consing, so it rescans the whole list on each insert: n entries cost O(n²) FilePath comparisons and stack n lazy filters before anything is forced. A full Agribalyse ILCD export (53 508 files) exhausted memory and outlived Warp's 600 s timeout without emitting a byte; the same export now returns in 70 s with no measurable heap growth. Build zEntries directly, in a leaf module the three writers share. Uniqueness of the paths — the only thing addEntryToArchive bought — is already guaranteed: each is derived from a dataset UUID.
The export endpoint wrapped the serialized database in a base64 JSON envelope: +33% on the wire and four full copies in memory (strict bytes, base64, Text, Aeson escaping) with nothing emitted until the last one. A full Agribalyse ILCD export produced a 483 MB response for a 362 MB zip. The response is now the bytes themselves, as the upload endpoint already reads them, so Warp streams it chunked and re-arms its timer per block. Approximation warnings move to an X-Volca-Export-Warnings response header, percent-encoded because activity names are arbitrary Unicode and the warnings are newline-joined. The CLI reports them as [WARN] lines; pyvolca surfaces them through the warnings module. Measured on Agribalyse 4.0 (22 822 processes): HTTP 200 in 72 s, 362 MB raw body, flat server RSS.
The CLI's HTTP manager kept http-client's default 30 s response timeout, but the server legitimately computes for minutes before the first byte of a large database export or batch scoring leaves — so `volca database export` died with ResponseTimeout on any real database. Client-mode managers now wait indefinitely; connection failures still error immediately.
Splitting a multi-output activity into separate process datasets keeps every product but silently drops their grouping on re-import. The export already has a warnings channel for exactly this kind of best-effort approximation, so emit one warning per split activity, and make the local CLI export surface warnings instead of discarding them.
This was referenced Jul 10, 2026
ccomb
added a commit
that referenced
this pull request
Jul 11, 2026
…es (#195) The `[Unreleased]` section only covered #191. Two other user-visible changes since v0.9.0 had no entry: - #190 — immediate groundwater emissions are characterized again in ecotoxicity and human toxicity (worth ~10% of the freshwater-ecotoxicity score on the witness process); - #185 — two EF 3.1 synonym bridges that stop flows listed under a sibling name from silently scoring zero. The changelog gate in the release preflight is a count heuristic, so it passed despite the gaps; found by hand-checking coverage against the git log. With these two entries the section fully covers v0.9.0..HEAD.
ccomb
added a commit
that referenced
this pull request
Jul 11, 2026
The export endpoint changed shape in #191: the response went from a JSON envelope (base64 `data` + `warnings` array) to `application/octet-stream` with warnings in the `X-Volca-Export-Warnings` header. That is breaking in both directions — in particular pyvolca ≥ 0.7.2 talking to a v0.9.0 engine saves the JSON envelope bytes as the exported file **without raising**. This bumps `currentWireVersion` and `REQUIRED_WIRE` to 2 so both stale pairings get a clear signal instead of silent corruption: a newer client refuses an older engine with an actionable error, an older client warns against a newer engine. `MIN_ENGINE_HINT` points at v0.9.1, the first release that will advertise wire 2. The README compatibility table and its generated block are regenerated, and the compat test's expected downgrade hint updated. Verified: engine builds under `-Werror` with the full test suite green, and the pyvolca suite passes (187 tests) including the README drift check.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two related fixes on database export, both triggered by exporting Agribalyse 4.0 to ILCD (22 822 processes, 53 508 files):
addEntryToArchiveover its entries, which rescans the whole archive on each insert — O(n²) onFilePathcomparisons. A shared linearZip.zipFilesreplaces all three call sites.X-Volca-Export-Warningsresponse header instead.Also fixes the CLI's HTTP client, which kept the default 30 s response timeout and so timed out on any real export regardless of format.
Measured on Agribalyse 4.0:
POST /db/agb40/exportreturns in ~72 s with a flat server RSS, versus never completing before.Test plan
cabal test— 1524 examples, 0 failurespyvolcatest suite — 180 passedfourmolu --mode check/hlintclean