Repository navigation
Machine store, sharing between machines and imports without sidecars - #115
Merged
Merged
Conversation
Blobs addressed by BLAKE2b over fixed 64 KiB pieces, put from bytes or files, received piece by piece with resume and range reads of partial blobs, refs per namespace, usage and sweep. Registered as a base engine package so the allocation lint applies; owning skill mirai-store. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Hash of bytes, an incremental hasher, the chaining value of each 64 KiB group and the root from those values, checked against the official vectors. Written with Codex. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Whole chunks are compressed straight from the caller's bytes with unrolled rounds, four chunks at a time through core:simd where there are that many. Written with Codex. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The package manifest, the build manifest and the list of a project (packages.mirai), as versioned texts, with their refs, the import key, the name of the build a program without importers reads (build_select) and packages_resolve, which lists the blobs of what a project uses and reports what is not built. Beside the sidecar: nothing uses them yet. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Completion claims the partial folder by renaming it, checks it under the claim and places from there; a full partial is completed by whoever finds it, and a failed placement gives the folder back. Range reads of partial blobs check each piece against its table. Temporary names come from system randomness. A complete blob keeps its piece table, and a Reader holds it open for range reads. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Receivers acknowledge at once every ACK_EVERY datagrams with reliable chunks. - SENT_WINDOW 4096 and reliable windows up to 16384 chunks, sized by the channel's queue_bytes; chunk records get room of their own. Acks go as gamma-coded runs (TRANSPORT_VERSION 9). - Congestion control starts low, doubles until a queue appears, then grows additively and cuts below the delivered rate; send_rate, min_send_rate, packets_per_update and Channel_Desc.window are removed. - Unreliable game channels go before reliable ones, reliable game channels take chunks in turn, and lost chunks are resent from a queue instead of a scan. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Pieces are BLAKE3 groups of 64 KiB: a piece's value is its group value, the address is the BLAKE3 root of the table, and a blob of one piece has its plain BLAKE3 hash; so the address equals blake3.hash of the bytes. Hashing a file holds its table. Layout version 2 refuses roots of the old addresses. std/blake3 joins the std engine library. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
A mount and the root take the entries packages_resolve gives; a scan lists them by their blob, with no source, sidecar or imported folder. An entry not built is listed Not_Built, a blob of another asset is Broken, hostile entries are refused. Registry_File.listed replaces the imported key, the cache layout goes to 6 and memory holds no listed path. Callers that passed an imported folder pass nothing until they resolve their packages. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An import writes every output to a scratch file and puts it in the store (Import_Options.store) as a blob; nothing goes under .mirai/imported and no sidecar is written. It records a package manifest (ids from the previous manifest, origin, identity without engine versions, settings, inputs, assets with owner and dependencies) and a build manifest (blob per output, baseline documents, encoders, Final), and names them under packages/, imports/ or imports-fast/ by asset.import_key, and builds/ by build_select. Reuse is a lookup in the builds of the package; the source an asset names is relative to its package. documents_follow writes or follows the baseline documents into a folder of a project, with today's rules; an import given documents does it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts: # .agents/skills/mirai-imports/SKILL.md # .agents/skills/mirai-imports/references/imports.md # cook/import/identity.odin # cook/import/outputs_test.odin
Each result of cook/cache is a blob of mirai:store named by the ref (cache-<name>, <key>-<version>). The store reaches the cache as a value (Import_Options.store, Sync_Options.store, Cook_Desc.store) and a miss asks the caller's cache.Source before building. A missing or damaged blob is a miss; a blob that no longer has its bytes is removed. Trimming by age and its stamp, the cache folders and --texture-cache/--mesh-cache go: the store owns space. mirai import takes --store. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Past RECORD_WINDOW a datagram still goes each keepalive, so a stream whose every datagram in flight was lost resumes; an interval that delivered next to nothing starts the rate again, and the rate grows only with acks. - The ack takes what the clock or inputs leave of the datagram (time_max_bits). - A reliable channel delivers only while the command buffer has room; held chunks pause reading datagrams until delivered. - Reliable windows grow by chunks acknowledged, bounded by the size those chunks had, so streams of small messages open their window. - The journal channel holds 256 KiB again. - The prefab replica test waits for the reads its jobs began instead of racing them with ticks that run faster than any frame. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts: # .agents/skills/mirai-imports/SKILL.md # cook/import/outputs_test.odin # cook/import/write.odin
asset.root_packages resolves the packages.mirai of a project or plugin root against the machine store (or one the caller names) and words what it cannot list as problems naming the package. The application, cook/author, the world generator, mirai asset and the editor's import library go through it; the imported folder of the project layer is gone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The sweep keeps blobs younger than a grace age, claims an old one by renaming it away before judging it again, skips what will not delete, and treats the addresses of fresh leases as live. A put of a blob already here refreshes its time. Written with Codex. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
strings.replace_all gives back its input when nothing changes, so the session freed the temp allocator's path on close and the editor aborted. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
cook/import/sources/library.odin replaces sync.odin. ingest takes a source from anywhere into the store as a package, finds an origin ingested before by the ref origins/<origin_name> and keeps its ids, skips one whose identity and build are current, and takes turns with a lock per origin. project_add and project_ingest pin a package in packages.mirai (project_pin, its only writer) and follow its documents and, for a model whose source is here, its prefab. project_refresh takes each pinned package to a build this engine reads: local refs, the caller's cache.Source, an ingest of an origin on this machine; else it reports by package name, also when only Fast is here and Final is required. It never removes and never builds without the source. Source hashes are kept per origin in the store (sources/<origin_name>). mirai import <file> ingests and adds to a project (--project, --folder); mirai import --refresh <root> replaces --sync, and the build calls it (imports_refresh). The identity of a package includes its id, so two packages of the same bytes no longer share a build key; an explicit parts setting decides the owner of documents again; cache_key_build hashes empty bytes without asserting. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts: # cook/import/identity.odin # cook/import/sources/sync.odin
A new layer-3 package moves blobs and refs between peers on the connections of net hosts its owner runs: stateless requests by address and runs of pieces, one answer with size, table and first pieces, receiver-owned priority with two requests in flight per peer, pieces spread over several sources, liars in the table, a piece or the size marked and routed around, refs asked and listed under an owner's expose procedure, a blocking fetch for jobs, and every peer message validated. The store serves the table of a blob it is still receiving, and a last piece that proves the begun size wrong returns Wrong_Size. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts: # store/receive.odin
A table of one piece begins nothing until its piece hashes to the address (the store refuses to begin one), and a failed piece never blames a peer for another's table. Wrong_Size discards only the partial blob. Tables and pieces are taken only from the peer asked; an oversized table fails that peer for the blob, never the want. Reservations live on progress of their want, a lapsed flight releases once, marks leave with a peer's slot, ended questions keep their credits until answered, success is what the store says now, destroy wakes waiters with Closed, and requests no longer depend on the server's cap. want_addresses gives the owner what its lease must hold. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A datagram is lost only once reordering cannot explain its absence: datagrams sent after it were acked and it has been out longer than the RTT and a window from its variation, widened by any loss a late ack takes back. Late datagrams are acked when everything in them was taken, and stay in the acks until the other end read one; late acks are read. The RTT comes from the datagram that arrived last, the queue test against the lowest interval mean with its uncertainty, and unreliable game channels keep their order per channel. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The session's net host also carries the exchange, and a node of cook/share over it serves the host's project snapshot and the library of the store the editor started with. A guest, before its Hello, brings a folder to that snapshot with the join of cook/share: its open project by three ways, keeping and listing its own edits, or another folder, which it then opens and joins again from. The handshake no longer carries the world layout, so a guest of another project still gets the files; its Hello does. collab.join takes a folder, collab.status gives the join's phases, progress and report and what the session moved, and Session shows the same. A manual editor opens another project without waiting for frames or asking about its own turn. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An Initiate enters the replay filter as soon as it opens, also when it is refused. Initiates end in a MAC under a key from s and the secret, so a host with an invitation answers nothing to whoever lacks it. A server that ran out of key exchanges, a filter that holds an Initiate or is full answer with a Retry cookie, and the client seals the same datagram 0 again with a new e. The filter of Initiates without a cookie stops at SEEN_LIMIT; those with one turn with the cookie. API fields hold a whole code. The private key of a network game reaches the session of its run through invite_give, never a World storage, and the session zeroes it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The host watches its project; once its files are quiet it takes the snapshot again, at most every COLLAB_OFFER_EVERY, and tells the guests with a Project message when it differs. A guest runs the same join into its open project: only what differs is fetched and its own edits are kept and listed. A repeated offer replaces what the node's lease held. The test of the session now gives the editors' frames a temporary arena that is freed and written over, which a join thread sharing it would read. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts: # .agents/skills/mirai-editor/references/collaboration.md # cook/share/command.odin # editor/collab.odin
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
cook/share lists what a peer's machine library exposes (packages with their builds, sizes and whether this machine has them) and takes one into this machine's library with its build, refreshed in a project of its own so a peer that leaves changes no project. In a session, Add from library holds This machine and one submenu per other editor; packages.peers and packages.peer_add do the same through the API. What a peer exposes is what the exchange already served. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Mode 6 of an opaque block quantizes with both p-bits 1, so its alpha decodes 255 and the guard that dropped mode 6 for mode 1 is gone; it also wrote a black block when mode 1 had no partitions to scan or was masked out. - An opaque block with neither mode 6 nor a usable mode 1 takes the alpha modes. - The cutout retry pins the mode 5 alpha endpoints of 0 and 255, so mode 5 always keeps the extremes it promises. - textures BUILD_VERSION and COOK_KEYS_VERSION rise: the blocks are other bytes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A mask with no mode 5, 6 or 7 and, for an opaque block, no mode 1 that scans partitions left the block unwritten; mode 6, which encodes any block, stands in. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This machine's library and, in a session, each other editor's are roots beside the project's content, in every arrangement and the search; a root holds the packages the project does not pin, a peer's package this machine has shows once under This machine, two of one name take the folder of their source, and a peer that leaves takes its root. A package shows its size and what is still to fetch, and its fetch and failure on its card; carried onto a folder of the project, or Add to project of its menu, it is added. The Add from library menu is gone. A peer's listing reads whether a package is here before it asks for sizes, which may bring a small object whole. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tes the scene Scenes are written without a compressor now: the fixtures that broke a compressed column mark its stored bytes a stream in the directory instead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s them Previews are derived data of the machine's store under THUMBNAIL_NAMESPACE, named by their key, in place of the thumbs folder of each project, which keeps only the views of scenes the editor took. An asset of a package is keyed by its object and the objects of its build, which any machine with the build computes without the asset, so a library card shows the preview its content has in a project. A visible card of a peer's root that lacks it asks that peer by the key, a few at a time (share.fetch_start): a peer gives one it has and makes none. A peer's listing measures sizes from roots alone (exchange.measure), so no chunk of an asset travels for the browser. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ine with the build The browser draws one path for every kind: a texture's preview is read from the largest of its levels that fits, decoded alone; a material's is the studio sphere wearing it; both are kept in the store by key like a model's, so a peer gives them as it gives a model's. A card of This machine whose store lacks its preview makes it as the project's content does: the packages the project does not pin are listed by id under a hidden mount, and only visible cards make theirs, one a frame. asset.build_shown also gives the asset id. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…sks for pages, not a ratio Files of a project hold their chunks as they are since objects are addressed by their bytes: the 25 tracked .ui of the demo and the game template are saved again, and the terrain test checks that a file holds the pages a surface crosses instead of a size that came from compressing them inside the file. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this brings
A machine-wide content-addressed store, sharing a project between machines with no server, a
derived-data cache by key, and imports without sidecars.
store: chunks of at most 256 KiB addressed by BLAKE3 of their bytes, objects as trees ofchunks with content-defined boundaries, append-only packs, refs, leases, a sweep that compacts,
and per-chunk compression recorded in the pack frame. An object's address does not depend on
any compressor.
std/blake3: BLAKE3 in Odin.exchange: peers ask for objects by address and verify on receive; they agree on the codecsthey decode before the first ask.
net: reliable bulk transfer (batched datagrams, congestion control, loss detection thattolerates reordering), and an invitation that carries the host's key so an invited peer sends
data in its first datagram.
cook/share:mirai shareandmirai join: a project snapshot as an object, three-wayupdate of a folder, the builds its packages need.
.importsidecars and no.mirai/imported/. A project pins packages inpackages.mirai; builds live in the store by key and are shared.--store <folder>on everycommand.
while it is open, and the content browser shows this machine's library and each peer's as
folders, with thumbnails stored by key and served by the peer that has them.
std/compressor/blocks: two defects of the exact-alpha rule of BC7 fixed, and mode 6 keptfor opaque blocks.
Incompatible changes
says to remove its folder; the default one is in the user's application data.
(the
.uiof the demo and the game template among them). A volume terrain file is larger ondisk than before; its compact form is a separate task.
mirai-unrealhas it on itsstore-importsbranch.Verification (macOS ARM64)
plugins/animation/editor: 109 of 117 clean before thelast fixes; the three failures this work caused are fixed and their packages pass. Still
failing, as on
main: two ofcook/build, one ofcook/workspace, two ofeditor, one ofstd/subprocess.mirai build editor,mirai scene check,mirai play testandmirai build gameof the demo.mirai shareandmirai joinover loopback: identical folders, and the editor buildsin the joined copy; the same between a build with another compressor and this one, both ways.
Not verified
mirai package engine.🤖 Generated with Claude Code