diff --git a/.agents/skills/mirai-agent-api/references/methods.md b/.agents/skills/mirai-agent-api/references/methods.md index 46c6ef73e..675d6f3c1 100644 --- a/.agents/skills/mirai-agent-api/references/methods.md +++ b/.agents/skills/mirai-agent-api/references/methods.md @@ -872,19 +872,30 @@ of an instance" are `-32602`; the rest, `-32000` another asset (that of an imported prefab), is its owner. - `asset.import`: what the Import button of the content browser does with any source it imports ([import](../../mirai-editor/references/panels.md#import)), wherever it is on disk: the - editor reads it and writes only in the assets folder. `folder` replaces the folder of its type; a - source that a plugin imports whole (a map) does not accept it. It imports, writing to disk (it - is an operation on files) while the editor goes on, and answers when it ends: whether it wrote - (`ok`), its message, the scene of a map and the assets it wrote (`kept`: its file was already - the one it gives). `asset.import_status` gives the same, and whether one runs (`importing`), at - any moment. While it runs, another import and whatever writes assets fail saying what is being - imported. Reimporting keeps ids and files, which the sidecar of the source lists + source stays where it is, its package goes to the store of the session and the project pins it, + listed in `folder` (else the root, or the folder of a source in the assets). It imports, + writing to disk (it is an operation on files) while the editor goes on, and answers when it + ends: whether it wrote (`ok`), its message, the scene of a map and the assets it wrote + (`kept`: its file was already the one it gives). `asset.import_status` gives the same, and + whether one runs (`importing`), at any moment. While it runs, another import and whatever + writes assets fail saying what is being imported. Reimporting keeps ids and files, which the + package of the source lists ([identity](../../mirai-imports/references/imports.md#identity-and-reimport)); over a document - that changed since it was imported it fails naming it, and `discard_changes` writes over it - ([changed documents](../../mirai-imports/references/imports.md#changed-documents)). + of the project that changed since it was written it fails naming it, and `discard_changes` + writes over it ([changed documents](../../mirai-imports/references/imports.md#changed-documents)). `image` (`color`, `data`, `normal`, `normal_green_down`) says what an image imported on its own holds, over its settings and its name (a packed metallic-roughness or a mask is `data`); a source in the project keeps the choice in its settings +- `packages.list {name?}`: the packages of the library of the machine (the store of the session) + with their assets (`name`, `kind`, `bytes`, `built`) and whether the project pins each (`pinned`, + `folder`). `packages.add {name, assets?, folder?}` pins one, by id or name, or some of its + assets by id, listed in `folder`, as carrying it from a library root of the content + browser; it runs as an import and answers as `asset.import`. An unknown package or asset, an + invalid folder or a session without a store fail saying how to fix it. +- `packages.peers {refresh?, seconds?}`: in a session, the libraries of the other editors, each + package with `bytes`, `built`, `here` and `pinned`; `listing` while they are read, `adding` for + the last package taken. `packages.peer_add {peer, name, folder?}` takes one into this machine's + library and adds it as `packages.add` does; one whose peer left says so in `adding.error`. - `asset.detach`: `asset` (path or id) is a part that leaves its owner: it is left without `owner` and shown alone; the import keeps writing it and leaves it out ([parts](../../mirai-imports/references/imports.md#parts)). It is an edit of its file, with undo; @@ -898,14 +909,17 @@ of an instance" are `-32602`; the rest, `-32000` `{id,path}`; an unknown type fails naming the valid ones, and a taken or invalid name says why. - `asset.delete`: `asset` (path or id) goes to the system trash with the import parts it owns, like Delete of the [content browser](../../mirai-editor/references/panels.md#content-browser): at - once, without undo. Returns `{moved,used_by}`; those that use it keep the - reference. Fails saying why with an asset of a plugin, of the engine or of the editor, a scene - open in a tab or outside Edit. + once, without undo; the root of a package also takes the package out of the project, and what + it listed leaves with it while the store keeps its blobs. Returns `{moved,used_by}`; those that + use it keep the reference. Fails saying why with another asset a package lists, an asset of a + plugin, of the engine or of the editor, a scene open in a tab or outside Edit. - `asset.move {asset, folder?, name?, copy?}` takes an asset by path or id, or a folder by path, and writes at once, without undo; it returns the resulting path and id ([rename, move and duplicate](../../mirai-editor/references/panels.md#content-browser)). No `folder` keeps the one it is in and `/` is the root. Moving keeps the id and takes the parts - along; what is being edited fails asking to finish the edit. With `copy` it + along; the root of a package moves where the project lists it, keeping its name, and another + asset a package lists fails naming that root; what is being edited fails asking to finish the + edit. With `copy` it copies with new ids and the copy is not an output of the import; without `name` it takes a free one. - `asset.users`: `asset` (path or id). Returns `{used_by}`, the files that reference it or a part @@ -1125,7 +1139,12 @@ The methods read the source the editor has. [Collaboration](../../mirai-editor/references/collaboration.md). In `peers` of `collab.status`, `playing` says that editor is in a Play or Simulate of its own and `serving`, that it serves a networked game; `unsent` lists the asset files changed in the session that the others do not have -because they are too large to travel. an invalid code is `-32602`. +because they are too large to travel. an invalid code is `-32602`. `collab.join` takes an optional +`folder` (the open project when empty) and returns at once; `join` of `collab.status` follows how +that folder is brought to the host's project, phase by phase, and lists after it what was kept, +not written and the packages no peer has a build of; `project`, the chunk bytes the session moved +and the peers that went bad; `offers` and `updates`, the new snapshots a host offered and a guest +took during the session. ## Project panels diff --git a/.agents/skills/mirai-assets/SKILL.md b/.agents/skills/mirai-assets/SKILL.md index 7d9dd9be8..4c6beb0e4 100644 --- a/.agents/skills/mirai-assets/SKILL.md +++ b/.agents/skills/mirai-assets/SKILL.md @@ -27,6 +27,9 @@ types, scenes and their prefab instances, registry). [container](references/formats.md#container), [typed tables](references/formats.md#typed-tables), [asset types](references/formats.md#asset-types), [versions](references/formats.md#versions), [registry](references/formats.md#registry) and [text](references/formats.md#text). +- The records of imports in the machine store (package and build manifests, `packages.mirai` + and the manifest files a root carries), their refs and `packages_resolve`: + [packages](references/formats.md#packages). - The collection `compressor`, profiles, codecs, streams and decoders: [compression](references/formats.md#compression). - Streams, schemas, migration, paths, text: [serial](references/serial.md#stdserial). @@ -108,9 +111,11 @@ types, scenes and their prefab instances, registry). package: `compressor:data`, `compressor:texture` and, engine-time, `compressor:blocks`. The open one (`std/compressor`) is a real implementation; the private one has the same packages and declarations. Its sources are never versioned in the engine nor packaged. -- **What the repository of the engine versions is open**: its projects declare - `build.compression` `open`, and the assets of the engine and its plugins are always written - open. A project chooses for itself (`none`, `open`, or the best its engine has). +- **What is hashed and shared never depends on a compressor.** The documents of a project, the + assets of the engine and its plugins, imports and objects of the store are written with + `Codecs.None`: no stage of a compressor, each kind keeping the modeling of its data, chosen + without one. The store compresses where bytes rest and the exchange where they travel; only the + files a cook ships are compressed as the project's `build.compression` says. - **A stream says what decodes it**: a read needs no profile nor codecs. - **Decoders are preallocated.** A `Stream_Decoder` holds all the scratch, allocated once by whoever reads; a decode never allocates, and for any input never reads outside `src` nor writes @@ -131,6 +136,12 @@ types, scenes and their prefab instances, registry). - **The header says Scene or Prefab.** One `.scene` extension and kind; `Header_Flag.Prefab` is what whoever made the file chose, never guessed from its roots, and a prefab has exactly one root ([prefab](references/scenes.md#prefab)). +- **A package asset is an object of the store read by parts**, never a file opened by path: what + a load needs reads only its chunks, a part not here is `Not_Here` to its caller, and encoding + the same input twice gives the same object ([store](references/formats.md#container)). +- **A project pins the manifest of each package it uses, never a build**, and carries the text + of each manifest it pins; builds are found by key through refs, and resolving a list never + removes nor rebuilds ([packages](references/formats.md#packages)). - **A guest mount never takes an id the registry has**: what the program shipped wins over what a player adds ([guest mounts](references/formats.md#registry)). @@ -157,7 +168,15 @@ types, scenes and their prefab instances, registry). `a_scene_saved_with_another_layout_of_a_component_loads_it_by_field_names` of `engine/scene`; - the document of a scene or a `.ui` from a file outside the assets (what `dump` and git show): `the_commands_read_write_and_check_the_document_of_an_asset` of `cook/author`; + - records of packages: `package_records_read_back_what_they_write`, + `packages_resolve_lists_the_objects_of_what_a_project_uses`, + `package_records_refuse_hostile_text` and `a_root_carries_the_manifests_it_pins` of + `asset/packages_test.odin`; - registry: `asset/registry_test.odin` and, in the editor, `editor/content_test.odin`; + - assets of the store: `every_kind_loads_equal_from_a_file_and_from_the_store`, + `the_small_levels_of_a_texture_load_before_the_large_ones_arrive`, + `a_chunk_stored_as_it_is_is_read_in_place` and `hostile_object_meta_and_groups_fail_the_asset` + of `asset/parts_test.odin`; - layers of an imported material: `asset/kinds_test.odin`, and the overrides shared with prefabs in the editor, `editor/prefabs_test.odin`; - JSON inspection with transient fields: `std/serial/text_runtime_test.odin`. diff --git a/.agents/skills/mirai-assets/references/cli.md b/.agents/skills/mirai-assets/references/cli.md index 08503aff6..d52f43c9e 100644 --- a/.agents/skills/mirai-assets/references/cli.md +++ b/.agents/skills/mirai-assets/references/cli.md @@ -23,10 +23,9 @@ subcommands with their usage. of the engine), the one whose assets have its id. A file outside the assets (another version of one, as git gives it) reads as the asset with its id. `unpack`, `pack` and `dump --schema` refuse them. -- **`repack`** writes each file again compressed as the project that holds it saves - (`build.compression`), or open outside a project; a texture with its blocks as they are. A file - that already is so stays untouched: what a project that changes its compression runs over its - assets. +- **`repack`** writes each file again as a project keeps it, without a stage of a compressor + (`Codecs.None`); a texture with its blocks as they are. A file that already is so stays + untouched: what a project runs over the files of its assets a compressor wrote. - **`verify`** checks container, hash, decompression, schema and tables, that the stored dependencies are the derived ones and that each value meets its schema. For the engine's types it checks their version (for one the build does not read, the reason is `version_message`, with the diff --git a/.agents/skills/mirai-assets/references/formats.md b/.agents/skills/mirai-assets/references/formats.md index f9689523e..5d7674c80 100644 --- a/.agents/skills/mirai-assets/references/formats.md +++ b/.agents/skills/mirai-assets/references/formats.md @@ -23,6 +23,20 @@ each at a multiple of its alignment. - `header_read` validates only the header (what a scan uses). `blob_open` validates the whole file: exact size, each chunk inside the file and aligned, uncompressed size below `PACK_MAX` and a total memory limit (`DEFAULT_MEMORY_LIMIT`). +- **An asset of the store** (`asset/parts.odin`) is an object of the + [store](../../mirai-store/SKILL.md): one group per container chunk in directory order, and the + header and directory as the meta of its root. It holds its file as `Codecs.None` writes it: + `object_put` writes again a file a compressor touched (a stream chunk, a block or mesh page not + `STORED`, texture layouts, planes chosen otherwise), so the same asset is the same object for + every compressor, and the store compresses each chunk in its frame. It cuts each chunk where the + format reads alone: between the levels of `texels` (smallest first, so the first chunks are the + small levels), between the blocks of a mesh's `page_data` and of a block table. The file is the + meta, each group at its chunk's offset and zeros between: `object_file` gives the bytes + `object_put` kept. `asset_open_object` checks meta against groups (tags, order, sizes, + limit) and reads each chunk the first time a table needs it: one chunk of a sealed pack is a + slice of its mapping, several are gathered into memory of the asset, and a stream that holds + levels reads only the pieces asked for. A chunk the store does not hold yet is `Not_Here` to + whoever reads it; nothing checks a whole-file hash. - `blob_build` stores a chunk compressed only when that shrinks it. The dependencies, names, schema and table directory are stored as they are: a scan reads them without a decoder. @@ -45,11 +59,13 @@ journal sends over the network, editor thumbnails, the shaders the executables e imports and a running game save; a cook writes with `Ultra` (a shipped game) or `Balanced` (a dev game), and what streams while playing with `profile_streamed` of it (`Speed` in place of `Ultra`). A read needs no profile: the stream names its format. -- **Codecs.** What a write may use (`Codecs`): `Open`, the formats every build reads; `Best`, what - the compressor of the build has; `None`, nothing. `Types.codecs` says it for every document made - with those types (the zero value is `Open`): a program that edits a project sets it from - `project.compression`, and everything else (the assets of the engine and its plugins, caches) is - written open. A stream of a format the build does not decode is `Codec_Missing`. +- **Codecs.** What a write may use (`Codecs`): `None`, the zero value, no stage of a compressor; + `Open`, the formats every build reads; `Best`, what the compressor of the build has. Every + document is written `None` but the files a cook ships, which take `project_codecs` + (`build.compression`). Under `None` the modeling a kind chooses by what compresses smaller is + chosen without a compressor, as what compresses smaller where the bytes rest: a mesh page keeps + its coding and BC1, BC3 and BC5 their planes of fields. A stream of a format the build does not + decode is `Codec_Missing`. - **Block tables.** A kind whose asset stores blocks it reads alone keeps them in `_blocks`, with the `Block_Record`s of its pages in `_pages`; a cook compresses them again from what the records say (`document_repack`). @@ -61,10 +77,10 @@ journal sends over the network, editor thumbnails, the shaders the executables e without one makes its own the first time it decodes. A mesh page or a terrain block decodes with a `Stream_Decoder` of its reader (the renderer keeps one per worker). - **Texture levels.** The `texels` column holds the smallest level first: a read of the small - levels decodes only the first pieces. A texture written with `Best` codecs may store layouts of - runs of levels (the `layouts` table, `blocks.layout_encode`), and BC1, BC3 and BC5 may store - planes of fields ([texture](kinds.md#texture)); `texture_decode` restores them into memory of the - asset before the renderer takes them. + levels decodes only the first pieces. A texture a cook writes with `Best` codecs may store + layouts of runs of levels (the `layouts` table, `blocks.layout_encode`), and BC1, BC3 and BC5 + may store planes of fields ([texture](kinds.md#texture)); `texture_decode` restores them into + memory of the asset before the renderer takes them. ## Typed tables @@ -180,8 +196,8 @@ One rule for all types, of the engine and of plugins: plugin's type, in the `version` of its `asset_kinds`, which the build generates as `_VERSION` ([manifests](../../mirai-projects/references/projects.md#manifests)). - **A change in what a type stores or in what a stored value means raises its version**, and every - file of the type is made again: the [sync](../../mirai-imports/references/imports.md#sync) of - the next build imports again the source of an imported one. `version_message` says which asset + file of the type is made again: the [refresh](../../mirai-imports/references/imports.md#library) + of the next build ingests again the source of an imported one. `version_message` says which asset it is, its type, its version and the one the build reads and, if it was imported, the `mirai import` that rebuilds it in its folder with the root where its source is; for a newer file, a newer engine. `mirai asset verify` @@ -221,17 +237,25 @@ scan opens every file and writes it again. is a prefab ([prefab](scenes.md#prefab)). - `registry_dependents` comes from an inverse index built with each change: how many use an asset is its length, without walking the files (`asset.list` reports it as `used_by`). -- **Imported folders.** An assets folder may have an imported folder with it (that of the root - and of each mount: `.mirai/imported/` of the project or the plugin): what the - [imports](../../mirai-imports/references/imports.md#identity-and-reimport) of its sources build - that nobody edits, never versioned, each file named by its id and the key of what it was built - from. A scan reads the sidecars it walks (`asset/sidecar.odin`, the one reader of the format) - and lists each output of the cache like any asset (`Registry_File.imported` is that key) at - `//`; two outputs of one kind and - name are told apart by their index, and a path another file or output has takes the name of - the source: never the order of a scan. One whose source is gone is not listed, and one that - moved is listed there at the next scan, which a refresh of a sidecar or of its source runs. - `registry_file_path` gives the file of either folder. A cooked folder has none: the cook +- **Listed outputs.** What the + [imports](../../mirai-imports/references/imports.md#identity-and-reimport) build that nobody + edits is not a file of the assets folder but an object of the machine store. The root and each + mount take the entries `root_packages` gives for the packages their root pins: `packages_resolve` + of its `packages.mirai` against the store (`asset/packages_root.odin`). Every program that sets + up the registry of a root that is not shipped goes through it, with the store root as a value + (the machine's unless it names another); a root without the list opens no store, and what it + cannot list is a problem that names the package, reported where that program reports asset + problems, never a failure to start. A scan lists each like any asset (`Registry_File.listed`), + with no source anywhere, from the header and dependencies `packages_resolve` read of its object. + An entry whose build is not on this machine, or whose object is not here, stays listed as + `Not_Built` (its id and kind, not found by id), never removed and never an error of the scan; + an object that is not the asset its entry names is `Broken`. A path a file, a folder or another + entry has takes the id of the entry after its name, never the order of a scan, and an entry with + no id or a path that is empty or leaves its folder is refused (`Registry.refused`). Objects do + not change: a refresh keeps what is listed, unless a file of the folder appears at its path + (which runs a scan), and memory holds no listed path. A listed path has no file + (`registry_file_path` gives ""): `registry_listed_object` gives its object, which `host` opens + by parts, and `registry_read` its file from `Registry.store`. A cooked folder has none: the cook writes every asset at its path. - **Mounts.** `registry_mount` adds a folder under a prefix (a plugin's assets, in `plugins/`). A prefix that starts with a dot belongs to the program's assets (the editor's, @@ -270,6 +294,47 @@ scan opens every file and writes it again. on the creation times of a machine, so a game ships neither until the copy is written with its own id. +## Packages + +The records of the imports kept in the [machine store](../../mirai-store/SKILL.md) +(`asset/packages.odin`), three texts with a version each. + +- **Package manifest** (`Package`): an object, named by `manifests/`, what a + pin names; `packages/` holds that BLAKE3, not an object, on the machine that + ingested it, so a refresh asks a source or a peer for a manifest only by `manifests/`. The + import that ingests a source writes it. A root that pins it also keeps + its text, versioned, in `packages/.mirai` beside `packages.mirai` + (`package_file`). +- **Build manifest** (`Build`): an object that names the object of each output, named by + `imports/` (Final) or `imports-fast/` by the import that builds it, and by + `builds/` by the refresh of a project that chose it. +- **Project list** (`Packages`): `packages.mirai` beside `project.mirai`, versioned. The library + (`project_pin`, `project_unpin`) writes it and the manifest files, and nothing else does. + +- **A project pins the manifest, never a build.** The manifest holds what the source is (its + `identity`), where it was (`origin`, a hint) and each asset with its id, and nothing that + depends on the engine version: an engine that changes a version leaves the list as it is. +- **A root carries the manifests it pins.** The address a pin names is that of its manifest + file's bytes, so a clone or a copy of the root, with an empty store, has every id and setting; + only builds need the store, a source or a peer. `package_load` takes a manifest the store + lacks from that file and puts it in the store; a file that is not its pin's bytes is a problem + of that package, also when the store has the manifest. +- **The origin of a source in a root is relative to it** (`assets/...`, from the root whose + assets folder holds it), so every copy of the root writes the same manifest and finds the + source; a source in no root keeps its absolute path. +- **Builds are found by name.** `import_key` combines the identity with the versions of what + imports; any machine with the manifest computes it, without the source, and a peer that built + that key gives it. `build_select` combines the manifest with the version of each kind of the + cache it holds, so a program without importers (the dev player) finds the build the refresh + chose. The caches of `cook/cache` name theirs in `cache-`. +- `packages_resolve` lists, for each package of the list, its assets of the cache the project + uses with what they depend on, at `/`, with their object and its + header. A missing build lists them not built; a manifest that neither the store nor the root + has, that does not read or whose file is not its pin, a build that does not read and an asset + the manifest lacks are problems of the result. It removes, builds and fetches nothing. +- An asset id is random at the first ingest and the manifest keeps it; an ingest again takes + each id from the manifest before it. + ## Text `document_text_write` produces the JSON that `mirai asset dump` prints and `unpack` saves, of diff --git a/.agents/skills/mirai-assets/references/kinds.md b/.agents/skills/mirai-assets/references/kinds.md index 1f25a40b4..5b6dbb86f 100644 --- a/.agents/skills/mirai-assets/references/kinds.md +++ b/.agents/skills/mirai-assets/references/kinds.md @@ -33,7 +33,8 @@ model that consumes it (hulls, hierarchy and limits) belongs to cracks open. The normal is octahedral and the tangent a signed angle (`math.tangent_angle_pack`); `uv1` and `color` are separate streams, only with `Has_Uv1` and `Has_Colors`. Every bulk column goes compressed, except `page_data`, which is compressed page - by page, each page coded or as its raw words, whichever is smaller (`coded_bytes`, 0 for raw). + by page, each page coded or as its raw words, whichever is smaller (`coded_bytes`, 0 for raw); + written without a compressor (`Codecs.None`) a page is coded whenever it codes. - Clusters go in pages (`Mesh_Page`): whole groups (the clusters that share a parent) of a single level, contiguous, up to `MESH_PAGE_WORDS`; a DAG group never crosses two pages. Their clusters, vertices and triangle words are contiguous ranges of the mesh's, from the finest to the coarsest. @@ -118,15 +119,17 @@ per bone. - **Levels, smallest first.** `texels` holds the levels from the smallest to level 0, compressed: `texture_open` reads a texture without its texels and `texture_decode` from a level decodes only the pieces of the stream that hold that level and the smaller ones (`texture_view` decodes all); - `texture_mip` gives the bytes of a level, empty for one before `first`. Written with `Best` - codecs, a texture of 16-byte blocks `cook/textures` writes stores the layouts the compressor has - for it (`compressor:blocks`), which restore only whole: one of its tail (the levels of side + `texture_mip` gives the bytes of a level, empty for one before `first`. Written by a cook with + `Best` codecs, a texture of 16-byte blocks stores the layouts the compressor has for it + (`compressor:blocks`), which restore only whole: one of its tail (the levels of side `TEXTURE_TAIL` or less, which every read takes) and one of its larger levels; one per level compresses larger. -- **Planes of fields.** A BC1, BC3 or BC5 texture whose blocks are not `Fast` and that is - compressed stores each level as planes of the fields of its blocks (`Fields`: endpoints and - indices apart, `asset/texture_fields.odin`) when that stream is smaller, which the writer decides - per texture compressing both with the profile `Fast` of its codecs; the read joins them back. +- **Planes of fields.** A BC1, BC3 or BC5 texture whose blocks are not `Fast` stores each level as + planes of the fields of its blocks (`Fields`: endpoints and indices apart, + `asset/texture_fields.odin`): written without a compressor (`Codecs.None`) always, since they + compress smaller where the bytes rest; written compressed when that stream is smaller, which the + writer decides per texture compressing both with the profile `Fast` of its codecs. The read joins + them back. - `Srgb` says it is color: stored with the sRGB curve in `RGBA8`, `BC1`, `BC3` and `BC7`, linear in the others. - **Layers.** `layers` is the number of images of each level (one if the file does not store it): diff --git a/.agents/skills/mirai-core/references/host.md b/.agents/skills/mirai-core/references/host.md index 06348be96..416948741 100644 --- a/.agents/skills/mirai-core/references/host.md +++ b/.agents/skills/mirai-core/references/host.md @@ -39,7 +39,13 @@ without pacing (deterministic runs). settings only); `options`: folder of the player options, which the player provides (`app_user_folder`) and the editor does not; `assets_root`, `shaders_root`. `root`, `engine_root` and `options` are copied into the application. -- `asset_mounts`: plugin assets mounted under `plugins/`. +- `store_root`: the machine store the packages of `root` and of its plugins list from + (`project_assets`); empty is the machine's. The application mounts the assets of the active + plugins of `root` under `plugins/` with the outputs of their packages, logs and keeps + (`assets_package_problems`) what a package could not list, and opens no store for assets that + carry the index of a cook; one that lists keeps it open (`Assets.store`). A part of a listed + output not in it yet is `Not_Here`: the stand-in shows, nothing fails, and a later request + loads it. `asset_mounts`: more folders mounted beside them. - `gpu_validation`. `App` stores the platform, `jobs`, `info`, `shared` (the `Shared` of the application), `world` (the @@ -343,13 +349,13 @@ of pieces) is not a fixed cap, and a frame that does not load does not allocate. stores none are kept ([`asset.mesh_collision_pieces`](../../mirai-assets/references/kinds.md#mesh)); physics asks for them in a job. -- `assets_file(a,id,kind,&Asset)`: transactional load of any type by Fourcc, validating identity and - hash; the caller closes with `asset_close`. It serves plugin formats without adding them to the - GPU caches. It reads with `registry_read`: what the registry has in memory before the disk - ([in-memory files](../../mirai-assets/references/formats.md#registry)); likewise the `content_*`. -- **Loading in jobs.** `assets_source` gives, on the assets thread, where a job reads the file of an - asset from (its path, or a copy of what the registry has in memory); `assets_source_open` opens it - as `assets_file` on any thread. `assets_prepare` does in a job what loading a mesh, a texture or a +- `assets_file(a,id,kind,&Asset)`: transactional load of any kind by Fourcc (plugin formats too), + checked by identity and hash. It reads what the registry has in memory before the disk + ([in-memory files](../../mirai-assets/references/formats.md#registry)), and a listed output by + parts of its [object](../../mirai-assets/references/formats.md#container), no whole-file read. +- **Loading in jobs.** `assets_source` gives, on the assets thread, where a job reads an asset + from (its path, what memory holds, or its object); `assets_source_open` opens it as + `assets_file` on any thread. `assets_prepare` does in a job what loading a mesh, a texture or a material does before the renderer (read, verify, decompress, decode the levels that the device takes), with the same code as the direct load, and `assets_take` installs it between frames as if it had been requested; it does nothing if the asset was loaded in the meantime or its file changed diff --git a/.agents/skills/mirai-editor/SKILL.md b/.agents/skills/mirai-editor/SKILL.md index 06d13fbf1..ca855026e 100644 --- a/.agents/skills/mirai-editor/SKILL.md +++ b/.agents/skills/mirai-editor/SKILL.md @@ -111,7 +111,10 @@ of the `Editor_World` and its functions are `@(step)` of the `editor` loop, like variants and Apply to base, in [variants](references/prefabs.md#variants). - **Shared session and the host's networked game**: - [entering and leaving](references/collaboration.md#entering-and-leaving), + [entering and leaving](references/collaboration.md#entering-and-leaving), the project and its + builds that travel with the session ([the project](references/collaboration.md#the-project)), + the libraries of the other editors + ([libraries of the session](references/collaboration.md#libraries-of-the-session)), [the host's networked games](references/collaboration.md#the-hosts-networked-games) and [where each one plays](references/collaboration.md#where-each-one-plays). - **Figure of agents and editors, glyph menu, Follow, palette, pings, the editor's own assets @@ -159,6 +162,10 @@ of the `Editor_World` and its functions are `@(step)` of the `editor` loop, like are off, and the project library does not swap while a plugin's handler is on the stack. - Asset folders are thematic, never by type (the browser filters by type): what the editor imports or creates goes where the user is, and it creates no folder named after a type. +- The editor imports through the library and the store of its session + ([import](references/panels.md#import)): a source is read where it is and never copied into a + project, and importing, deleting or moving the root of a package changes only the project's + `packages.mirai` and its documents, never a blob of the store nor a source outside the project. - State located according to [the session](references/editor.md#the-session). `Api` keeps only a pointer because the server must stay where it was born. - A procedure takes the storage it reads and writes (`^Ai`, the chat store, `^Git`, `^Edits`), @@ -324,6 +331,11 @@ of the `Editor_World` and its functions are `@(step)` of the `editor` loop, like (the program a material document reads, its picks for the next message, the values of a variant); their view in an open editor, `editor.document` (`Open` with the asset's path) and `capture` with the UI. +- The project a session carries: `editor/collab_share_test.odin` (a guest with an empty folder + ends with the host's files, packages built and scene; a guest's own edit is kept and listed; + what the host changes in the session reaches the guest by three ways; the join's thread keeps + off the frame's temporary memory), + over `Sim_Network` in virtual time with temporary stores. - Build and reload, networked Play (`templates/game`) and collaboration (`--share`/`--join`) are checked by API in an open editor. Images: [rendering](../mirai-rendering/SKILL.md#verification); protocol: diff --git a/.agents/skills/mirai-editor/references/collaboration.md b/.agents/skills/mirai-editor/references/collaboration.md index fbf24617e..d708e2024 100644 --- a/.agents/skills/mirai-editor/references/collaboration.md +++ b/.agents/skills/mirai-editor/references/collaboration.md @@ -1,6 +1,7 @@ # Collaboration -Code: `collab.odin`, `collab_api.odin` and `collab_play.odin` (the host's networked games). +Code: `collab.odin`, `collab_api.odin`, `collab_project.odin` (the project that travels) and +`collab_play.odin` (the host's networked games). - Several editors share one's scene. The **host** owns the file and orders the edits; a **guest** receives the whole scene on entering and keeps its copy in memory. @@ -15,13 +16,14 @@ Code: `collab.odin`, `collab_api.odin` and `collab_play.odin` (the host's networ `--join `, Session > Join or `collab.join`, with the host's invitation code. The name comes from `--name`, the Name field of Session or `name` of the API. It is left with Session > Stop (host) or Leave (guest), or `collab.leave`, and the scene stays as it is. -- Joining **replaces** the guest's scene. Sharing and joining require `Edit`, a project with an - assets folder and no open session (`Busy`). `COLLAB_PEERS` editors fit. -- On sharing, the host creates a token and shows its invitation code +- Joining **replaces** the guest's scene. Sharing and joining require `Edit`, the store the editor + started with (`--store`, else the machine's) and no open session (`Busy`); sharing, and joining + into the open project, a project with an assets folder. `COLLAB_PEERS` editors fit. +- On sharing, the host makes the key of the session and shows its invitation code ([invitations](../../mirai-networking/references/networking.md)): the address of its network (`address_lan`: the one the system uses toward the internet, not a virtual adapter's), the one the router mapped for it ([port mapping](../../mirai-networking/references/port-mapping.md)) and - the token. Whoever enters without that token gets no answer and times out; the host's log shows + the key. Whoever enters without that key gets no answer and times out; the host's log shows the code each time it changes. - The host sees one code in Session, once the router answered: a guest asks at both addresses at once, so the same code joins from the host's network and from outside it. A guest's networked @@ -30,14 +32,62 @@ Code: `collab.odin`, `collab_api.odin` and `collab_play.odin` (the host's networ the host holds its previous connection takes its place and its color, and that connection is closed with what it had selected. Whoever leaves the session's list retains nothing. - `collab.join` returns before the scene arrives: query `collab.status` until `Joined`. The phases - are `Collab_Phase`, with the reason for the end. + are `Collab_Phase`, with the reason for the end; `Updating` is the guest bringing its folder to + the host's project ([the project](#the-project)). + +## The project + +- The session carries the project as `mirai share` and `mirai join` do + ([sharing a project](../../mirai-projects/references/projects.md#sharing-a-project)), with the + same node of `cook/share`: the session's net host also carries the channels of the + [exchange](../../mirai-store/references/exchange.md), at `COLLAB_EXCHANGE_CHANNEL`, so its + connections move the project and its builds; what a peer sends enters only through the checks + of `cook/share` and `exchange`. +- **Hosting** takes the snapshot of the project into the editor's store and offers it under the + session's key, with the builds its packages pin; the whole library of that store is exposed to + whoever holds the invitation, as in `mirai share`. A lease of the session holds what it serves + and what it fetches. +- **Joining** first runs the join of `cook/share` (phase `Updating`), and only then says Hello and + receives the scene. Into the open project (Session > Join, or `collab.join` without `folder`) it + is the three-way update: a file both changed stays the guest's and is listed as kept, never + overwritten. Into another `folder` (new, or joined before), once the files and the packages are + there the editor opens that folder as File > Open project does, and the editor started on it + joins the session again with the same invitation, which then finds nothing to fetch. A guest + needs none of the files beforehand. +- Session shows the join's step (files and bytes, rate and time left, the builds) where it shows + the phase, and after it the files kept, those not written and the packages no connected peer + has a build of; `join` of `collab.status` gives the same. +- **What the host changes during the session** (an import, a saved asset, a source file, a file + moved or removed, a package build it now pins) reaches the joined guests: the host watches its + folder, and once its files have been quiet for `COLLAB_OFFER_QUIET` it takes the snapshot again, + at most once a `COLLAB_OFFER_EVERY`; when it differs it offers it under the same key and sends + `Project` through the session. A guest that gets it runs the same join into its open project, + as soon as the update it runs ends: only what differs is fetched, and the three-way rule keeps + and lists what it changed. What it wrote reloads as any file changed on disk; its packages are + listed again. A guest's document has no file of its own, so the scene file the host writes does + not apply its edits a second time. +- A guest's own imports and file changes do not travel to the host as files; the asset files + edited in the session travel as below. + +## Libraries of the session + +- Each editor of a session sees the machine libraries of the others it is connected to (a guest, + the host's; the host, each guest's): the packages each one's store exposes, which `exchange` + already serves to whoever holds the invitation. They are read on a thread + (`collab_library.odin`, `share.peer_library`) when the content browser or `packages.peers` shows + them, at most once a `COLLAB_LIBRARY_SECONDS`. +- Adding one takes it into this machine's library first (`share.package_take`): its manifest and + a build this engine reads, as a join fetches builds, refreshed in a project of its own under the + generated folder, so nothing is reimported when its source is not here. Only then is it added + as a package of this machine. A peer that leaves before the build is here leaves the project and + the library as they were, and says so; once taken, the package stays in this machine's library. ## What travels - **The whole scene**, once on entering: what `scene.capture` writes, as one message over the reliable channel. The guest loads it with `scene.replace`: one that does not fit whole leaves its - world as it was and ends the session. Assets do not travel, except those an editor of the session - changes (below): both editors must have the same files. + world as it was and ends the session. The files it needs arrived before, with + [the project](#the-project). - The network half of the session runs in its own thread (`worker_start`): an editor busy for a while (generating a game's world, a long frame) keeps responding and nobody gives it up for gone. - Reliable messages wait in each peer's queue (`Collab_Peer.out`) and stream into the channel as @@ -114,11 +164,13 @@ Edits: ## Compatibility -- The handshake carries the world layout version: a hash of `COLLAB_VERSION` (that of the - messages) and of names and sizes of the `World` tables. -- If it differs, the guest stops on reading the host's challenge (`Denied_Version`: "has another - version of the project"). Both editors must be generated and compiled from the same project and - engine. +- The handshake carries `COLLAB_WIRE`, the version of the messages and of `cook/share`, so a guest + built from another project still gets the host's files. Its Hello carries the world layout: a + hash of `COLLAB_VERSION` and of names and sizes of the `World` tables. +- If the layout differs, the host refuses the Hello (`Denied_Version`) and the guest's session + ends saying to build its editor, whose files are the host's by then, and join again. A guest + that joins into another folder never meets it: the editor opened there is built from the host's + files. ## When the host changes scene diff --git a/.agents/skills/mirai-editor/references/extensions.md b/.agents/skills/mirai-editor/references/extensions.md index bc99bc652..70f0f32c9 100644 --- a/.agents/skills/mirai-editor/references/extensions.md +++ b/.agents/skills/mirai-editor/references/extensions.md @@ -393,29 +393,28 @@ and joins them with the engine's. those of `sources.FORMATS`, with the same reading, writing, cache and identity ([plugin formats](../../mirai-imports/references/imports.md#plugin-formats)); its `Format.folder` chooses the folder of the import. Its extensions enter the dialog's Models - filter and what the editor recognizes on dropping a file. With `prepare(assets, info, + filter and what the editor recognizes on dropping a file. With `prepare(library, root, info, allocator)` it gives, before a model is imported, the format's options for that import - (`Format.data`) from the registry of the project and the library of its plugins (for example - what other providers give), with no session: the import program of the engine prepares it the - same way when a build syncs the project (`import_library`). `release(data, allocator)` - releases them on finishing. + (`Format.data`), with no session: `library` is the store the packages are in (the session's in + the editor, the one of the command in the import program), `root` the project or plugin the + import is for (`asset.root_packages(root, library.root)` lists its packages) and `info` the + types of its game, with what other providers give. The import program of the engine prepares + it the same way when a build refreshes the project (`import_library`). `release(data, + allocator)` releases them on finishing. - `editor.Import_Source_Provider{source}` gives a `Source_Importer`, a source the plugin imports whole with `importer.Outputs`, the single source of every asset it makes ([plugin formats](../../mirai-imports/references/imports.md#plugin-formats)). Its fields are `name` (the dialog's filter and the messages), `extensions`, `claims(path, head)` (claims the file by its extension and its first `importer.SNIFF_BYTES`; it is tried before the formats), - `prepare(assets, info, allocator)`, `run(state, source, options, allocator)`, - `release(state, allocator)` and `bring(source, folder, allocator)`. + `prepare(library, root, info, allocator)`, `run(state, source, options, allocator)` and + `release(state, allocator)`. The source is read where it is: nothing is copied into a project. - `prepare` runs before starting, like that of a format, with no session: what `run` needs from - the registry of the project and the library of its plugins (for example, what other providers - give) is gathered there, in the received allocator, and is the `state`. `run` runs in the + the library, the root and the types of its game (for example, what other providers give) is + gathered there, in the received allocator, and is the `state`. `run` runs in the import's thread with a job system of its own in `options.worker` and returns the report, the scene it wrote (`""` if none) and the error. `release` releases `state` on finishing. The - import program prepares and runs it the same way when a build syncs the project + import program prepares and runs it the same way when a build refreshes the project (`import_library`). - - `bring`, when given, takes a source from outside the assets: it copies it with the files it - reads into `folder` (absolute, in the assets) and returns where the source is now. Without it - the editor copies the source alone. - A plugin can define its own providers so that other plugins join it: the one that gives the source gathers them in `prepare` and passes them to `run` through the `state`. This way an import plugin delegates to automatic bridge plugins the actors of optional domains, without diff --git a/.agents/skills/mirai-editor/references/panels.md b/.agents/skills/mirai-editor/references/panels.md index 05f77512a..db5058417 100644 --- a/.agents/skills/mirai-editor/references/panels.md +++ b/.agents/skills/mirai-editor/references/panels.md @@ -558,9 +558,9 @@ assets. They are off outside Edit and while watching. (`content_focused`, which `shortcuts_run` looks at before the entity Delete), opens `Modal.Delete_Asset` with the asset, its parts and what uses it. Confirming sends it to the trash (`platform.trash`) immediately, without undo: it is an operation on files, like import. -What uses it keeps the reference. Assets of a plugin, the -engine or the editor, an open scene or prefab, are not deleted, nor outside Edit; the modal says -why. +What uses it keeps the reference. The root of a package takes the package out of the project +([import](#import)). Assets of a plugin, the engine or the editor, an open scene or prefab, are +not deleted, nor outside Edit; the modal says why. **Rename, move and duplicate** (`asset_move.odin`) also change the files immediately, without undo, and rescan the registry. Identity is the id, so references, Recent, Favorites and open @@ -579,8 +579,9 @@ documents follow the asset; the selection passes to it (`Content.reveal`). registry paths that the scan replaces. - **Parts and imports.** An asset moves with the parts it owns; each keeps its place relative to the owner's folder (the one outside it goes to the destination folder), and if one cannot go, - nothing moves. The link with the source does not change: `Import_Info.source` names the source - from the project and reimporting finds the outputs by their `Import_Info` + nothing moves; a part a package lists goes with the listing of its package ([import](#import)). + The link with the source does not change: `Import_Info.source` names the source from its + package and reimporting finds the outputs by their ids ([identity](../../mirai-imports/references/imports.md#identity-and-reimport)). - **History and references.** The history names the moved files at their new path, so that undo and redo write there, and the project's `start_scene` follows its scene. @@ -731,17 +732,25 @@ browser cards and asset fields): ([thumbnails](../../mirai-plugins/references/previews.md#animation-thumbnails)); without it they show their icon. - **Stored.** The preview of an asset (mesh, prefab or a type of a project plugin, not that of an - entity, a shape or the default of a type) stays in `.mirai/thumbs/-.thumb`, one per - asset and size, with premultiplied alpha, compressed, and the key it was drawn with. - - The key (`thumbnail_store_key`) combines `THUMBNAIL_RENDERER`, the size, the subject without - the revision and the `content_hash` of the asset and of the whole dependency closure the - subject names, as the registry has it. Changing the asset or a dependency - gives another key and the preview is redrawn. + entity, a shape or the default of a type) is derived data of the machine's store: an object + under `THUMBNAIL_NAMESPACE` named by its key, with premultiplied alpha, compressed. It is drawn + once per machine, whatever project shows the asset, and a peer of a session gives it to the + others ([import](#import), Libraries). + - The key of an asset of a package (`thumbnail_package_key`) combines `THUMBNAIL_RENDERER`, the + size, the programs, the asset's object and every object of the build it is in: any machine + with the manifest and the build computes it without the asset. That of the project's own files + (`thumbnail_store_key`) takes the subject without the revision and the `content_hash` of the + asset and of the whole dependency closure the subject names, as the registry has it. Changing + the asset or a dependency gives another key and the preview is redrawn. - When the signature changes, a cell that already has the image of the current key is not - redrawn; otherwise, the file is read (up to `THUMBNAIL_LOADS` per frame, apart from drawing) and - it is drawn only without a file or with another key. - - Those of an id that the registry notes in `Registry.removed` are deleted in the next frame - (`thumbnails_forget`). + redrawn; otherwise, the store is read (up to `THUMBNAIL_LOADS` per frame, apart from drawing) + and it is drawn only when the store has no preview of that key. + - A texture's preview is read on the processor from the largest of its levels that fits the + cell, decoded alone (`thumbnail_texture_image`); a material's is the sphere of the studio + wearing it. Both are stored previews like the others: the browser draws nothing else. + - The view of a scene the editor took is no derived data: it stays in + `.mirai/thumbs/-.thumb`, and those of an id that the registry notes in + `Registry.removed` are deleted in the next frame (`thumbnails_forget`). - Raise `THUMBNAIL_RENDERER` when the same subject would give another image (studio, reduction, geometry or how a provider draws its image). A program's Slang source does not enter the key: changing it does not redraw the stored previews. @@ -846,15 +855,16 @@ plugin [vfx](../../mirai-world/references/vfx-editor.md#particle-document). ## Import -`import_file(s, path)` converts a source file into project assets, only in `Edit`, with -`sources.import_source` like `mirai import`. A source in the assets of the project or of a plugin -imports where it is; one from outside the project is first copied into `folder` -(`asset_folder_clean`, the folder of the browser), and one in the project but outside its assets -is refused. The source is versioned with its sidecar beside what it makes, and what it builds -that nobody edits goes to the imported folder -([identity](../../mirai-imports/references/imports.md#identity-and-reimport)). The imported folder -is brought up to date with the sources by the build that opens the editor -([sync](../../mirai-imports/references/imports.md#sync)). +`import_file(s, path)` ingests a source file or folder from anywhere on disk into the library of +the machine, where it is, and pins its package in the project, only in `Edit`, with +`sources.project_ingest` like `mirai import` +([library](../../mirai-imports/references/imports.md#library)). +Nothing is copied: the package is listed in `folder` (`asset_folder_clean`, the folder of the +browser; for a source in the assets without one, its own folder), in the project or in the plugin +whose assets that folder is. What nobody edits is listed from the blobs of the store, and the +documents are files of that folder. The store is the session's: `--store`, else the machine's +(`store.default_root`); the registry lists from the same one. The build that opens the editor +refreshes the packages ([refresh](../../mirai-imports/references/imports.md#library)). File > Import, the content browser, a file dropped on the window and `asset.import` call it. It imports in a thread of its own, with its own job system, while the editor keeps drawing; the content browser shows the dot of what works. `import_poll`, between frames, takes what it wrote on @@ -870,32 +880,53 @@ output folders in [importers](../../mirai-imports/SKILL.md). imported before the formats and takes no folder. - An image is sRGB unless its settings or its name say it is data or a normal (`sources.image_use`, which `mirai import` applies too); `asset.import {image}` chooses it. -- The sidecar of a source (`.import`) keeps its settings and the id of each asset; - reimporting writes the same assets with their ids where they are and keeps what did not change. - Over a document changed since its import wrote it, the dialog and a file dropped ask first - (`Modal.Import_Changes`, with what changed: discard and import, or cancel), and `asset.import` - fails naming it - ([changed documents](../../mirai-imports/references/imports.md#changed-documents)). - Textures with +- The package of a source keeps its settings and the id of each asset; reimporting writes the + same assets with their ids where they are and keeps what did not change. Over a document + changed since it was written, the dialog and a file dropped ask first (`Modal.Import_Changes`, + with what changed: discard and import, or cancel), and `asset.import` fails naming it + ([changed documents](../../mirai-imports/references/imports.md#changed-documents)). Textures with the same image and settings and meshes with the same geometry and settings come from their caches ([textures](../../mirai-imports/references/imports.md#texture-cache), [meshes](../../mirai-imports/references/imports.md#mesh-cache)). -- On finishing, the written files are refreshed in the registry and the replaced assets are - reloaded; if Surface programs were written (`Import_Report.programs`), the project's programs - are published (`programs_publish_request`) without waiting for another change. +- On finishing, the registry lists the packages of the root again and rescans, and the replaced + assets are reloaded; if Surface programs were written (`Import_Report.programs`), the + project's programs are published (`programs_publish_request`) without waiting for another change. - **Parts.** Meshes and skeletons of a model, and the materials and textures that are only its, are parts of its prefab (`asset.Header.owner`, [rule](../../mirai-imports/references/imports.md#parts)). `asset_detach` (`editor/assets_parts.odin`, `asset.detach`) takes one out of its owner as an edit of its file with undo: it only clears `owner`, and the import leaves it out; there is no UI that calls it yet. -- **What an import keeps in the imported folder is not edited where it is.** Moving, renaming, - detaching or deleting a part on its own is refused, saying to duplicate it. Moving the root of - an import of the cache (`asset_move`, the texture of an image) moves its source and sidecar, and - moving a folder moves the sources in it: the registry lists their assets there at once, without - an import. Deleting the root of an import (its prefab, the texture of an image) gives the import - up: its source and its sidecar go to the trash with what only the import can build. A - copy made outside the editor receives an id of its own on scanning - ([repeated ids](../../mirai-assets/references/formats.md#registry)). +- **What a package lists is not edited where it is.** Moving, renaming, detaching or deleting + another asset it lists is refused, naming the root of the package and saying to duplicate it. + The root of a package (the prefab of a model, the scene of a whole source, else its one asset + that is no part: the texture of an image) stands for it: moving it (`asset_move`) moves where + the project lists the package (`packages.mirai`, `sources.project_pin`), keeping the name its + source gives a listed asset, and moving a folder moves the packages listed in it. Deleting it + takes the package out of the project (`sources.project_unpin`): its documents go to the trash, + what it listed leaves the registry, its blobs stay in the store, and a source of it inside the + project goes to the trash while one outside stays. None of these is undone by undo; version + control brings `packages.mirai` back. A copy made outside the editor receives an id of its own + on scanning ([repeated ids](../../mirai-assets/references/formats.md#registry)). +- **Libraries.** `packages.list` lists the packages of the store with their assets (name, kind, + bytes, whether this machine has them built) and whether the project pins each; `packages.add` + pins one, or some of its assets, in a folder, as an import that runs in the thread of imports + and answers as `asset.import` (`sources.project_add`, `library.odin`). In the browser + (`content_library.odin`) the libraries are roots beside the project's content, as virtual + folders are: This machine and, in a session, one per other editor by the name the session gives + it ([libraries of the session](collaboration.md#libraries-of-the-session)). A root holds the + packages the project does not pin, in every arrangement and the search; a peer's package this + machine has shows once, under This machine, and one whose root goes with its peer. A package + shows its thumbnail, else the icon of its kind, its size and what is still to fetch, and its + fetch and why it failed on its card; the thumbnail is the stored preview of what it shows + (`asset.build_shown`: its root when its build holds it, else its first mesh). A card of This + machine whose store lacks it makes it as the project's content does: the packages the project + does not pin are listed by id under `LIBRARY_MOUNT`, a folder no path of the project reaches, + and only the visible cards make theirs, one a frame; a visible card of a peer's root that lacks it + asks that peer by its key, a few at a time (`collab_thumbs_update`, `share.fetch_start`): a + peer gives one it has and makes none, and nothing of the package is fetched for it; two of one + name in a root take the folder their source was in. Carried onto a folder of the project, or + Add to project of its menu (into the folder Create makes in), it is added whole; nothing else + adds from a library. ## Graphics diff --git a/.agents/skills/mirai-gameplay/references/demo.md b/.agents/skills/mirai-gameplay/references/demo.md index 030b9dc10..889a9617e 100644 --- a/.agents/skills/mirai-gameplay/references/demo.md +++ b/.agents/skills/mirai-gameplay/references/demo.md @@ -281,11 +281,11 @@ Content: others, the light meter alone: `lights` used to be clipped and `skeletal`, without light. No demo camera carries `Exposure`: the viewport and the game expose the same. - **Imports**: each image is a source beside the materials that show it (`assets/surfaces/bricks`, - `assets/sky`, `assets/gallery`), read as its sidecar says; each model is a source beside a - folder with its materials and the prefab of its nodes (`assets/gallery/plant.gltf` and - `assets/gallery/plant/`). The textures and the meshes are not versioned: they are in - `demo/.mirai/imported`, which `mirai build` makes again from the sources. - Re-importing keeps the ids, which the sidecars list + `assets/sky`, `assets/gallery`), and each model a source beside a folder with its materials and + the prefab of its nodes (`assets/gallery/plant.gltf` and `assets/gallery/plant/`), each a + package `demo/packages.mirai` pins. The textures and the meshes are not versioned: they are + blobs of the store, which `mirai build` refreshes from the sources or from a peer. + Re-importing keeps the ids, which the manifests of the packages list ([identity](../../mirai-imports/references/imports.md#identity-and-reimport)), and what the demo generates over what was imported reads them from there (`imported.odin`); what it generates on its own uses `asset.content_id`. What did not change is kept diff --git a/.agents/skills/mirai-imports/SKILL.md b/.agents/skills/mirai-imports/SKILL.md index 927b2f670..beafa55a9 100644 --- a/.agents/skills/mirai-imports/SKILL.md +++ b/.agents/skills/mirai-imports/SKILL.md @@ -20,11 +20,14 @@ the assets. ## Map - Packages, `import_source`, the imported model and CLI: [imports.md](references/imports.md#api). -- Where a source lives (in the assets, beside what it makes), its sidecar (settings, folder, ids), - what an import keeps in the imported folder and what in the assets, reimporting, a moved or - renamed source and parts with `owner`: [Identity](references/imports.md#identity-and-reimport). -- A checkout without its imported folder, what a build imports and when a source is imported - again: [Sync](references/imports.md#sync). +- The package of a source in the store (settings, ids, build of objects), documents written or + followed in a project (`documents_follow`), reimporting and parts with `owner`: + [Identity](references/imports.md#identity-and-reimport). +- The records of the store (package and build manifests, refs, `packages.mirai`): + [packages](../mirai-assets/references/formats.md#packages). +- Ingesting a source from anywhere, adding it to a project (`packages.mirai`), what a build + refreshes, a peer without sources, and when a source is imported again: + [Library](references/imports.md#library). - Adding or changing a format (`Format`, `Reader`, `Source_Space`, version): [Formats](references/imports.md#formats). - Import cache and versions: [Cache](references/imports.md#import-cache), @@ -53,13 +56,13 @@ the assets. ## Workflow -1. **Importing a file.** Put it in `assets/`, in the folder of its theme beside what it makes - (the editor copies a file from outside into the folder of the browser), and - [`mirai import`](references/imports.md#what-is-written) it; review the assets with - `mirai asset verify`, `ls` and `info`. Commit the source, its `.import` and the documents it - wrote beside it; what it wrote in `.mirai/imported/` is never versioned. The editor, the CLI, - the sync of builds and the content tools import with the same `sources.import_source`: same - assets, prefab included. +1. **Importing a file.** An import takes the store (`Import_Options.store`) and puts in it the + package of the source and a build of its objects; given `documents`, it also writes or follows + the documents in that folder of a project. Review the assets with `mirai asset verify`, `ls` + and `info`; commit `packages.mirai` and the documents, never the objects. The editor, the CLI, + the library and the content tools import with the same `sources.import_source`: same assets, + prefab included. `mirai import ` ingests it and adds it to a project + ([library](references/imports.md#library)). 2. **From code.** `sources.import_source(path,options)` writes everything; for the model without writing it, `importer.import_model(path,sources.FORMATS[:],options)` reads it into an `Imported_Model` (plain data, no engine components) and `importer.import_write` writes its @@ -75,8 +78,10 @@ the assets. model format) or an `Import_Source_Provider` (a source that imports whole); the editor and the import program of the engine add its formats to `sources.FORMATS`, which holds only the engine's own ([plugins](references/imports.md#plugin-formats)). -6. **Regenerating imported assets.** `mirai build` builds what is missing or stale of a project - and its plugins once its target stands ([sync](references/imports.md#sync)). +6. **Regenerating imported assets.** `mirai build` refreshes the packages a project and its + plugins pin once its target stands (`mirai import --refresh `): builds from the store, + from a peer, or by ingesting a source this machine has + ([library](references/imports.md#library)). ## Invariants @@ -91,40 +96,60 @@ the assets. place after the read: from the `front` its settings state, or else, for a file that is a rig, from where its body is found facing ([front](references/imports.md#front)). - **The source is not modified.** The same bytes and options give the same assets. -- **A source lives in the assets, beside what it makes, in folders by theme.** Never a folder per - asset type; the browser filters by type. One outside the assets is refused. Decided by the +- **The library never removes and never builds without the source.** A refresh takes a build + from the store, the caller's source or an ingest of an origin on this machine; else it names + the package and what gives it. Nothing is deleted because a source is missing. Decided by the owner. -- **The ids of an import live in the sidecar of its source, and nothing derives them.** The first - import gives random ids and `.import` lists them; importing again, on any machine and - in any order of sources, gives each part the asset it had by its kind and name, even if its - document was moved, and a source moved or renamed with its sidecar keeps its import +- **A source imports from anywhere.** In a project, in a plugin or in no project: what it makes + goes to the store, and its documents to the folder of the project given. In a project, it lives + beside what it makes, in folders by theme, never a folder per asset type. +- **The ids of an import live in the manifest of its package, and nothing derives them.** The + first ingest gives random ids; importing the package again, on any machine, gives each part + the asset it had by its kind and name, even if its document was moved ([identity](references/imports.md#identity-and-reimport)). A part renamed in the source is a new asset. Only what is surely part of a model belongs to its prefab (`owner`), and it is decided once ([parts](references/imports.md#parts)). - **An import never touches another.** Two models that embed the same image get a texture each, - encoded once in the content cache; no import edits the files or the sidecar of another source. - Decided by the owner. -- **What nobody edits is a cache of the source, never versioned, whatever reads it.** The meshes, - textures, sounds and fonts of every import, of an engine format or of a plugin's, are files of - the imported folder of its root (`.mirai/imported/`), named by their ids and listed by where - their sidecar is, built again from the source and its sidecar by - [sync](references/imports.md#sync); materials, prefabs, skeletons, clips, texts and prompts, - which the editor edits, are documents of `assets/`. Decided by the owner. -- **The source, its sidecar and the build decide the cache.** The same three give the same files, - byte for byte: a clone, a worktree and a branch switch are brought up to date by the same sync, - explicitly, in `mirai build`; the editor never imports by itself. + encoded once in the content cache; no import edits the documents or the package of another + source. Decided by the owner. +- **Every output is an immutable object of the store, the same on every machine** + (`asset.object_put`, cut where its format reads alone). The build lists the objects, documents + among them as their baseline. No object is rewritten in place (a changed owner is a new + object), and the source an asset names is relative to its package. A document of a plugin's + kind a refresh writes takes the extension the plugin's manifest gives its kind. Materials, + prefabs, skeletons, clips, texts and prompts, which the editor edits, are also documents of a + project, followed from the baseline. Decided by the owner. +- **The identity holds no engine version; the key holds the identity and the versions.** The + identity (package, source, inputs, settings, plugin code: two packages of the same bytes never + share a build, since its objects hold their ids) is in the manifest; the key + (`asset.import_key`) adds the writer, format, kind and texture builder versions, so any machine + with the manifest finds a build without the source. A Final build is named under `imports`, a + Fast one under `imports-fast`, and peers share both; `builds` names the one this machine chose. + The editor never imports by itself. +- **A share serves what the host has; a refresh takes what its build accepts.** A refresh that + does not cook (the editor's, a play build's, a join's) takes a Final or a Fast build, from the + store first and then from the source, before importing anything; one that cooks asks only + `imports` and takes a build only under the namespace its manifest's `final` matches, so a Fast + build never reaches a cook, and a package with no Final reachable fails by name. +- **What imports and cooks build is kept as refs into the machine store** (`cook/cache`): an + object named by `(cache-, -)`, found by any program or machine that shares + the store's objects; a miss asks the caller's `cache.Source` before building, what a pass needs + is looked up at once (`cache.lookup`, one fetch for the pass), and `cook/cache` removes nothing: + the store owns space ([texture cache](references/imports.md#texture-cache)). +- **Following a package is a pure function of the package, its build and the game's types**: + two refreshes on any machine give the same bytes, and a document, prefab or record that would + be written as it is is not written (`serial.file_write_changed`). - **Reimporting keeps what a person changed.** A material stores what the source gave and the next import rewrites only that: fields changed afterwards stay, like the overrides of a prefab ([materials](references/imports.md#what-is-written)). Any other document it writes over is never written over unasked: one changed since the import wrote it stops the import, which names - it, and only `discard_changes` writes over it; a sync leaves it as it is and warns + it, and only `discard_changes` writes over it; a refresh leaves it as it is and warns ([changed documents](references/imports.md#changed-documents)). - **What an asset was built from is its `Import_Info`**: source (size and hash), writer and format - version, and only the settings the asset is made from. Whether a file of the cache is current - is said once, by its name (the key of what the import builds from), and a mesh or a texture - whose file under another key records what it would be built from now takes that file; the rest - is rebuilt, and a texture of the same image and settings or a mesh of the same geometry and - settings comes from the content cache. + version, and only the settings the asset is made from. Reuse is a ref lookup: a mesh or a + texture that a build of the package has, recording what it would be built from now, is taken; + the rest is rebuilt, and a texture of the same image and settings or a mesh of the same + geometry and settings comes from the content cache. - **In parallel, the same bytes.** What is expensive (levels, clusters and colliders of a mesh, levels of a texture) is built in jobs; what decides ids and files is written in a single thread, in the usual order. Importing with or without the job system gives the same files. @@ -133,8 +158,8 @@ the assets. order of the scalar loop: the tool and the dev engine (`-o:minimal`) write the same as `--release`. - **Importing is a transaction.** `import_model` leaves nothing if it fails (own arena, freed by - `model_destroy`); `import_write` removes the files it created if it fails midway. Errors name - the file, the line or the prim. + `model_destroy`); `import_write` removes the documents it created and names nothing if it fails + midway. Errors name the file, the line or the prim. - **A source does not choose what is read from disk.** A path with a control character or, on Windows, a device (`CON`, `NUL`, `COM1`) is not opened. Models only ask for files next to them or by path, never more than `MAX_SOURCE_BYTES`, and the hierarchy is bounded (`MAX_DEPTH`). @@ -157,10 +182,10 @@ the assets. `TEXCOORD_0/1`, Draco, meshopt indices and normals, GPU instancing, subdivision surfaces and time samples fail with a named error. What is ignored without breaking the source (material extensions with no equivalent, missing images) leaves a warning. -- **Settings live with the source.** `.import` decides how it is imported and wins over the - options of whoever imports (except `ignore_stored`); every import writes it. Everything that - changes an asset is a setting, also the alpha coverage of a loose image. An unreadable sidecar - is an error, not the defaults nor new ids. +- **Settings live in the package.** Its manifest decides how the source is imported again and wins + over the options of whoever imports (except `ignore_stored`). Everything that changes an asset + is a setting, also the alpha coverage of a loose image. An unreadable manifest is an error, not + the defaults nor new ids. - **What is imported is used as is.** Levels of detail are made on import and stored in the `.mesh`; loading does not recompute them. - **Collision is the source's, as authored.** The colliders the source brings are the mesh's and @@ -172,11 +197,11 @@ the assets. - **Textures in blocks are encoded with a perceptual metric and RDO** at `RDO_LAMBDA_DEFAULT` unless the setting `rdo_lambda` says otherwise. - **The editor never waits for the final encode** (`textures.Encode_Speed`): its imports take the - Final of the texture cache, else build Fast (no RDO, low effort, `Texture_Flag.Fast`). The sync - of a build that does not cook (the editor's) and `mirai import` build Fast too and take Fast or - Final as current, so the command takes what the editor's sync left; only the sync of a build - that cooks, or of `mirai package engine` (`--final`), builds Final, and so does the editor for a - texture that is a document. + Final of the texture cache, else build Fast (no RDO, low effort, `Texture_Flag.Fast`). The + refresh of a build that does not cook (the editor's) and `mirai import` build Fast too and take + Fast or Final, so the command takes what the editor's import left; only the refresh of a build + that cooks, or of `mirai package engine` (`--final`), requires Final, and the editor builds + Final a texture that is a document. - **One algorithm, once.** Mips, color rules and block compression in `cook/textures` (the encoders are `compressor:blocks`); PBR conversions in `cook/import/pbr.odin`; triangulation, simplification, hulls and convex decomposition in `std/geom`; transforms with `std/math`. @@ -194,13 +219,20 @@ the assets. inverted green, round trip through `.tex`, cache by content; HDR, every BC6H mode, 16-bit and layers in `layers_test.odin`). The fixtures of `assets/fixtures/` and the models of `demo/assets` import (without warnings) and what is written reads back with `asset`. -- Caches: `cook/textures/cache_test.odin` and `cook/import/sources/cache_test.odin` (a texture or - a mesh that another import built is taken from the cache; one whose file under another key - records what it is built from now is taken without writing it, and one that records other +- Caches: `cook/cache/cache_test.odin` (a result kept through one store is found through another + of its root or through a source on a miss; an object not here is a miss; a lookup of many keys + asks once; hostile names), `cook/textures/cache_test.odin` (and one texel changed changes few + chunks of the object) and `cook/import/sources/cache_test.odin` (a texture or a mesh + that another import built is taken from the cache or from a source; one whose file under another + key records what it is built from now is taken without writing it, and one that records other encoders or another builder is built again); `cook/build/cook_cache_test.odin` (a cook takes the blocks of a texture from the cache of its - content) and `cook/import/sources/cook_test.odin` (the sync of a cook builds a fast texture - final from its source; that of a build that does not cook keeps either). + content, ships package assets from their objects and looks the cache up once per pass) and + `cook/import/sources/cook_test.odin` (the refresh of a cook builds a fast texture + final from its source; that of a build that does not cook keeps either; a peer with only + fetched builds cooks what the sources cook, sources inside a root stay behind and a missing + build fails by package; a refresh takes builds from a second store; a cook never takes a Fast + build a source holds, and ships once that source holds the Final one). - Parallel: import a map with `mirai import --rebuild` over a copy of an import made without the job system (or before the change) and compare the trees with `diff -r`: the same bytes. - Also test `cook/prefab` if the imported hierarchy changes, and `asset` if an asset format or its @@ -211,25 +243,31 @@ the assets. `prompts_kenney` pack again ([command](../mirai-plugins/references/plugins.md#first-party-plugins)) and verify it with `mirai asset verify`. -- Identity, the imported folder and sync: `cook/import/sources/reimport_test.odin` (the parts of - an import name their prefab and keep their ids where they were moved, a source keeps the ids of - its import with its sidecar, a part renamed in the source is a new one, two models with the same - image get a texture each and neither touches the other, a checkout without its imported folder - gets the same assets byte for byte and only what changed is imported again, the sources of a - plugin format build in the cache and sync reads them with the formats of the plugins, a source - outside the assets is refused, when a material is a part of a model, sync runs the importer of - a plugin for a whole source whose inputs changed), `cook/import/outputs_test.odin` (the models - of a whole source share their parts by name, names with `/` go to subfolders, inputs count in - the key), - `asset/registry_test.odin` (the listing of the cache by where the sidecar is, never by the order - of a scan), `cook/import/sources/settings_test.odin` (a sidecar that is not one is refused) and - `cook/import/sources/write_test.odin` (importing again keeps what was changed in a material and - takes the rest from the source); a document changed since its import wrote it: - `an_import_does_not_write_over_a_document_changed_since_it_wrote_it` and the whole source of - `sync_imports_again_a_whole_source_when_what_it_is_built_from_changes` (reimport tests), the - question of the editor in `editor/import_api_test.odin`. After changing what an import writes, - prove it on the demo: delete `demo/.mirai/imported`, `mirai build editor demo`, and - `git status` shows no sidecar or document changed. +- Identity: `cook/import/sources/reimport_test.odin` (the parts of an import name their prefab and + keep their ids where they were moved, a source keeps the ids of its package with the manifest a + project pins and a copy is another package, a part renamed in the source is a new one, two + models with the same image get a texture each and neither touches the other, when a material is + a part of a model), `cook/import/outputs_test.odin` (the outputs of a source are the assets its + package lists, the models of a whole source share their parts by name, inputs count in the + identity, importing again keeps ids and objects, a source outside any project imports, a setting + changes the identity and an engine version only the key, a Fast build is named apart, two stores + hold the same objects), `cook/import/sources/settings_test.odin` (the package keeps the settings; + settings that are not this build's are refused) and `cook/import/sources/write_test.odin` + (importing again keeps what was changed in a material and takes the rest from the source). +- Library: `cook/import/sources/library_test.odin` (a peer without sources fetches, lists and + loads what a project pins, and without a source of builds reports by package and neither + removes nor builds; a source outside any project ingests once and two projects list the same + objects; an engine change ingests again where the source is and is reported by name elsewhere, + also when only Fast is here and Final is required; hostile `packages.mirai` and manifests are + reported; two ingests of one source at once import it once; a copy of a root with its sources + and an empty store builds the same ids, manifests and objects, and without them reports each + package; refreshing from two empty stores writes nothing the second time) and + `cook/import/sources/reimport_test.odin` (a refresh reads a source again with the formats of + the plugins, ingests a whole source again when what it is built from changes, and finds the + build of a whole source by key without the source, missing it for another importer). +- Changed documents: `an_import_does_not_write_over_a_document_changed_since_it_wrote_it` and + the edited case of `a_refresh_ingests_again_a_whole_source_when_what_it_is_built_from_changes` + (reimport tests), the question of the editor in `editor/import_api_test.odin`. - `cook/import/sources/cli_test.odin` needs `MIRAI_ROOT` (defined by `mirai test`); run it when touching `cook/import/sources/command.odin`. It imports in-process with `command`; the whole path (`mirai import` → `mirai-import`) needs the built engine. @@ -244,7 +282,7 @@ the assets. rejected. The binding is regenerated with `uv run scripts/ufbx_binding.py` ([ufbx](../../../third_party/ufbx/UPSTREAM.md)). - `fuzz_test.odin` truncates each fixture to all its lengths and changes bytes (OBJ, glTF, GLB, - USDA, binary and ASCII FBX, WAV, Ogg, PNG and the `.import` text): each attempt is imported or + USDA, binary and ASCII FBX, WAV, Ogg, PNG, fonts and prompts): each attempt is imported or rejected, without crashes, hangs or leaks. A new format extends those tests. - A change that must give the same bytes is compared by importing with the CLI before and after (`--rebuild` on the second pass, or in new folders) and with `cmp` of each asset. diff --git a/.agents/skills/mirai-imports/references/imports.md b/.agents/skills/mirai-imports/references/imports.md index 47dad01c9..ace2002d6 100644 --- a/.agents/skills/mirai-imports/references/imports.md +++ b/.agents/skills/mirai-imports/references/imports.md @@ -9,19 +9,19 @@ Packages, all *engine-time*: - `cook/import/obj`, `cook/import/gltf`, `cook/import/usd` and `cook/import/fbx` (on top of ufbx): one format each, with its `FORMAT`. They import the core and nothing from the other formats. - `cook/import/sources`: the `FORMATS` table (the engine's formats), `source_kind` and - `import_source`, used by the editor (`editor/import.odin`), `mirai import` (`command.odin`, in the - engine's `mirai-import` program) and `sources.sync`. The editor adds the formats of plugins - ([plugin formats](#plugin-formats)). + `import_source`, used by the [library](#library) (`ingest`, `project_ingest`, `project_add`, + `project_refresh`), through which the editor (`editor/import.odin`) and `mirai import` + (`command.odin`, in the engine's `mirai-import` program) import. The editor adds the formats of + plugins ([plugin formats](#plugin-formats)). - `cook/textures`: images to `.tex` levels. ## API - `sources.import_source` imports a file of the assets folder of a project or a plugin: a model writes its assets and, if it has nodes, its prefab; an image, its `.tex`; a sound, a font or a - prompt pack, its asset. The [sidecar](#identity-and-reimport) of the source - keeps its settings, the folder of its assets and the id of each asset: importing it again gives - the same assets. `sources.sync` imports every source of a root whose outputs are missing or - stale ([sync](#sync)). + prompt pack, its asset. The [package](#identity-and-reimport) of the source in the store keeps + its settings and the id of each asset: importing it again gives the same assets. The + [library](#library) ingests sources, pins them in projects and refreshes their builds. - `sources.source_kind` says what a file is from its first bytes ([sniffing](#formats)), except what the signature does not settle: TGA by extension and prompts by `.prompts.json`. It carries the formats it searches with (`formats`, `FORMATS` by default), like `model_folder` @@ -30,23 +30,24 @@ Packages, all *engine-time*: (`sources.FORMATS[:]` or those of a test) into an `Imported_Model` in its own arena (`model_destroy`); `importer.import_write` writes its assets and returns the `Import_Report`. `prefab.model_import` does both with the prefab (`cook/prefab.model_prefab`). -- `Outputs` (`output_root`, `output_part`, `output_model`, `output_input`): the assets of an - import that is not that of one model (a scene, a source a plugin imports whole) are those the - sidecar of their source lists, like those of a model: the root by its kind (and its name when - the sidecar lists several of the kind) and every other by kind and name; `outputs_end` writes - the sidecar. `output_model` writes a model as parts of the import with the writer of - `import_write` (cooking ahead, Fast textures, kept files, the caches); its prefab is a scene of - the model's name that the caller writes with `prefab.model_prefab`. A part whose kind and name - the import already gave is that asset, written once: two models that name the same material - or texture share it. `output_input` lists a file the import reads in the sidecar and in its - key; every input comes before the first output of the cache, which fixes the key. +- `Outputs` (`output_root`, `output_part`, `output_model`, `output_input`, `output_written`): the + assets of an import that is not that of one model (a scene, a source a plugin imports whole) are + those the package of their source lists, like those of a model: the root by its kind (and its + name when the package lists several of the kind) and every other by kind and name. The caller + writes each to the file its part gives and `output_written` takes it; `outputs_end` puts the + manifest and the build in the store. `output_model` writes a model as parts of the import with + the writer of `import_write` (cooking ahead, Fast textures, taken objects, the caches); its prefab + is a scene of the model's name that the caller writes with `prefab.model_prefab`. A part whose + kind and name the import already gave is that asset, written once: two models that name the + same material or texture share it. `output_input` lists a file the import reads in the package + and in its identity; every input comes before the first output of the cache, which fixes it. - For engine-time tools: `mesh_levels` ([levels](#levels-of-detail)) and the PBR conversions of `pbr.odin`. - `Error` is `{code,message}`; `message` lives in the caller's `temp_allocator`. `Import_Options` - carries the settings and how they apply: `ignore_stored`, `root` (the project or the plugin - whose assets folder has the source; `import_source` finds it from the path), `registry` (the - scanned assets of the root, else the import scans them), `rebuild`, the caches and `worker` - ([in parallel](#in-parallel)). + carries the settings and how they apply: `ignore_stored`, `store` (required), `manifest` (the + package to ingest again), `documents` (the folder of a project its documents go to), `root` and + `registry` (the project and its scanned assets, where documents are found by id), `rebuild`, + the caches and `worker` ([in parallel](#in-parallel)). ## Formats @@ -153,11 +154,13 @@ part ([extension](../../mirai-editor/references/extensions.md#importers-of-a-plu - `Import_Format_Provider{format}` is one more `importer.Format`: it is imported like the engine's, with the same storage (the cache and the documents, [identity](#identity-and-reimport)). `Format.folder` says the folder of its assets. Its extensions are in the Models filter of the - dialog. Its `prepare` gives the format's options for an import (`Format.data`) from the assets - of the project and the library of its plugins, and `release` frees them when it ends. -- The editor imports with the providers of its library; the import program of the engine - (`mirai import`, the [sync](#sync) of every build) loads the editor library of the project and - takes the same providers from it (`editor.import_library`, `sources.Library`), only when a + dialog. Its `prepare` gives the format's options for an import (`Format.data`) from the store of + the library, the root the import is for and the types of its game, and `release` frees them + when it ends. +- The editor imports with the providers of its session, which it prepares and gives the library + (`Library_Options.imports`); the import program of the engine (`mirai import`, the + [refresh](#library) of every build) loads the editor library of the project and takes the same + providers from it with its store (`editor.import_library`, `sources.Library`), only when a source is read by no format of the engine or a model imports, whose prefab is written in the types of the game the library has. `mirai import` first builds, as `mirai build editor` does, the editor library of the project of what it imports while it is missing or not built from @@ -167,17 +170,21 @@ part ([extension](../../mirai-editor/references/extensions.md#importers-of-a-plu is the single source of every asset it makes (a scene, the models it reads with `output_model`), and the files it reads besides it are its `inputs`, like the `.bin` of a glTF. It appears with its name as a filter of the dialog and claims a file with `claims(path, head)` - before the formats. One from outside the assets is brought in by its `bring`, with the files it - reads, else copied alone. The import program takes these importers from the same library and - [sync](#sync) runs them. The Unreal content importer is the `unreal` plugin of the catalog + before the formats. Its `name` is the importer its packages record (the `importer` its run gives + `outputs_begin`) and its `version` the one its run gives `import_version`: with the identity of + the plugin's code they make the key of a build (`importer.build_key`), so a machine with the + plugin and without the source finds the build by key, and a changed importer builds again. The + import program takes these importers from the same library and the [refresh](#library) runs + them. The Unreal content importer is the `unreal` plugin of the catalog ([git plugins and catalog](../../mirai-plugins/references/plugins.md#git-plugins-and-catalog)). ## Import cache -The `Import_Info` of each asset says what it was built from: source (relative to `root`), name, +The `Import_Info` of each asset says what it was built from: source (relative to its package: the +name of its file), name, index, size and hash of what was read (the source; the image of a texture; the mesh as read, with its slots, joints and skeleton, of a mesh), version and `settings_hash`. It is not what identifies -the asset: the [sidecar](#identity-and-reimport) is. +the asset: the [package](#identity-and-reimport) is. - `importer_version` combines `IMPORTER_VERSION` (the writer) with `Format.version` (`import_version`); loose images and sounds use format version 0. @@ -186,93 +193,91 @@ the asset: the [sidecar](#identity-and-reimport) is. (`mesh_build_hash`); for a texture, its `Build_Options` with `BUILD_VERSION` and the block encoders (`textures.encoders_id`); for a sound, whether it is `Stream`. Materials, skeletons and clips store 0. Changing a texture setting keeps the meshes, and one of the meshes, the textures. -- What is current is decided once, by the name of a file of the cache: `import_write` keeps a mesh - or a texture whose file under the key of what the import builds from now exists - (`Written.kept`) and builds nothing ahead when every output of the cache is kept; if only the - `owner` changes, it rewrites the header. A mesh or a texture with no file under that key takes - the file of its id under another key, renamed to it, when that file is of the version its kind - is written in and its `Import_Info` is what the import records now: a change of the key that - does not reach it (the code of a plugin, a setting of other assets) neither builds nor writes - it. A texture of an import of fast textures takes one built Fast only while the texture cache - holds no Final of it. Materials, skeletons, clips and prefabs, which are - cheap, are always written, with what a person changed in them ([what is - written](#what-is-written)), also with `rebuild`. -- A change of writer or format version, of the code of the plugin that imports a source whole, - of the settings or of `kind_version` of a kind of the cache - ([version rule](../../mirai-assets/references/formats.md#versions)) changes the key and - rebuilds; `rebuild` (`mirai import --rebuild`) rebuilds everything. +- What is current is a ref lookup: `import_write` takes the object a build of the package has for a + mesh or a texture (`Written.kept`), from the build of the key of what the import is made from + now, else from the build this machine chose for the previous manifest, when the object is of the + version its kind is written in, has the `owner` decided now and its `Import_Info` is what the + import records now: a change that does not reach it (the code of a plugin, a setting of other + assets) neither builds nor writes it. It builds nothing ahead when the build of the key has + every output of the cache. A texture of an import of fast textures takes one built Fast only + while the texture cache holds no Final of it. Materials, skeletons, clips and prefabs, which + are cheap, are always written, also with `rebuild`. +- A change of writer or format version, of a kind version or of the builder of textures changes + the key (`asset.import_key` of the identity and `import_versions`); one of the source, an + input, the settings or the code of the plugin that imports it whole changes the identity too; + `rebuild` (`mirai import --rebuild`) rebuilds everything. ## Identity and reimport -A source lives in the assets folder of a project or a plugin, beside what it makes, in the folder -of its theme (never a folder per asset type: the browser filters by type). `.import`, the -sidecar, is what is versioned of an import with its source (`asset/sidecar.odin`, read by the -registry and by the importer alike): a text with the settings, the `folder` beside the source its -assets go to (a model's: its stem, or what `Format.folder` says; an image's, a sound's: none, they -go beside it), the files the import read besides the source (`inputs`) and one output per asset -the source gives, by kind, name and index, with its id. Nothing else decides an id: the first -import gives each part a random one and the sidecar keeps it; importing again, on any machine and -in any order of sources, gives each part the asset it had. A sidecar that does not parse is an -error that names it, never the defaults nor new ids. A source in no assets folder is refused, -saying where it goes; `import_source` finds its root (the project or plugin) from its path. - -What an import writes is of two sorts (`asset.import_cached`), whatever the format that reads the -source, the engine's or a plugin's: - -- **The cache**: meshes, textures, sounds and fonts. Nothing edits them after the import, so the - source and the sidecar give them back whole. Each is a file of the imported folder of the root - (`project.imported_path`: `.mirai/imported/`), named by its id and by the key of what it was - built from (`import_key`), never versioned. Nothing stores where it is listed: the - [registry](../../mirai-assets/references/formats.md#registry) lists it at - `//`. [Sync](#sync) builds it again where it is - missing. -- **Documents**: materials, prefabs, skeletons, clips, texts and prompts. The editor edits them, - so they are files of the assets folder, versioned. An import finds each by its id in the - registry of the root, wherever a person moved it, writes again what the source gives and keeps - what was changed ([what is written](#what-is-written)); a new one goes to the folder of the - import. -- **A name with `/`** is a folder under that of the import, for the cache as for documents: - each part made safe (`asset.safe_path`), without empty, `.` and `..` parts. +An import ingests its source as a package of the store +([packages](../../mirai-assets/references/formats.md#packages)), from wherever the source is: in +the assets of a project, of a plugin or in no project. Its manifest is what identifies the +import: the origin (where the source was, a hint), the identity (`identity_of`: the package, the +bytes of the source and its inputs, its settings and the code of a plugin that imports it whole, +no engine version), the settings, the inputs and one asset per thing the source gives, by kind, +name and index, with its id, owner and dependencies. Nothing else decides an id: the first ingest +gives each part a random one; importing again (`Import_Options.manifest`, else the manifest the ref +`origins` of this machine names for the path of the source) gives each part the asset it had. A +manifest that does not read is an error that names it, never the defaults nor new ids. + +What an import writes is an object of its build (`writer_put`, `asset.object_put`), whatever the +format that reads the source, the engine's or a plugin's; each part is written to a scratch file +first, and the store dedupes its chunks. Objects never change in place and are the same on every +machine: the source an asset names is relative to the package, a changed owner is a new object. +`Written.path` is the file of a document, and an output of the cache has only its object +(`written_open` opens either; `mirai import` prints `object
`). An import writes with +`Codecs.None` ([compression](../../mirai-assets/references/formats.md#compression)), so its +objects never depend on a compressor and every engine of the same formats reads every build. The +build lists the object of each output and whether it is Final: the import key leaves the encoders +out, so engines with either compressor share builds, from their store or a source. It is named under +`imports/` when Final, else under `imports-fast/`, and as the build +this machine chose (`builds/`), which a program without importers finds. Of two sorts +(`asset.import_cached`): + +- **The cache**: meshes, textures, sounds and fonts. Nothing edits them after the import: the + project lists them from its packages. +- **Documents**: materials, prefabs, skeletons, clips, texts and prompts. The build keeps them as + the source gives them (their baseline); a project edits a file of each, written or followed in + a folder of its assets by `documents_follow` (`cook/import/documents.odin`), or by the import + itself when given `documents`: found by its id in the registry of the root wherever a person + moved it, else a new file there ([what is written](#what-is-written)). The prefab of a model is + written in the types of the game, by the caller, at `Import_Report.prefab_path`. +- **A name with `/`** is a folder under that of the documents: each part made safe + (`asset.safe_path`), without empty, `.` and `..` parts. - **The root** of an import is the prefab of a model, the texture of an image or the sound of a - sound, without `owner`: the output of its kind the sidecar lists. A model with nodes that has + sound, without `owner`: the asset of its kind the package lists. A model with nodes that has none receives a new prefab. -- **A part** is the output of its kind and name (among equal names, the one of its index); one the - sidecar does not list is new. A part renamed in the source is a new asset: what named the old - one does not silently get another. An output the source no longer gives leaves the sidecar with - a warning: its file goes with it when it is of the cache, and stays as a plain asset when it is - a document. -- **A source moved or renamed** keeps its import as long as its sidecar goes with it, and takes - what it has in the cache with it without an import: the registry lists those assets where the - sidecar is at its next scan, which a refresh of a sidecar or of its source triggers (the editor - moves a source with its sidecar, and a folder with its sources). A source that is deleted takes - them out of the listing at once. A copy of the source without its sidecar is another import, - with ids of its own. +- **A part** is the asset of its kind and name (among equal names, the one of its index); one the + package does not list is new. A part renamed in the source is a new asset: what named the old + one does not silently get another. An asset the source no longer gives leaves the package with + a warning; a document of it stays in a project as a plain asset. - **An import never touches another.** Two models that embed the same image get a texture asset - each; the [texture cache](#texture-cache) keeps one encoded copy. No import writes a file or a - sidecar of another source. Decided by the owner. + each; the [texture cache](#texture-cache) keeps one encoded copy. No import writes a document + or a package of another source. Decided by the owner. - **The textures of a model** name the model as source and the image as name, with the size and hash of the image. ### Changed documents -An import never writes over a change it did not make without being told to. The sidecar records, -for each document the import writes over, the `content_hash` of its header as the import last -wrote it (`written`); the registry already has the hash of each file, so finding what changed -reads no document. A change of owner alone (`asset.detach`) is none. A material records nothing: -the import follows its source only where nobody changed it. - -- `Import_Options.changed` says what an import does with a changed document: `Refuse` (the - default) writes nothing and fails with `Error_Code.Changed`, naming the files and how to go on; - `Discard` writes over them, with what its writer keeps of a person's (the body parts and sockets - of a skeleton, the roots of clips); `Keep` leaves them as they are, imports the rest and warns of - each. +An import never writes over a change it did not make without being told to. The project's pin +records, for each document of a package whose file the import or the refresh writes over, the +`content_hash` of that file's header as it last wrote it (`asset.Package_Written`): a fact of the +project, never of the manifest. The registry already has the hash of each file, so finding what +changed (`importer.documents_changed`) reads no document. A change of owner alone +(`asset.detach`) is none. A material records nothing: the import follows its source only where +nobody changed it. + +- `Import_Options.changed` says what an import into a project does with a changed document: + `Refuse` (the default) writes nothing and fails with `Error_Code.Changed`, naming the files and + how to go on; `Discard` writes over them, with what its writer keeps of a person's (the body + parts and sockets of a skeleton, the roots of clips); `Keep` leaves them as they are, imports + the rest and warns of each. The objects of the build are written either way. `discard_changes` of `asset.import`, `--discard-changes` of `mirai import` and the question of - the editor are `Discard`; [sync](#sync) is `Keep`, so a build never loses an edit and still - builds the cache. -- What a writer gives of a kept document goes to a file of the imported folder that the import - removes when it ends, so a plugin that imports a whole source keeps the same calls; the prefab of - a model it keeps has no `prefab_path`. + the editor are `Discard` (`Library_Options.changed`); a [refresh](#library) and the follow of a + package added are `Keep`, so a build never loses an edit and still builds the package. +- A document kept keeps what the pin recorded, so it stays changed until the changes are + discarded; the prefab of a model it keeps has no `prefab_path`. ### Parts @@ -297,8 +302,8 @@ what surely belongs to that model is a part; what is doubtful stays independent edges do not tile` or `of its own: `. - **It is decided once.** A document keeps the `owner` its file has: a part taken out of its owner (`asset.detach` in the editor, which only clears `owner`) is still an output of the import, which - writes it and leaves it outside. An output of the cache keeps what its `part` says in the - sidecar. Only `parts` decides it again. + writes it and leaves it outside. An asset keeps what its package says (`part` of the cache, + `owner` of a document). Only `parts` decides it again. - `Import_Settings.parts` (`mirai import --parts auto|all|none`): `Auto` the above; `All` makes parts of all the materials and textures of the model, and `None` of none, also of those that already existed. @@ -306,54 +311,68 @@ what surely belongs to that model is a part; what is doubtful stays independent an id of its own: no import writes it ([repeated ids](../../mirai-assets/references/formats.md#registry)). -## Sync - -`sources.sync(root, options)` brings the imported folder of a project or a plugin up to date with -the sources of its assets folder (`cook/import/sources/sync.odin`): it is what makes a checkout -that was just cloned, a worktree that was just made or a branch that changed a source run. - -- It imports each source whose sidecar lists an output of the cache with no file named by the key - of what the import builds from now (`importer.import_key`): the bytes of the source and of every - file it read (the buffers and images of a glTF, the material library of an OBJ, the images of a - pack of prompts), its settings, the versions of the importer and of what it writes and the block - encoders of the program, with textures Final. An import of documents alone (clips of a glTF - without meshes) leaves instead an empty stamp named the same way (`importer.STAMP_EXTENSION`). - A sync or an import with `fast_textures` (what `mirai import` and a build that does not cook - run; `--final` turns it off) also takes outputs built Fast as current and imports with Fast - textures; Final stays current there. One notion of - current, the bytes: a source that only moved is not imported, and one whose `.bin` or image - changed is. The hash of a file whose size and modification time are those the cache of the root - keeps for it (`importer.Source_Hashes`, in `.mirai/cache/sources`) is taken from there, so a - build reads only the files that changed. A USD is composed by `usdcat`, which does not say which - layers it read: after changing one alone, import the stage. -- A document changed since its import wrote it is left as it is, with a warning that names it - ([changed documents](#changed-documents)): the rest of the import is written. -- It also imports a source with a document of a version of its kind this build does not read (the - engine's in `asset.KINDS`, a plugin's in the `version` of its `asset_kinds`): the import writes - it again in the current version, with its id and what was kept of it. -- A source no format of the engine reads goes to the importer of a plugin that claims it whole, - else to the formats the plugins add (`Sync_Options.library`); one nothing reads is a failure - that says so. A whole source is keyed like any source of no format (`import_version(0)`) plus - the code of the plugin that claims it (`sources.code_identity` of its folder: its Odin sources - but tests, with the plugins it holds, as `Whole_Source.code` and - `Import_Options.importer_code`), so a change of that code imports it again with no version to - raise; its importer reads the settings its sidecar keeps, as every import does. -- It finds what to import first, then imports it, saying as it goes (`Sync_Options.progress`) a - line per source with why and how many of the sources it is, and under it, every - `importer.PROGRESS_EVERY`, the assets written of those its sidecar listed - (`Import_Options.progress`). `mirai import` passes its output, and the build shows the lines of - the import program as it prints them. -- A file of the imported folder that no sidecar with its source next to it lists under its - current key is removed. A source that does not import is a failure of the report that names it: - the others are imported, and nothing is removed. -- **Who runs it.** Every build of a project once its target stands (`build.job_imports`, with - `imports_sync`): `mirai build` of the editor, the game or the server, `mirai play`, and the - builds the editor starts, with the engine's import program, which reads the formats of plugins - from the editor library just built. `mirai package engine` runs it for the plugins. A plugin - with assets is a root of its own. -- What costs comes from the [texture cache](#texture-cache) and the [mesh cache](#mesh-cache), - which the worktrees of a repository share: a new worktree reads its sources and writes its - imported folder without compressing or simplifying anything. +## Library + +The library of the machine is what its store keeps of imports and what a project does with it +(`cook/import/sources/library.odin`). A project names the packages it uses in its +`packages.mirai` and carries the text of their manifests beside it +([packages](../../mirai-assets/references/formats.md#packages)); the store, shared by every +checkout, worktree and project of the machine, keeps manifests and builds. + +- **`ingest(origin, options, settings)`** imports a file or a folder from wherever it is: in the + assets of a project or a plugin, or in no project. Nothing is copied: the package goes to the + store, named by `packages/` and by its origin (`origins/`, of this machine + alone), so ingesting the same origin again keeps its ids. `settings`, when given, replace those + the package keeps. A source whose identity is that of its package and whose build this engine + reads is not imported again: the report names its package and lists no assets. Two processes + or worktrees that ingest one origin take turns with a lock per origin (`cook/workspace`, under + `/locks/`), and the second finds the work done. +- **`project_add(root, manifest, folder, assets, options)`** pins a package in the project + (`project_pin`, which with `project_unpin` is the only writer of `packages.mirai` and of the + manifest files beside it) and follows it there; `project_ingest` ingests a source and pins its + package in one import. **Following** writes + or follows its documents under `folder` of the assets: from the baselines of its build + (`documents_follow`), and for a model whose source is here and whose game types the library + of the project's editor gives, by importing it with that folder, which also writes its prefab + (`prefab.model_prefab`). Without the source or those types a model's prefab cannot be written, + and the report says so. +- **`project_refresh(root, options)`** takes each package the project pins to a build this engine + reads. A manifest the store lacks comes from the root's file of it, else from the caller's source; + so a clone with its sources and an empty store ingests every package again with the same ids, + manifest and key, and one without them reports each package. The key (`asset.import_key` of the + identity and the versions of what reads its origin, an engine importer or a format of the engine + or of the plugins) is looked up in `imports` (and `imports-fast` with `fast_textures`), then asked + of the caller's source (`cache.Source`, the same procedure value the caches of `cook/cache` take: + given `(imports, )` or `(imports-fast, )` it makes each build and every object it + lists whole, given `(packages, )` the manifests; it takes many names at once, and a refresh + asks every key and id it will need in one call before it goes package by package, `imports-fast` + only with `fast_textures` and for the keys the source gave no Final of), then built by ingesting + the origin when it exists on this machine. Otherwise the report says, by package name, that a + machine with its source must ingest it with this engine; when only a Fast build is here and Final + is required, it says that. It then names the build under `builds/` and, when that + changed, follows the documents. A source here whose identity changed (its bytes, an input, the + code of the plugin that imports it whole) is ingested again with the ids its manifest gives, and + the project pins the new manifest. A document changed since it was last written is left as it is, + with a warning that names it ([changed documents](#changed-documents)): the rest is written. +- **It never removes** an object, a ref or a file of a project because a source is missing, and + never builds without the source. What the store does not have stays listed as not built. +- A source a plugin imports whole has a key only its importer knows: the refresh takes the build + this machine chose for its manifest, and a machine without its source reports it. +- **Source hashes.** The address of each file an identity reads is kept per origin by size and + modification time (`importer.Source_Hashes`, `sources/` in the store), so a + refresh reads only the files that changed. A USD is composed by `usdcat`, which does not say + which layers it read: after changing one alone, ingest the stage. +- **Progress.** `Library_Options.progress` gets a line per package and per source imported, and + the assets written every `importer.PROGRESS_EVERY`. +- **Commands.** `mirai import ` ingests and adds to the project of `--project`, else the + one whose assets hold the file, under `--folder` (else the folder of the file); `mirai import + --refresh ` refreshes a project or a plugin. Both build textures Fast and take a Fast + build, as the editor does; `--final` requires Final. `--from ` + naming a second store as its source (`cache.store_source`, which copies a ref with its objects, + as a peer gives them); an installed engine takes its `builds/` store as that source. Every + build of a project refreshes it and its plugins with a `packages.mirai` once its target stands + (`build.job_imports`, `imports_refresh`) with the engine's import program, which reads the + formats of plugins from the editor library just built. ## In parallel @@ -379,40 +398,46 @@ single thread and in the usual order: with or without a job system the same byte when it ends. Both sum with `importer.profile_totals`: each zone under the one that contained it, with its seconds on the importing thread, which add up to its wall time, and what no zone covers. -- The imports of a sync share `Import_Options.registry`, which finds documents by their ids: one +- The imports of a refresh share `Import_Options.registry`, which finds documents by their ids: one scan of the root for all of them. ### Texture cache What matters about a built texture does not depend on its id or its path: `cook/textures/cache.odin` -stores it by content in `.mirai/cache/textures` of the main checkout of the repository the project -is in (`project.content_cache_path(root, project.TEXTURE_CACHE)`; of the project itself outside a -repository), one `.tex` per key whose id is the key. Every project and worktree of the repository -shares it. - -- `Import_Options.texture_cache` is that folder (`""`, none); the editor, cooking and - `mirai import` of a map pass it, taking it from its project. +keeps it by content in the cache `textures` of the machine store (`cook/cache`), one texture per +key whose id is the key. Every project, worktree and program of the machine shares it, and a +machine that shares the store's objects with another finds what that one built. + +- `cook/cache` keeps each result as an object of `mirai:store` (`asset.object_put`) named by the ref + (`cache-`, `-`) (`asset.cache_namespace`). The store reaches it as a value + (`Import_Options.store`, `Cook_Desc.store`; nil, none): the editor's + imports (the session's: `--store`, else the machine's), `mirai import` (the machine's store, or + `--store`) and the cook of a build (that of the build: `--store`, else the machine's). Tests + make one in a temporary folder. +- A miss asks `Import_Options.cache_source` (a `cache.Source`) once before building: it makes the + object whole in the store and sets the ref, or says it has none. What a pass needs is looked up + at once (`cache.lookup`): the cook asks the source in one fetch for the textures of its pass and + its jobs then read the cache without asking. - A texture that is not kept looks for its `textures.cache_key_build` (the bytes of the image, or the texels the format made with their size, layers and format, and the `settings_hash`): if it is there, its levels are written with the id and the `Import_Info` of this import; if not, it is built and stored. The same image from another source, in another folder or with the settings of before is not compressed again. `rebuild` does not read the cache and rewrites it. -- With `Import_Options.fast_textures` (the editor's imports and `mirai import`) a texture of the - cache takes the Final the texture cache keeps for its key; without one it is built Fast under the - key of Fast settings, and a texture kept Fast is written again as soon as the cache holds its - Final, or by the [sync](#sync) of a build that cooks. `Import_Info.settings_hash` says which of - the two the file holds. +- With `Import_Options.fast_textures` (the editor's imports and `mirai import`) a + texture of the cache takes the Final + the texture cache keeps for its key; without one it is built Fast under the key of Fast settings, + and a texture kept Fast is written again as soon as the cache holds its Final, or by the + [refresh](#library) of a build that cooks. `Import_Info.settings_hash` says which of the two + the file holds. - [Cooking](../../mirai-projects/references/build.md#asset-cooking) stores there the blocks of an RGBA8 texture by `textures.cache_key_blocks`. -- A key names what made the blocks (`textures.encoders_id`), so checkouts with other encoders - share the folder without sharing entries. -- `cache_read` checks hash and key: a damaged file or one of another `TEXTURE_VERSION` is a miss and - is built again. Reading an entry marks it used. -- An entry belongs to no asset (an old one serves again when switching branch or setting): an - entry no build used for `cache.UNUSED_AGE` is deleted (`cache.trim`, at most once per - `cache.TRIM_EVERY` after a write), never one by size, so a project larger than any budget keeps - what its next import reads. Deleting the folder only costs time. The mechanism (file per key, - key id and version, read that marks use, age) is `cook/cache`, the same as the mesh cache. +- A key names what made the blocks (`textures.encoders_id`), so machines with other encoders + share the store without sharing results. +- A read checks the id (key and version) of the asset and reads its chunks: a ref whose object is + not whole here, does not read or is of another version is a miss and is built again. The store + checks its chunks (its scrub), not the cache. +- A result belongs to no asset and `cook/cache` removes none: the store owns space, and what goes + is decided by the live set given to `store.sweep`. - An image only its format decodes (`Imported_Image.make`, added with `importer.image_made`) is keyed by its `data`, what the format makes the texels of: a texture the cache keeps for it is written without decoding it, and the format raises a version inside its data when the same data @@ -420,10 +445,10 @@ shares it. ### Mesh cache -A built mesh is worth by its geometry as read and its settings, not by its id: `mesh_cook` stores -it in `.mirai/cache/meshes` (`project.content_cache_path(root, project.MESH_CACHE)`, shared like -the texture cache; `Import_Options.mesh_cache`, which the editor, sync and `mirai import` of a map -pass), one `.mesh` per key with its levels, clusters and colliders, without materials. +A built mesh is worth by its geometry as read and its settings, not by its id: `mesh_cook` keeps +it in the cache `meshes` of the machine store (`importer.MESH_CACHE`, the same `cook/cache` and +`Import_Options.store` as the texture cache), one mesh per key with its levels, clusters and +colliders, without materials. - The key (`mesh_cache_key`) is the content of the mesh as the reader gives it (vertices, indices, skin, submeshes, morph targets and flags: `mesh_content_hash`) with `mesh_build_hash`; the @@ -433,13 +458,12 @@ pass), one `.mesh` per key with its levels, clusters and colliders, without mate - A mesh that is not kept looks for its key; if it is there, it is written with the id, the slots, the skeleton and the `Import_Info` of this import, the same bytes as building it. `rebuild` does not read the cache and rewrites it. -- It saves simplifying, the cluster DAG and the pages; reading the source, no. Its entries go by - age, as those of the texture cache. +- It saves simplifying, the cluster DAG and the pages; reading the source, no. ## Settings of a source `Import_Settings` is how a source becomes assets; its zero value is the default of each setting and -its fields are the options of `mirai import` (`IMPORT_USAGE`). The sidecar writes only the settings +its fields are the options of `mirai import` (`IMPORT_USAGE`). The package keeps only the settings that differ from their default, so it shows what was chosen. What is not obvious: - `image` says how a loose image is read; `Normal_Green_Down` is a normal map with green pointing @@ -461,16 +485,16 @@ that differ from their default, so it shows what was chosen. What is not obvious materials, skeletons, meshes or prefab. It is how the motion of another file (a set of clips exported with a body) joins a character imported before. In the editor it is `skeleton` of `asset.import`, the way to give it to a [plugin format](#plugin-formats). The reader sees it in - `r.options.settings` (the settings the import applies, those of `.import` when it has - one): a format whose clips keep no skeleton of their own (Cast) reads that `.skeleton` to bring - its keys to the frames of its bones, and adds its bones as a skeleton of the model, which is - not written. + `r.options.settings` (the settings the import applies, those of its package when it was + ingested before): a format whose clips keep no skeleton of their own (Cast) reads that + `.skeleton` to bring its keys to the frames of its bones, and adds its bones as a skeleton of + the model, which is not written. -Settings live in the [sidecar](#identity-and-reimport) of the source, one entry per field name (an +Settings live in the [package](#identity-and-reimport) of the source, one entry per field name (an absent field is its default), and win over the options except with `ignore_stored`: importing a -source again gives the same assets whoever does it. An unreadable file (unknown field or value, -negative number, coverage outside 0 to 1) is `Malformed`, never the defaults. Every import writes -its sidecar when it ends, and does not touch it if it already says the same. +source again gives the same assets whoever does it. Unreadable settings (unknown field or value, +negative number, coverage outside 0 to 1) are `Malformed`, never the defaults. Every import puts +its manifest in the store when it ends. ## The imported model @@ -483,13 +507,14 @@ those a mesh uses; repeated names are told apart with `name (n)`. ## What is written -What is new gets a flat file in the folder of the import (`.mat`, `.skeleton`, `.anim` and, -with `import_source`, the model's `.scene`), with a valid file name that does not overwrite -another asset; what is of the cache is a file of the imported folder named by its id. What already -existed is written where it is ([identity](#identity-and-reimport)). +Every output is an object of the build. In the folder of the documents, a new document gets a flat +file (`.mat`, `.skeleton`, `.anim`, a plugin's kind with the extension its manifest declares +and, with `import_source`, the model's `.scene`), with a +valid file name that does not overwrite another asset; one that already existed is followed where +it is ([identity](#identity-and-reimport)). Each imported asset carries the `Cooked` flag and its `Import_Info` ([cache](#import-cache)). -- **Ids and owner.** Random the first time and those of the sidecar afterwards; the +- **Ids and owner.** Random the first time and those of the package afterwards; the [parts](#parts) name their prefab as `owner`. Dependencies come from the `Asset_Id`s that each asset names. - **Compression.** Texels go in BC blocks ([textures](#textures-cooktextures)); the bulk columns of @@ -500,11 +525,13 @@ Each imported asset carries the `Cooked` flag and its `Import_Info` ([cache](#im previous import it rewrites `imported` and follows the source except in the fields that differ from what it gave before (`asset.material_follow`, the algorithm of prefab overrides); the parameters of a program stay the same by name, and `overlays` and `hidden` are kept: what was - changed after importing stays and a new texture of the source arrives. The skeleton likewise - keeps the body parts set by hand. + changed after importing stays and a new texture of the source arrives. A skeleton keeps the body + parts set by hand and the sockets added on bones it still has, and a clip the root set by hand + while its bone is in its skeleton. A document keeps the `owner` its file has. The follow starts + from the baseline: the grips of a hand are those of the body found on the bones. - **Images that cannot be used** leave a warning and the map without a texture; a loose image that - does not decode fails. If `import_write` fails, it deletes what it created in that call; what - already existed is replaced one at a time with atomic writes. + does not decode fails. If `import_write` fails, it deletes the documents it created in that call + and names nothing; what already existed is replaced one at a time with atomic writes. - `Written.detail` says what each asset has (triangles per level and colliders of a mesh; format, size and mips of a texture; entities of a prefab), also for a kept one. @@ -514,11 +541,11 @@ file is in the assets folder of a project or a plugin; it imports where it is. - It first builds the engine if it is not up to date, in dev unless `--release` is given; what that build says goes to stderr and stdout is left with the report. -- It starts from the settings of the sidecar, applies the given options (`IMPORT_USAGE`) and writes - the sidecar again: repeating the command without options gives the same assets. A source a format +- It starts from the settings of the package of the source, applies the given options + (`IMPORT_USAGE`): repeating the command without options gives the same assets. A source a format of a plugin reads takes the formats of the editor library of its project. - The report gives one line per asset (`id kind bytes path (detail)`), how many were already up to - date, the sidecar and the warnings. Exit codes: 0, 1 (failed) and 2 (usage). + date and the warnings. Exit codes: 0, 1 (failed) and 2 (usage). ## Template rigs (`cook/rig`) diff --git a/.agents/skills/mirai-networking/SKILL.md b/.agents/skills/mirai-networking/SKILL.md index b7a66d232..f49383a0b 100644 --- a/.agents/skills/mirai-networking/SKILL.md +++ b/.agents/skills/mirai-networking/SKILL.md @@ -41,6 +41,8 @@ neither the data world nor physics. A replicated object is an opaque game id plu [clock and inputs](references/networking.md#time-and-inputs), [interpolation](references/networking.md#smoothing-and-interpolation-interpolationodin) and [known limits](references/networking.md#known-limits) +- Blobs and refs of the store between peers on a host's connections → + [exchange](../mirai-store/references/exchange.md#what-the-owner-calls) - Opening a host's UDP port (`mapping_start`, `mapping_poll`, `mapping_stop`, `mapping_close`; PCP, NAT-PMP, UPnP) → [API and lifecycle](references/port-mapping.md#api-and-lifecycle), [protocols](references/port-mapping.md#protocols) and @@ -141,21 +143,44 @@ In `plugins/multiplayer/runtime`: - After `host_init`, every capacity comes from `Host_Desc`/`Replication_Desc`; exhausting one returns `Error` or counts in the statistics. Replicated traffic, even damaged, does not allocate. -- **Every connection is sealed, and nothing answers an unproven sender:** before a returned cookie - only the challenge, and nothing to a proof that does not open. There is no unsealed mode. +- **Every connection is sealed, and its first datagram carries data:** the handshake is the Noise + pattern `NKpsk0` over the key and the secret of the invitation, and the Initiate carries the + client's datagram 0. An address hears only the Response, one per copy of its Initiate (and the + Disconnect if the game closes it), until a datagram under the keys of the connection comes from + it; an Initiate that does not open, or a Hello to a host with an invitation, hears nothing. There + is no unsealed mode. +- **An Initiate opens once** while the key of its server lasts, also one refused: a later copy, + from anywhere, makes no connection and delivers nothing again. What the replay filter cannot take + or holds by mistake, and what comes while the server ran out of key exchanges, gets a Retry: a + full filter or a flood costs a round trip, never a connection. - **A connection changes address only on a datagram that opens under its key** and is newer than - any it accepted ([roaming](references/networking.md#connections)). A wrong proof costs its + any it accepted ([roaming](references/networking.md#connections)). A wrong Initiate costs its address, never everyone: guessers do not shut out who brings the invitation. +- **The key of an invitation belongs to its share:** made when it starts, kept in memory by its + owner and its `Host`, gone when it ends; its private key is never written down. - **Two ways in, nothing else:** by address (a server without an invitation admits anyone who - reaches it) and by invitation code (only its holders). The code carries the host's real - addresses, on its network and on its router; a client asks only those, never guesses others. + reaches it, after a Hello and its Key) and by invitation code (only its holders). The code carries + the host's key and secret and its real addresses, on its network and on its router; a client asks + only those, never guesses others. - **Every byte from the network is hostile.** It is read with `Bit_Reader`, which never leaves the buffer; every index, type, mask and length is validated before use. A datagram that cannot be interpreted closes the connection with `Protocol_Error`; it is never half-applied over another connection. -- **Datagram acks and losses are final.** Late datagrams are not acked and only allow - [the partial delivery described in connections](references/networking.md#connections). Do not - extend that exception without reviewing channels, links and baselines. +- **A connection finds its rate by itself.** Nothing in `Host_Desc` starts or caps it; a reliable + channel holds in flight what its `queue_bytes` holds, and the game channels of a connection stay + on time next to a stream on it. +- **Nothing waits a fixed tick:** the worker waits in the transport, and a datagram that arrives or + a frame the simulation queues wakes it. Datagrams go and come in batches, one call where the + system has one. +- **A reliable byte is copied into its queue once and stays there until its ack:** a chunk is a + range of the queue, as long as the datagram has room for, and a lost one fits in the next + datagram (`chunk_longest`). +- **A datagram is lost only when reordering cannot explain its absence**, and an ack is final. A + late datagram is acked only if everything in it that a resend would bring was taken + ([connections](references/networking.md#connections)); objects, the server time and the library + channels are never taken late. Do not extend what is taken late without reviewing channels, + links and baselines. A loss taken back by a late ack does not count for the rate; what was + resent for it stays resent. - **Nothing changes until it is known that the entry fits.** `entry_write` writes, checks and only then modifies the link; if it does not fit, it restores the writer. - **A baseline is used only after its ack**, and the receiver stores it from the state that results @@ -251,7 +276,9 @@ In `plugins/multiplayer/runtime`: - `mirai test net [test...]`, `mirai test std/serial` (bitstream) and `mirai test plugins/multiplayer/runtime`. Run the `*_test.odin` of the touched area: `address`, - `connection`, `secure`, `invite`, `congestion`, `flow` (reliable window, a streamed part), + `connection`, `secure`, `invite`, `congestion`, `flow` (reliable window, a streamed part, a bulk + stream on a 100 Mbit/s link, a shared link, ack runs), `chunk` (hostile chunks, a small message + beside the batches and the worker's wait), `conditioner`, `replication`, `descriptor_boundary`, `large_state`, `collection`, `references`, `convergence`, `grid`, `scope`, `authority`, `interpolation`, `clock`, `worker`, `baseline` (pools and shared baselines), @@ -264,7 +291,10 @@ In `plugins/multiplayer/runtime`: - `multiplayer`: [multiplayer verification](references/multiplayer.md#verification). - They are deterministic (`Sim_Network` with a fixed seed), except those that open UDP sockets on `127.0.0.1`. -- Optional benchmarks: the server with 64 clients, +- Optional benchmarks: a reliable stream on UDP loopback (rate, CPU per side, datagrams and + transport calls per GiB), + `mirai test net a_reliable_stream_on_udp_loopback -define:NET_BENCHMARK=true`; the server with 64 + clients, `mirai test net server_with_64_clients_and_2000_objects --release -define:NET_BENCHMARK=true`, and the frames of a client with interpolated replicas, `mirai test plugins/multiplayer/runtime bench_client_frames -define:NET_CLIENT_BENCH=true`. diff --git a/.agents/skills/mirai-networking/references/multiplayer.md b/.agents/skills/mirai-networking/references/multiplayer.md index b00e33bc1..15a9cba8d 100644 --- a/.agents/skills/mirai-networking/references/multiplayer.md +++ b/.agents/skills/mirai-networking/references/multiplayer.md @@ -280,7 +280,7 @@ A third library declares its operation bytes, applies them and writes its parts: `host.launch_flag` and `host.launch_value` ([plugin arguments](../../mirai-core/references/host.md#plugin-arguments)): `--server [port]` (`Net_Settings.port` without a port), `--dedicated` (serves like `--server`, in a player without - the phases of whoever watches), `--invite [token]`, `--connect `, + the phases of whoever watches), `--invite`, `--connect `, `--new-player`, `--until-empty`, `--status `, `--latency`, `--jitter` and `--loss`. A value that does not work, or `--server` with `--connect`, says why and ends the player with `host.PLAYER_USAGE`. Without @@ -321,8 +321,10 @@ reason, and with `--until-empty` a server that emptied ends normally. - **Two ways in.** By address: `--server` serves, anyone who reaches its port joins with `--connect
`. By invitation: `--server --invite` serves only who brings its code, which it prints and `invite_code(session)` gives once the router answered; it carries the host's - address on its network and on its router, so the same `--connect ` joins from both. - `--invite ` fixes the token and maps nothing: the code carries only the LAN address. + address on its network and on its router, so the same `--connect ` joins from both. The + key of the invitation lasts while the server serves; `invite_give` hands the session one made by + whoever shows the code, never through a storage of the World, and then the session maps nothing. + The session zeroes it when it ends. - A server saves and loads its world with `save`; a client does not (`Unavailable`). Loading with clients is a [scene change](#scene-change). @@ -407,13 +409,13 @@ this plugin the editor shows nothing of networking. shares its scene as host, Play serves the game from its own world; without online it serves only guests. - Serving: the scene is written to `.mirai/play/play.scene` and the world of Play opens a server - (`Session_Request.open`) on the first free UDP port from `Net_Settings.port`, with a new token; - its local players play in it and, as soon as the server listens, one `mirai-player` is opened per - online player (`--connect` through `127.0.0.1`, `--players 1` and `--latency` equal to half the - ping) over the editor's library, without compiling anything - ([player processes](../../mirai-editor/references/editor.md#networked-play)). Server and clients - run the same library this way; the protocol hash still rejects a client of another build, like an - external player or a guest with other code. + (`Session_Request.open`) on the first free UDP port from `Net_Settings.port`, with a new + invitation key; its local players play in it and, as soon as the server listens, one + `mirai-player` is opened per online player (`--connect` through `127.0.0.1`, `--players 1` and + `--latency` equal to half the ping) over the editor's library, without compiling anything ([player + processes](../../mirai-editor/references/editor.md#networked-play)). Server and clients run the + same library this way; the protocol hash still rejects a client of another build, like an external + player or a guest with other code. - `play.network.status` reports the state of `Net_Play`, with `invite`, the code to enter from an external player. Stop ends Play, the session and the online players; a paused Play keeps its connections. diff --git a/.agents/skills/mirai-networking/references/networking.md b/.agents/skills/mirai-networking/references/networking.md index 2bfe94cec..71e075377 100644 --- a/.agents/skills/mirai-networking/references/networking.md +++ b/.agents/skills/mirai-networking/references/networking.md @@ -14,13 +14,22 @@ Descriptors, defaults and statistics (`Host_Desc`, `Replication_Desc`, `Object_D - Values: single bits, bool, ranged integers, varuint and zigzag varint, raw floats or floats by range and precision, octahedral unit vectors, smallest-three quaternions, and bytes and strings with a length. `quantize_*`/`dequantize_*` do the conversions without a stream. +- Bytes on a byte boundary (`bit_write_align`, `bit_read_align`) are copied whole, and + `bit_read_view` returns them where they lie without a copy. ## Transport -`Transport` is two non-blocking procedures, `send` and `receive`; `PACKET_BYTES` is the largest -datagram. - -- **`Udp`**: a non-blocking UDP socket from `core:net` (port 0: the system chooses). Bound to +`Transport` moves datagrams a batch at a time: `send` and `receive` never block, and `wait` blocks +until a datagram arrives, `wake` is called from any thread, or a time passes. A worker needs `wait` +and `wake`; a host the caller updates uses neither. `PACKET_BYTES` is the largest datagram. A host +builds its datagrams in a batch of slots and hands them over when the batch fills and at the end of +the update, and asks for as many as its command buffer has room for. + +- **`Udp`**: a non-blocking UDP socket from `core:net` (port 0: the system chooses). A batch is one + call where the system has one (`sendmsg_x`/`recvmsg_x` on macOS, `sendmmsg`/`recvmmsg` on Linux) + and a call per datagram on Windows, behind the same signatures (`udp_send_batch`, + `udp_receive_batch`). `wait` polls the socket and a second one on `127.0.0.1` that `wake` sends a + byte to. Bound to `address_any6` it is dual-stack: it turns off `IPV6_V6ONLY` before the bind (`udp_dual_stack`, one signature per system) and receives both IPv6 and IPv4 peers. An IPv4 peer arrives as `::ffff:a.b.c.d` and `address_from_endpoint` returns it as IPv4; sending to it from that socket @@ -52,7 +61,10 @@ like `replication.hysteresis`) and the values it admits. and updates statistics, replicas, the input buffer and the clock. - **Network half**: `host_update(h,now)` in the calling thread, or `worker_start(h)` in its own thread. It applies the frames, receives, evaluates scope, packs and sends. It never reads game - memory: the frame carries copies of the quantized state. + memory: the frame carries copies of the quantized state. Between updates the worker waits in the + transport until a datagram arrives, the simulation half queues a frame or empties the command + buffer it filled (both wake it), or something is due: at most a millisecond while a connection + has anything in flight, waiting or unacknowledged, else its keepalive or handshake. - The two halves only touch when swapping buffers under a mutex, with preallocated frames and command buffers. If the network falls behind, the frame keeps filling and goes out with the next publication (`publish_stalls`); if the command buffer has no reserve for what one datagram can @@ -62,90 +74,153 @@ like `replication.hysteresis`) and the values it admits. ## Connections -- **Four-datagram handshake, always sealed.** The client's request and response are padded. - - *Request*: the client salt, to every address `host_connect` got at once: one, or the two of - an invitation. The client proves itself to each that answers and goes on with the one that - accepts; answers from any other address are ignored. A denial or another version ends the - attempt once every candidate gave one: the other may be the host while this one is another - machine at a private address of the client's own network. - - *Challenge*: a SipHash cookie of the address, the salt and the - time, the server's version and its lasting X25519 key; every request gets it and the server - stores nothing. - - *Response*: the cookie, the protocol hash, the client's key for the attempt and its proof, - sealed with a key from the exchange of that key with the server's lasting one and the - invitation. - - *Accepted*: the server's key for the connection and, sealed with the session keys, its salt, - object capacity, history and the slot of the connection. - - Nothing but the challenge answers an address that has not returned a valid cookie, and - nothing at all a proof that does not open (a wrong invitation): both count in - `handshakes_dropped`. Denials (`Denied_Protocol`, `Denied_Full`) go only to a returned cookie, - and `Denied_Full` only to a proof that opened; they count in `handshakes_refused`. The idea - is netcode's: answer only when necessary, and deny only a request that passed every check - ([standard](https://raw.githubusercontent.com/mas-bandwidth/netcode/main/STANDARD.md)). - - A wrong proof costs its address (the IP without the port, an IPv6 /64): a few back to back, - then one per interval; what comes sooner is put off before the key exchange +- **Two datagrams to connect, and the first carries data** (`net/secure.odin`): the handshake is + the Noise pattern `NKpsk0` (`Noise_NKpsk0_25519_ChaChaPoly_SHA256`). The client knows the + server's key `s` before it writes anything, and the pre-shared key is the invitation's secret. + - *Initiate* (`-> psk, e, es`): its header (salt, version, protocol hash and the cookie of a Key) + is the prologue, and its payload is the client's datagram 0 of the connection: what it queued + with `message_send` to connection zero after `host_connect`. The game of the server has it half + a round trip after the client connected. The same bytes go to every address `host_connect` got, + each interval, until a Response opens; a denial or another version ends the attempt once every + candidate gave one: the other may be the host while this one is another machine at a private + address of the client's own network. + - *Response* (`<- e, ee`): sealed, the object capacity, the history and the slot of the + connection. Both ends then split the keys of the connection. + - What each end proves: the client, that it holds the invitation (an Initiate opens only for the + holder of `s` and only if its sender knew `s` and the secret), and nothing of who it is; until + a datagram under the keys of the connection comes from its address, not even that it receives + there. The server, that it holds `s` and the secret. + - Without an invitation the client first sends a padded *Hello* to every candidate, and the + first server that answers with its *Key* (`s`, its version and a SipHash cookie of the + address, the salt and the time, keeping nothing) gets the Initiate, which returns the cookie + and mixes a zero secret. That is one round trip before data, and the same handshake: the client + takes `s` from the Key, so only an active attacker on the path could pose as the server. +- **The MAC of the invitation.** An Initiate ends in an HMAC-SHA256 under a key derived from `s` + and the secret: a host with an invitation answers nothing, not even a denial, to one without it, + and it costs no key exchange. Without an invitation `s` comes in the Key and the MAC filters only + strays. +- **Replays and early data.** The server keeps the key `e` of every Initiate it opened in Bloom + filters, whether it then accepts or refuses it: those without a cookie in one that lasts as long + as `s`, those with one in two generations that turn every `COOKIE_SECONDS`, as long as a cookie + is good. A copy that comes later, from any address, while its connection lives or after it ended, + opens nothing: no second connection, no early data delivered again, no key exchange. A copy from + the address of a connection still waiting for its client gets the stored Response again. So the + early data is applied once and may carry anything the client queues; what needs the server's + answer is sent after `Connected`. The server's own traffic waits for the client: until a datagram + under the keys comes from its address, that address hears only the Response, one for each copy + of the Initiate, and the Disconnect of a game that closes the connection. The game of the server + hears `Connected` and the early data at once; a connection whose client never answers ends with + `Connect_Timeout` after `connect_timeout`, without a word to its address. +- **Retry.** Each filter takes at most `SEEN_LIMIT`, which keeps a false positive under one in + 10^4. An Initiate the filter holds, one without a cookie when that filter is full, and, for + `BUSY_SECONDS` after an update ran out of key exchanges, any without a cookie, gets a Retry: a + cookie of its address, smaller than the Initiate. The client seals the same datagram 0 again with + the cookie and a new `e`. Spoofed senders never return the cookie, so the exchanges go to those + who do; a filled filter or a flood costs a client one round trip, never its connection, and an + unloaded server keeps none. A generation that is full puts off what comes until it turns. +- **Amplification and work.** The Hello and the Initiate are padded to `HANDSHAKE_BYTES`, more than + any answer to them: an unconfirmed address never hears more than it sent. A server answers no + Hello when it has an invitation. An Initiate costs at most one X25519 exchange before it opens, + and two more when it does, `handshakes_per_update` at a time per update. Denials go to an + Initiate with the MAC: `Denied_Version` read from the header, `Denied_Protocol` and `Denied_Full` + after it opened; they count in `handshakes_refused`, and what gets no answer in + `handshakes_dropped`. + - A wrong Initiate with the MAC costs its address (the IP without the port, an IPv6 /64): a few + back to back, then one per interval; what comes sooner is put off before the key exchange (`handshakes_deferred`). The allowance is a token bucket kept as one time per address in a fixed table of sets; a full set gives its slot that frees soonest to the newcomer, so a crowd - of guessers never shuts out who brings the invitation, which joins with no added round trip. -- **Format.** Every datagram starts with a type byte. The handshake ends in an XXH3 checksum keyed - with `protocol_id`, which only filters strays; data ends in the ChaCha20-Poly1305 tag, with the - type, sequence and the slot of the connection at the receiver as associated data. The protocol - hash includes fragments, events, input, channels and `TRANSPORT_VERSION`, which rises when the - format of the handshake or of data datagrams changes. + of guessers never shuts out who brings the invitation. +- **Format.** Every datagram starts with a type byte. A handshake datagram ends in an XXH3 checksum + keyed with `protocol_id`, which only filters strays; data ends in the AEGIS-128L tag, with the + type, sequence and the slot of the connection at the receiver as associated data. AEGIS runs on + the AES instructions where the processor has them and on a portable implementation, about ten + times slower, where it does not. The protocol hash includes fragments, events, input, channels and + `TRANSPORT_VERSION`, which rises when the format of the handshake or of data datagrams changes. - **Roaming.** A data datagram from an address no connection has goes to the connection of the slot it names; if it opens under that key and is newer than any the connection accepted, the connection moves to that address, as WireGuard roams: a router that changes the port, or a switch of network, loses nothing and costs no round trip. A forged datagram does not open, and one replayed from elsewhere is not newer, so neither moves a connection. -- **Sealing** (`net/secure.odin`, only with `core:crypto`; there is no unsealed mode): the session - keys come from HKDF-SHA256 over the exchange of the two keys made for the connection and that of - the client's key with the server's lasting one, salted with the key of the invitation. The - client takes the server's key from the challenge, so only an active attacker on the path could - pose as the server, and only knowing the invitation. One key per direction, with the 64-bit - sequence as nonce, which every `REKEY_INTERVAL` datagrams is derived from the previous one; the - receiver's only advances when a datagram with the new one turns out authentic. A tampered, - truncated, repeated or older-than-the-last-accepted datagram is discarded. A system without - secure random bytes cannot make a host (`host_init` refuses `invite`) nor an invitation - (`invite_token` is not ok). -- **Two ways in.** By address: a server whose `Host_Desc.invite` is zero admits anyone who reaches - it, and the client calls `host_connect` with that address. By invitation: `Host_Desc.invite` is - in every key, so both ends must have the same one, and a client without it times out with - `Connect_Timeout`. It is a 64-bit secret so that whoever received the code gets in, not a - credential: nobody can try invitations from the server's answers, and neither the guesses the - server lets through nor whoever poses as the server on the path, trying them against the - client's proof offline, get near it. +- **Sealing** (only with `core:crypto`; there is no unsealed mode): Split gives one key per + direction, and AEGIS-128L seals every datagram after the handshake with the first half of it and + the 64-bit sequence as nonce. Every `REKEY_INTERVAL` datagrams the key is derived from the + previous one; the receiver's only advances when a datagram with the new one turns out authentic. + The client's datagram 0 is its Initiate, sealed by the handshake, so its first under AEGIS is 1. + A tampered, truncated, repeated or older-than-the-last-accepted datagram is discarded. A system + without secure random bytes cannot make a host (`host_init` refuses `security`) nor an + invitation (`invite_key` is not ok). +- **Secrecy.** The keys that follow the Response come from `ee` and die with the two ephemeral + keys: with `s` and the secret a recorded connection still does not open. The early data depends + on `s` as well, and `s` belongs to a share: `invite_key` makes it when the share starts, its + owner and its `Host` keep it in memory, and it is gone when the share ends; its private key is + never written down nor put on a command line. Nobody can try secrets against recorded traffic: + that takes `s`. +- **Two ways in.** By address: a server whose `Host_Desc.invite` is none makes its `s` in + `host_init` and gives it in its Key; anyone who reaches it joins with `host_connect` and that + address. By invitation: `Host_Desc.invite` is the share's key at the server and what the code + carries at the client; a host with one answers nothing else, so a client without it hears nothing + and times out with `Connect_Timeout`. - **Invitation codes** (`net/invite.odin`): the host says where it is, as ICE gathers a host and a reflexive candidate: `invite_make` takes its address on its network (`address_lan`) and, once - the router mapped its port, the mapped one; `invite_encode` writes both with the token and a - CRC-15 in Crockford base32, tolerant of case, separators and confusable characters, and the - client passes both addresses of `invite_decode` to `host_connect`. On the host's network the LAN - address answers; from outside, the mapped one; a router that does not loop its public address - back does not matter, and the client asks no other address. -- **Amplification.** The Challenge is smaller than the padded request, and the server does X25519 - exchanges (two per accepted client, one per refused proof) only for whoever returned the cookie, - `handshakes_per_update` at a time per update. -- **Acks.** Each data datagram carries its sequence, the last one received and a variable number of - ack bits: from what the other end might not have resolved when reading the last ack this end - received (`ack_known`), capped by `ACK_WINDOW`. Everything the sender has in flight arrives this - way with its real fate, also after a receiver stall. - - A datagram older than the last accepted is not acked: when a later one is acked, the fate of - the earlier ones is final and what was in a lost one is resent. - - If it arrives late (at most `LATE_WINDOW` behind) and had not been received, only what is safe - out of order is taken (`packets_late`): the chunks of the game's reliable channels, which - assemble only once, and on the server the inputs of ticks after the last received. - - Skipped are unreliable and sequenced messages, the server time and the objects, because what - arrived later is newer; and the library channels (reliable events and replication messages), - because they talk about objects whose state in that datagram is discarded and delivering them - earlier would change the order in which the game applies them. + the router mapped its port, the mapped one; `invite_encode` writes the version of the layout, + both addresses, `s` and the secret with a CRC-15 in Crockford base32 by groups of four, tolerant + of case, separators and confusable characters, and the client passes both addresses of + `invite_decode` to `host_connect`. `invite_decode` says why a text is no code (`Invite_Error`, + `invite_error_text`): a code of the earlier format, of another version, a character changed or two + swapped, or a key of small order. Everything stands at 128 bits but the secret, 64 bits that only + whoever learned `s` without the code could guess, online, at the rate each address is allowed. On + the host's network the LAN address answers; from outside, the mapped one; a router that does not + loop its public address back does not matter, and the client asks no other address. +- **Acks.** Each data datagram carries its sequence, the last one received and which of the ones + before it arrived, from what the other end might not have resolved when reading the last ack this + end received (`ack_known`), capped by `ACK_WINDOW`, as runs of arrived and missing datagrams in + Elias gamma code: a stream with few losses acks thousands of datagrams in a few bytes. Everything + the sender has in flight arrives this way with its real fate, also after a receiver stall. The + ack takes only what the clock or the inputs leave of the datagram: runs that do not fit are left + out, and the sender takes those datagrams as lost. It also names the datagram that arrived last + and how long ago, which the RTT is taken from: on a path that reorders, the newest one received + is the one that came fastest. + - A receiver acks within `ACK_DELAY`, or with what it sends anyway, and at once after + `ACK_EVERY` datagrams that carried reliable chunks or one that came late: a stream is acked as + it arrives, and a connection without one sends no datagram more for it. + - **Loss.** A datagram missing from an ack is lost only when reordering can no longer explain + it, as RACK and QUIC judge it: `REORDER_COUNT` datagrams sent after it were acked, and it has + been out longer than the RTT and a window from its variation (`loss_delay`). Its records are + resolved then; acked ones are resolved as they are acked, in any order. A lost datagram that + an ack later reports as arrived takes its loss back, so it does not count for the rate, widens + the window to what it needed (fading over seconds), and what was resent for it is a duplicate. + - A datagram older than the last accepted came late. If it had not been received and lies within + `ACK_WINDOW`, its ack is read and what is safe out of order is taken (`packets_late`): the + chunks of the game's reliable channels, which assemble only once, the messages of a game + channel that delivered none from a later datagram (each channel keeps its order), and on the + server the inputs of ticks after the last received. If that was all it carried it is acked, + and the receiver keeps it in its acks until the other end read one that has it. + - Skipped are the server time and the objects, because what arrived later is newer; and the + library channels (reliable events and replication messages), because they talk about objects + whose state in that datagram is discarded and delivering them earlier would change the order + in which the game applies them. A late datagram with something skipped is not acked: what was + in it is resent. - With nothing to say it is sent anyway to ack, keep the connection alive or probe a queue with no ack. - **Send limit**: a token bucket with the connection's rate that stores at most one tick of rate (`send_burst`), so a network half that updates more often than the tick spreads the datagrams. - The object budget of each send is the lesser of `budget_bytes` and the tokens: replication - follows the rate. -- **Congestion**: AIMD driven by the queue (the mean RTT of the interval above the lowest of the - recent window), not by loss, to cut back before the path buffer overflows; a link that loses at - random without a queue keeps its rate, which never drops below `min_send_rate`. + An update sends while the tokens last, up to `RECORD_WINDOW` datagrams in flight; past it, one + datagram a keepalive still goes, so a connection whose every datagram in flight was lost to an + outage learns their fate from the ack of a newer one and resumes. A reliable + channel with a datagram of chunks ready sends at every update the tokens allow; the rest goes once + a tick, or at the update that takes a message its owner queued, so an owner that publishes no + ticks (a transfer, the exchange) does not wait a tick for the end of each message. The object + budget of each send is the lesser of `budget_bytes` and the tokens: replication follows the rate. +- **Congestion**: the connection finds the rate of its path; nothing in `Host_Desc` sets it. It + starts at `CC_START_RATE` and doubles each interval that wanted more, until the queue appears + (the mean RTT of the interval above the lowest mean of an interval in the recent window, beyond + `CC_QUEUE_DELAY` and two datagrams at the rate; each mean counts with the uncertainty its spread + and its number of samples leave, so jitter does not read as a queue). Then it drops below what + the path delivered and grows by the same bytes per second for every connection (at most a + quarter each interval), so connections on one path converge to equal shares and the queue stays + short for the game traffic on it. Losses cut it only with a queue or past `CC_OVERRUN_SHARE`; a + link that loses at random keeps its rate, which never drops below `CC_MIN_RATE`. An interval + that delivered next to nothing, as after an outage, starts again from `CC_START_RATE`. - **Close**: `host_disconnect` tells the other end with its reason; silence closes with `Timeout`, and an authentic datagram that cannot be interpreted, with `Protocol_Error`. The reasons for rejection and closing are `Disconnect_Reason` (`host.odin`). @@ -153,26 +228,42 @@ like `replication.hysteresis`) and the values it admits. ## Channels Game channels in `Host_Desc.channels`, the same at both ends; `message_send` and the `Message` -command. `budget` limits the bytes of a channel per update. +command. `budget` limits the bytes of a channel per update. In each datagram the library's channels +go first, then the unreliable and sequenced game channels, then the reliable game channels one +chunk each in turn, starting with a different one each datagram: a channel that streams leaves the +others their share. - `Unreliable` is sent once; `Sequenced` too, and a message that is still waiting is replaced by the next one. -- `Reliable` is resent until its ack, delivered in order and cut into chunks as the window draws - them from the queue: the one fragmentation of the transport. `message_send` returns `Full` and - sends nothing if the message does not fit in the queue of some destination connection (it knows - from `Connection_Stats.queue_free` of the last `host_poll` minus what was queued since): the - sender retries after `host_poll`. With `sent`, a message to one connection streams: each call - queues what has room, and the channel takes nothing else until it is in, so a message has no - size limit. A message longer than `PIECE_BYTES` arrives in pieces (`Command.more`), which the - receiver joins; the library's own reliable channel takes only whole messages. +- `Reliable` is a stream of its messages, each its size and its bytes, cut into chunks as each + datagram has room: the one fragmentation of the transport. A datagram carries the next messages + whole or a part of one, up to `chunk_longest` bytes, which is what is left past the longest clock + or inputs and an acknowledgement kept to `ACK_KEPT` bits, all a datagram spares while a lost chunk + waits: so a resend always fits in the next datagram. A chunk says where its bytes lie in the + sender's queue; the queue keeps them until the ack, so they are not copied anywhere else, and the + receiver keeps a chunk that comes ahead of a gap at the same place of its own. `message_send` + returns `Full` and sends nothing if the message does not fit in the queue of some destination + connection (it knows from `Connection_Stats.queue_free` of the last `host_poll` minus what was + queued since): the sender retries after `host_poll`. With `sent`, a message to one connection + streams: each call queues what has room, and the channel takes nothing else until it is in, so a + message has no size limit. A message up to `PIECE_BYTES` arrives whole: straight from the datagram + when it lies in one, joined first when it spans several. A longer one arrives in pieces as its + datagrams bring them (`Command.more`), which the receiver joins; the library's own reliable + channel takes only whole messages, no larger than it allows. - **Window.** The chunks a reliable channel keeps in flight grow by what the path acknowledges (`channel_window`, each congestion interval), so they double each round trip while they hold the - channel back. Twice what the connection's rate sends in a round trip bounds them, - so congestion control and `send_rate` decide what a stream takes, and `Channel_Desc.window`, a - power of two the same at both ends, is the most a channel holds and receives ahead. A loss does - not shrink the window: its resend waits a round trip with the chunks after it acknowledged. + channel back. What the connection's rate sends in a round trip and the time a loss takes to be + found, in chunks of the size the channel's acknowledged ones had, bounds them, so congestion + control decides what a stream takes, and the channel's `queue_bytes` sets the most it holds in + flight and receives ahead (`channel_chunks`, a power of two up to `MAX_WINDOW`). A loss does not + shrink the window: its resend waits with the chunks after it acknowledged until the loss is + found, ahead of new ones. - While objects wait for the update, the reliable game channels leave half of each datagram to them: a part that streams does not starve the state. +- **Receiving.** A reliable channel delivers what follows what it delivered only while the command + buffer has room besides what a datagram may still bring. A gap that closes over many small + messages leaves the rest held, and the network half reads no datagram until it is delivered: what + arrives is never acked without room to keep it. - The library channels count their queue the same way: `event_send` of a `Reliable` event and the authority requests return `Full` and send nothing if they do not fit in the queue of some connection they may go to, and the sender retries. The connection is not closed; @@ -449,7 +540,8 @@ The client predicts with copies of the world and interpolates with `replica_blen - Congestion control does not tell a queue from a path that lengthens: a route change to a higher RTT cuts the rate until the window forgets the previous minimum. A buffer smaller than the queue - threshold overflows without the queue being seen, and then the rate does not drop. + threshold overflows without the queue being seen, and the rate drops only once the losses pass + `CC_OVERRUN_SHARE`. - A late datagram is not acked: even though its reliable chunks were already delivered, the sender resends them (the receiver discards them), and its objects, unreliable messages and time are lost. diff --git a/.agents/skills/mirai-plugins/references/plugins.md b/.agents/skills/mirai-plugins/references/plugins.md index 38aecbf0f..88fbfff2d 100644 --- a/.agents/skills/mirai-plugins/references/plugins.md +++ b/.agents/skills/mirai-plugins/references/plugins.md @@ -216,8 +216,8 @@ demo enables. How an `editor/` adds steps, panels and tools: which gives an image to each `core.Key`, to the mouse buttons and movement and to the faces, shoulders, triggers, sticks and their press, d-pad and its directions, Start and Back of each gamepad (the colored buttons on the faces), and beside it `kenney.prompts`, imported with - `mirai import plugins/prompts_kenney/assets/kenney.prompts.json`; its atlases are of the - imported folder of the plugin ([prompts](../../mirai-imports/references/imports.md#prompts)). + `mirai import plugins/prompts_kenney/assets/kenney.prompts.json`; its atlases are blobs of the + package the plugin pins ([prompts](../../mirai-imports/references/imports.md#prompts)). - **`playtest`** (`Editor`, development): `play.observe`, the monitors of `play.qa.*` with their repros and the bot of `play.explore`, over the ticks the API runs ([`Run_Provider`](../../mirai-editor/references/extensions.md#runs-the-api-drives)). The bot diff --git a/.agents/skills/mirai-projects/SKILL.md b/.agents/skills/mirai-projects/SKILL.md index 5a21b35bc..e3fae16ff 100644 --- a/.agents/skills/mirai-projects/SKILL.md +++ b/.agents/skills/mirai-projects/SKILL.md @@ -4,10 +4,11 @@ description: Mirai projects and plugins - `project` (manifests, plugin discovery `cook/build` (generated world, engine binaries, libraries and shipped game, project shaders, fingerprint, cooking and engine-time exclusion), `cook/workspace` (worktrees, dependencies, shared cache and locks), the installed engine (`mirai package engine`, binary package and release - workflow), the `tools/mirai` commands, the external scripts and the `templates`. Use when changing + workflow), sharing a project peer to peer (`cook/share`, `mirai share`, `mirai join`), the + `tools/mirai` commands, the external scripts and the `templates`. Use when changing a manifest, the layout or how a project or the engine is generated, compiled or distributed, when - investigating why something does not compile or what went into the game, or when taking reference - frames or sound from a YouTube or X video. + investigating why something does not compile or what went into the game, when sending a project + to another machine, or when taking reference frames or sound from a YouTube or X video. --- # Mirai projects and build @@ -38,6 +39,8 @@ description: Mirai projects and plugins - `project` (manifests, plugin discovery - Turns between processes, build, measurement and GPU locks and implicit Vulkan layers: [turns](references/build.md#turns-between-processes) and [layers](references/build.md#implicit-vulkan-layers). +- Sharing a project peer to peer (`cook/share`, `mirai share` and `mirai join`), its snapshot and + updating a folder joined before: [sharing](references/projects.md#sharing-a-project). - `mirai` launcher, `mirai worktree` and `mirai doctor`: [tool and machine](references/projects.md#tool-and-machine). - `mirai` subcommands (their usage, with `mirai ` without arguments): @@ -161,7 +164,11 @@ description: Mirai projects and plugins - `project` (manifests, plugin discovery lock, GPU turns by the room of the card, a person and a measurement) and `mirai test std/subprocess` (low priority of a child, handles it inherits). GPU turns with real editors: two sessions at once on a heavy scene both run or the second waits naming the first, and - killing the editor of one frees its turn (`mirai locks`). + killing the editor of one frees its turn (`mirai locks`). Sharing: `mirai test cook/share` + (snapshot rule, hostile trees, join, update by three ways with renames by case, edits while a + join runs, an incomplete join run again, resume, a third peer, the bytes of a re-join and of an + insert, the packages as the host's), all on `Sim_Network`; the memory of a large file is the + opt-in `a_large_file_joins_within_bounded_memory` with `-define:SHARE_LARGE=true`, run alone. - Generator or build: `mirai build editor` and `mirai build game --out ` of the affected project, and the player with `--no-gpu --frames 3`. - User commands, by hand: `mirai new`, and `mirai build game` twice (the second says "unchanged"). diff --git a/.agents/skills/mirai-projects/references/build.md b/.agents/skills/mirai-projects/references/build.md index 8c9709fbc..e71a44b24 100644 --- a/.agents/skills/mirai-projects/references/build.md +++ b/.agents/skills/mirai-projects/references/build.md @@ -275,7 +275,7 @@ SDL3 if the system does not provide it, and the reached in each scene, in its `.mirai/cache`, where the game goes on filling them. The copy replaces files and merges folders without deleting anything from the destination, except -`assets/`, which ends up exactly with what was cooked, without the sources and sidecars among the +`assets/`, which ends up exactly with what was cooked, without the sources among the assets. The code, `cook.mirai` and the `plugin.mirai` are not copied. `mirai play` copies nothing: `mirai-player` runs with `--project ` and reads assets and shaders from the project. @@ -291,29 +291,47 @@ Settings > Input changed, and SDL3, which the executable links. ## Imported sources -The meshes, textures, sounds and fonts an import builds are not versioned: they are in -`.mirai/imported/` of the project and of each plugin with assets, and are built again from the -sources and their sidecars ([sync](../../mirai-imports/references/imports.md#sync)). - -- **A build imports, and says so.** Every build of a project runs the sync of the project and of - its active plugins once its target stands (`job_imports`, `build.imports_sync`): the editor, - the game, the server, `mirai play` and the builds the editor starts. It uses the engine's import - program, since the tool builds without the importers, which reads the formats the plugins of - the project add from its editor library; the game, compiled from the engine's sources, - builds those binaries too when its project has assets. A source that does not import fails - the build naming it (`Import_Failed`). There is no other step: a checkout that was just cloned - runs after any build. Only a build that cooks (and `mirai package engine`) encodes textures - Final; the others build them Fast, as the editor imports them, and take Final where it is. -- **What ships.** A game has no imported folder and no sources: the cook writes every asset it - reaches at its path in the assets of the game, whichever of the two folders it came from. -- **The installed engine** carries the imported folder of each of its plugins, built when it is - packaged (`mirai package engine`): nothing is imported on the machine that installs it. +The meshes, textures, sounds and fonts an import builds are not versioned: they are objects of a +store that the package manifests a project or plugin pins name +([library](../../mirai-imports/references/imports.md#library)). + +- **The store is a value.** The machine's (`store.default_root`), or the one `--store ` + names on `build`, `play`, `play test`, `edit`, `generate`, `session`, `import`, `asset ls|deps` + and `package`: it reaches the build (`Desc.store_root`), the import program, the cook, the + editor and the player alike. +- **A build refreshes, and says so.** Every build of a project runs the refresh of the project and + of each active plugin with a `packages.mirai` once its target stands (`job_imports`, + `build.imports_refresh`): the editor, the game, the server, `mirai play` and the builds the + editor starts. It uses the engine's import program, since the tool builds without the + importers, which reads the formats the plugins of the project add from its editor library; the + game, compiled from the engine's sources, builds those binaries too when it has packages. A + build is taken from the store, then from a second store (`--from` of `mirai import`; an + installed engine's own), then ingested from its source when the source is on this machine; a + package none of them gives fails the build by its name, saying what gives it + (`Import_Failed`). There is no other step: a checkout that was just cloned runs after any build. + A build that cooks requires Final textures; the others take Fast, as the editor imports them. +- **Without sources.** A machine whose store holds the builds (fetched from a peer, or a copy of + another store) builds and ships with no source: it imports nothing. +- **What ships.** The cook writes what the registry resolves, documents of the project and the + objects of builds alike, each as a file in the container format at its path in the assets of + the game (an object gives the file `asset.object_file` gives), with the index: a shipped game + never opens a store. One lookup of the texture cache serves the whole pass. +- **The installed engine** carries `builds/` (`project.ENGINE_STORE`), a store with the Final + builds of the packages its plugins pin, made when it is packaged: a refresh on the machine that + installs it copies them into that machine's store (`cache.store_source`, as a peer gives them), + so nothing is imported there. ## Asset cooking `cook_assets` (`cook/build/cook.odin`) leaves in a folder the assets the game can reach. The `Game` job calls it with the project assets, `cook.mirai`, the asset folders of the active plugins and the -texture cache of the project; a failure is `Cook_Failed` with `Result.cook.error` and `detail`. +store of the build, which lists their packages (`asset.root_packages_in`) and whose texture cache it +reads and fills; a failure is `Cook_Failed` with `Result.cook.error` and `detail`. + +- **Packages.** The registry of the cook lists the outputs of the packages of the project and of + each plugin beside their files. A package the store does not list whole is `Package_Missing`, + by its name, and a texture still Fast is `Source_Failed`, naming its package: nothing is + cooked. - **Roots.** Every `.scene` (code reaches a prefab only by instantiating it) and `.records` of the folder and what `cook.mirai` (`COOK_FILE`) names: the other assets only code refers to (a @@ -330,7 +348,8 @@ texture cache of the project; a failure is `Cook_Failed` with `Result.cook.error or an asset id. A root that names nothing is `Root_Missing`; a malformed `cook.mirai` or one with another key, `Manifest_Invalid`; in both cases nothing is cooked. - **Plugins.** The folder of each plugin is mounted under `plugins/` and everything in it is - a root (its code uses it). + a root (its code uses it), except what a package of a root was imported from (its origin and + inputs inside the root), which stays behind. - **Parallel.** Files are cooked on a job system (`Cook_Desc.worker`, else one of the cook with a thread per core), the largest first; what is written and the report are those of a cook on one thread, and an error is the first by path. Prefab resolution and texture cache writes take a lock. @@ -344,10 +363,10 @@ texture cache of the project; a failure is `Cook_Failed` with `Result.cook.error `content_hash` of the `.tex` ([texture cache](../../mirai-imports/references/imports.md#texture-cache)). - **Fast textures.** A texture the editor imported Fast ships Final, whatever `"textures"` says: - the [sync](#imported-sources) a game build that cooks starts with builds it again from its source, - with the id it has, and fills the texture cache, so the editor takes that Final from then on; the - sync of an editor build keeps textures Fast, so a session never waits for RDO. A cook that - finds one still Fast fails with `Source_Failed` and says so. + the [refresh](#imported-sources) a game build that cooks starts with builds it again from its + source, with the id it has, and fills the texture cache, so the editor takes that Final from then + on; the refresh of an editor build keeps textures Fast, so a session never waits for RDO. A + cook that finds one still Fast fails with `Source_Failed` and says so. - **Animations.** An `.anim` is written cooked: its clips sampled as the runtime plays them (`asset.animation_compress`, [storage](../../mirai-std/references/anim-storage.md)). - **Prefab instances.** A scene with a stale instance is written with its instances up @@ -362,10 +381,12 @@ texture cache of the project; a failure is `Cook_Failed` with `Result.cook.error with an `Auto` `Collider` (its `Rigid_Body`, or that of its nearest ancestor with one, not `Static`) ships with its convex pieces (`asset.mesh_pieces_put`), decomposed once and kept in the collision cache (`Cook_Desc.collision_cache`): a shipped game reads them and decomposes nothing. -- **Compression.** A cook writes with the codecs of its project (`Cook_Desc.codecs`, from - `project.compression`): every texture is written again with the layouts and planes of fields - those codecs take, and with `None` its blocks are made without RDO. What a cook compresses with, - and the name of the compressor, are part of the key of each file it keeps. +- **Compression.** The cook is the only place the container's own compression remains: a game + has no store, so each file is written in the container format and compressed then, with the + codecs of its project (`Cook_Desc.codecs`, from `project.compression`) that this build has; + every texture is written again with the layouts and planes of fields those codecs take, and + with `None` its blocks are made without RDO. What a cook compresses with, and the name of the + compressor, are part of the key of each file it keeps. #### Compressor diff --git a/.agents/skills/mirai-projects/references/projects.md b/.agents/skills/mirai-projects/references/projects.md index 21e0b3c55..8c481e7c9 100644 --- a/.agents/skills/mirai-projects/references/projects.md +++ b/.agents/skills/mirai-projects/references/projects.md @@ -10,9 +10,12 @@ Targets, jobs, fingerprint, project shaders, cooking and shipped game (`cook/bui project.mirai manifest: what makes the folder a project settings.mirai project settings, the bindings in Input_Settings: what differs (optional) cook.mirai cooking roots that no scene names (optional) + packages.mirai the packages of the machine store the project uses, by the hash of their + manifest, and the folder each is listed in (optional; a plugin may have one) + packages/ .mirai: the text of each manifest packages.mirai pins (settings, + origin and asset ids), so a clone has them without the store assets/ assets, by theme: what is authored, the art sources beside what they make, - each with its sidecar (.import: settings, folder and asset ids), and - the documents their imports write + and the documents their imports write game/ game library of the project editor/ editor package of the project (engine-time) plugins// plugin.mirai, runtime/, editor/, shaders/, assets/ @@ -20,9 +23,6 @@ Targets, jobs, fingerprint, project shaders, cooking and shipped game (`cook/bui panels/ editor panels built by the chat agent (engine-time) tests/scenarios/ gameplay tests the editor plays (`mirai play test`), optional .mirai/ generated; not edited or versioned - imported/ what imports build from the sources that nobody edits (meshes, textures, sounds, - fonts), each file named by its id and what it was built from; a build makes - it again gen/ generated packages: the worlds and the root of each target plugins/ libraries of the engine plugins, with their stages and declarations editor library of mirai-editor and of the players it opens; play, that of @@ -32,7 +32,9 @@ Targets, jobs, fingerprint, project shaders, cooking and shipped game (`cook/bui shaders/ shader generation of the project build/ files of the jobs, sources and fingerprint of the last build per target cache/ machine caches (asset registry, textures, built meshes, compressed .anim) - thumbs/ asset previews of the editor + share/ what sharing keeps on this machine: the last snapshot joined, the index of + the files and the name of the offer (sharing a project, below) + thumbs/ views of scenes the editor took ``` The first-party plugins are in `/plugins//` with the same shape; the engine binaries, @@ -77,10 +79,11 @@ keeps its default; one the struct does not have is an error. - `import_sources`: the extensions of the files the importers of a plugin read (`.umap`): a repository with LFS keeps them in it as it keeps the sources of the engine's formats ([git and assets](../../mirai-assets/references/cli.md#git)). -- `build.compression`: what the assets of the project and its game are compressed with: `none`, - `open`, or empty for the best its engine has ([compressor](build.md#compressor)). The projects - of the repository of the engine (the demo, the templates) declare `open`; a project made from a - template does not keep it. +- `build.compression`: what the files of its shipped game are compressed with: `none`, `open`, or + empty for the best its engine has ([compressor](build.md#compressor)). Only the cook compresses + with it: the documents of the project and what the store and the exchange hold never depend on + a compressor. The projects of the repository of the engine (the demo, the templates) declare + `open`; a project made from a template does not keep it. - `development`: a tool for making the game (`profiler`, `tracy`). It goes into the `editor` and `play` libraries; the shipped game is generated with `shipped_world`, the world without them, and if its code reaches them the build fails with `Engine_Time_Code`. @@ -194,7 +197,6 @@ the `Project` owns everything it points to. and, apart from that, in the project's order. A cycle between inactive plugins is not an error. - The caches of `.mirai/cache` (`cache_path`) and `.mirai/thumbs` cost only time if deleted: the [registry](../../mirai-assets/references/formats.md#registry), the - [textures](../../mirai-imports/references/imports.md#texture-cache), the [compressed animations](../../mirai-core/references/host.md#content) and the [previews](../../mirai-editor/references/panels.md#previews) are made again, and the [capacities](../../mirai-rendering/references/frame-graph.md#passes) of the scenes are learned @@ -337,7 +339,8 @@ Each mode has its folder: switching between them compiles nothing. - Programs `mirai-editor` and `mirai-player`, which load the `editor` library of the project ([host](../../mirai-core/references/host.md#project-libraries)), and over the editor library `mirai-import`, that of `mirai import` - ([import](../../mirai-imports/references/imports.md#what-is-written)), and `mirai-author`, + ([import](../../mirai-imports/references/imports.md#what-is-written)) and of `mirai share` + and `mirai join` ([sharing](#sharing-a-project)), and `mirai-author`, that of `mirai ui` and `mirai scene` ([documents](../../mirai-agent-api/references/api.md#mirai-ui-and-mirai-scene)): this way the `mirai` tool links neither importers nor renderer and compiles before the generated shaders @@ -441,6 +444,7 @@ release of the tag; run by hand, it leaves the packages as artifacts. .mirai/engine/release/ libraries, programs, SDL3, decl/, engine.declared and engine.stamp .mirai/cache/plugins/ libraries of the engine plugins, minimal and speed .deps/ Odin, Slang and spirv-val/opt/cross; SDL3 on Linux and macOS + builds/ a store with the builds of the packages its plugins pin core/ engine/ ... Odin sources without tests, plugins/, third_party/freetype already built shaders/ templates/ assets/fonts/ editor/assets/ .agents/skills/ THIRD_PARTY_NOTICES.md ``` @@ -505,19 +509,23 @@ by its absolute path, which hands it to Git Bash ([launcher](#tool-and-machine)) | Operation | Command | | --- | --- | | Create a worktree ready to build | `mirai worktree new [--from ]` | -| Build and open the editor | `mirai edit [project] [--release] [editor arguments]` | -| Build and run the game | `mirai play [project] [--release] [player arguments]` | -| Build the editor without opening it | `mirai build editor [project] [--release]` | -| Build the shippable game | `mirai build game [project] [--out folder]` | +| Build and open the editor | `mirai edit [project] [--release] [--store folder] [editor arguments]` | +| Build and run the game | `mirai play [project] [--release] [--store folder] [player arguments]` | +| Build the editor without opening it | `mirai build editor [project] [--release] [--store folder]` | +| Build the shippable game | `mirai build game [project] [--out folder] [--store folder]` | | Build its dedicated server | `mirai build server [project] [--out folder]` | -| Build only the engine | `mirai build engine [--release] [--force]`|--release] [--force]` | -| Generate the world without building | `mirai generate [project]` | +| Build only the engine | `mirai build engine [--release] [--force]` | +| Generate the world without building | `mirai generate [project] [--store folder]` | | Run the content tool of a project | `mirai content [project] [-- arguments]` | +| Ingest a source from anywhere and add it to a project | `mirai import [--project P] [--folder F] [settings]` | +| Bring the packages a project pins to builds of this engine | `mirai import --refresh [--final] [--store folder] [--from folder]` | +| Serve a project peer to peer until stopped | `mirai share [project] [--port N] [--store S]` | +| Receive a shared project, or update a folder joined before | `mirai join [folder] [--address A] [--port N] [--store S]` | | Documents of interfaces and scenes, no editor | `mirai ui\|scene read\|write\|check ... [--project P]` | | Git plugins of a project | `mirai plugin add\|update\|remove\|sync\|list\|catalog\|check ...` | | Check types | `mirai check [ [-flag...]]`: a package relative to the working folder or to the engine, or absolute; none inside a project: its game, editor and plugins. In a runtime package it also fails on an allocation whose error nothing receives (`tools/mirai/alloc_lint.odin` says which calls and which packages), and in the runtime or editor part of a plugin on an import of `render` or `rhi` outside its tests or of the physics solver (`tools/mirai/solver_lint.odin`) | | Run tests | `mirai test [] [test\|file.odin...] [--list\|--failed\|--no-gpu\|--quiet] [-flag...]`: the package as for `check`; none inside a project: its game | -| Headless background editor | `mirai session start\|restart\|stop\|status [project] [--release] [--copy]` | +| Headless background editor | `mirai session start\|restart\|stop\|status [project] [--release] [--copy] [--store folder]` | | Call the editor API | `mirai rpc [key=value...\|json] [--select c] [--wait-until c=v] [--project P]` | | Capture without UI | `mirai capture [project] --out x.png [--camera NAME] [--reference r.png]` | | Validate demo image and cost | `mirai render check [project] [--views a,b] [--quality] [--release]` | @@ -529,7 +537,7 @@ by its absolute path, which hands it to Git Bash ([launcher](#tool-and-machine)) | Search the code | `rg [folder]` | | Check the machine and a project | `mirai doctor [project]` | | Merge and diff assets with git, once per clone | `mirai setup git [folder]` | -| Assemble the installed engine (binary) | `mirai package engine --out ` | +| Assemble the installed engine (binary) | `mirai package engine --out [--store folder]` | - Every command that builds does it in dev unless `--release` is asked for ([AGENTS.md](../../../../AGENTS.md#common-commands)). @@ -576,6 +584,89 @@ by its absolute path, which hands it to Git Bash ([launcher](#tool-and-machine)) symbolic link to `.agents/skills`: without it, the skills do not reach the agents (on Windows developer mode is needed). +## Sharing a project + +`cook/share` gives one person's project to another, peer to peer, with no server and no git: the +files of the project travel as a snapshot, and its heavy content as the builds of its packages, +by key, through [exchange](../../mirai-store/references/exchange.md) over `net`. + +- **Snapshot.** Every file under the root that `project.own_path` accepts: all but the generated + folder and the folders of version control. It is a tree of objects of the store: each file an + object of its bytes, cut on content, so a changed file shares the chunks the change did not + touch; each folder an object whose bytes list its entries sorted by name (a file with its + executable bit, or a folder, by the address and size of its object, a folder with the files and + bytes under it), whose meta carries the name of the project in the root only; a folder holds at + least one file. The same files give the same tree, so two snapshots that differ in one file + share every folder off its path, and a join after a change asks for those folders, the file's + new chunks and the nodes above them, a bound of the depth whatever the size of the project. No + limit is set on the size of a file. The size of an entry is that of what was put, so a file + saved while the snapshot is taken gives a consistent entry. The index in `.mirai/share` keeps + the address of each file by its size and time, so a file that did not change is not read again; + it learns a file only once it is settled (the rule of the source hashes of the library), so an + edit of the same size within the same time is read. +- **What a file carries.** Its bytes and its executable bit. Not its time: a joined file gets the + time it is written, which keeps build tools' staleness right. Not its other permissions: a + read-only file of the host is shared as any other, and a read-only file or folder of the joiner + is never replaced or removed, but reported. A link or a special file is left out with a message + that says to replace it with the file it points to. +- **Share.** `mirai share` takes the snapshot, opens a server host with its port mapped as + [collaboration](../../mirai-editor/references/collaboration.md) does, prints one invitation + once the router answered, and serves until it is stopped (Ctrl+C), printing who comes and goes + and how much went out. The snapshot is the ref `shares/`, so a + joiner finds it by the code and the names peers list reveal no secret. Peers read the namespaces + of snapshots, package manifests, builds (Final and Fast, each under its own namespace) and + `cook/cache` results: the owner exposes the whole machine library to whoever holds the + invitation, never the choices, the origins or the sources of the machine. A lease holds the + snapshot, its folders and files and the builds the project pins. +- **Join.** `mirai join` asks for the snapshot of the invitation only the host it connects to, + so a peer that joined from the joiner cannot name another; folders and files come from any + peer, verified by address. It fetches the tree a depth of folders at a time, refusing one whose + names could leave the folder, are absolute, name a device, repeat or collide where case is folded, + whose folders nest beyond the bound, hold more entries or bytes than a folder can or declare + files and sizes they do not hold (so the files read never exceed the bound, also from a folder + repeated under many names); checks that the disks have room; fetches the files, the project's + manifest and its start scene first and then the smallest first, many at once, and writes each + chunk by chunk at its offset into a temporary file renamed into place, so memory does not grow + with a file; then refreshes the packages as an editor does, with the peers as the + `cache.Source` of the library: a Final or a Fast build from the store, then from the peers, by + key, and a source in the folder is imported, in the quality an editor imports, only when no + peer gives a build of it. A join ends within a bounded time without progress whatever a peer + does or withholds: what is wanted while no chunk byte arrives and no object completes for a + while fails, reported by path, and the files after it are asked. A build that cooks refreshes + requiring Final, so a Fast build a join took never ships. A refresh asks the source for every + manifest and build it lacks in one batch before it goes package by package, so it waits for the + peers about once, not once per package. It prints progress at a steady pace and, at the end, + how to open the project. The store keeps what arrived: a join stopped and run again fetches only + the rest. The lease of the node holds what a share serves and what a join fetches and wrote. +- **Updating.** A folder joined before keeps its last snapshot in `.mirai/share`, by path, the base + of three ways: a file the host changed or removed is replaced or removed when the joiner left it + as it was; a file both changed is kept and reported, never overwritten. Each file is checked + again just before it is written or removed, so an edit made while the join runs is kept too. + The join puts the base's tree in the store before it asks, so what the host did not change since + is not asked, and a file to replace whose base the store lost is put from the folder first, cut + as the host's, so the chunks both versions share are not asked either. +- **Links.** A join never follows a link (a symbolic link, or a junction on Windows) inside its + folder: a file whose path passes through one, the file itself included, is neither written nor + removed but reported by path, with the link to replace, and so is the whole join when the + generated folder's state goes through one. Each path is checked when the join plans it and + again just before its rename or removal. The folder itself may be a link: it is the place the + joiner named. +- **Case.** A join probes whether the folder's file system folds case (APFS, NTFS). There paths + compare in lower case, so a host's rename that changes only case is the same file: it is + renamed in place, folders first, and never removed. Elsewhere it is a removal and a new file. +- **Incomplete.** A file that cannot be written or removed (read-only, a read-only folder, held + open, an object no peer has) is listed by path with why and how to fix it, the rest is done, and + `mirai join` exits with 3. The base it keeps holds the last entry of each such file, so running + the same join again does only those. +- **Peers.** While a join runs it serves what it has, with an invitation of its own, so a third + machine joins from either. After a join, `mirai share` in the folder serves it too. +- **Editors.** A collaboration session of the editor carries its project with this same node and + join over its own connections + ([the project](../../mirai-editor/references/collaboration.md#the-project)). +- Reaching a host is the invitation's: on another network one side needs a router that maps a port; + the failures say so, as they say an invitation whose share ended, another version of Mirai, a + store of another layout, a disk without room and a package no connected peer has a build of. + ## Tool and machine - `mirai setup git [folder]` wires the clone the folder is in, once per clone diff --git a/.agents/skills/mirai-std/SKILL.md b/.agents/skills/mirai-std/SKILL.md index 8dac26832..c4d67a889 100644 --- a/.agents/skills/mirai-std/SKILL.md +++ b/.agents/skills/mirai-std/SKILL.md @@ -103,6 +103,8 @@ description: Mirai standard library for games, with the packages `std/base`, `st [octree](references/spatial.md#octree-loose-octree) and [which to use](references/spatial.md#which-to-use). The AABB tree and the BVH are in [physics](../mirai-physics/references/physics.md#stdspatial). +- BLAKE3 of bytes, and the values of its 64 KiB groups a blob is checked by piece by piece + (`std/blake3`): the code; the official vectors are its test. - Streams, schemas, hashes, import and migration of values (`std/serial`): [std/serial](../mirai-assets/references/serial.md#stdserial). - What the structure of a `.ui` is judged by (`std/ui_format`): diff --git a/.agents/skills/mirai-store/SKILL.md b/.agents/skills/mirai-store/SKILL.md new file mode 100644 index 000000000..2de8b0ab7 --- /dev/null +++ b/.agents/skills/mirai-store/SKILL.md @@ -0,0 +1,186 @@ +--- +name: mirai-store +description: Mirai machine-wide content-addressed store (the `store` package) - chunks in + append-only packs, objects as trees of chunks read by group and offset, refs, commits and + batches, index generations, the sweep that compacts packs beside running writers, the scrub, + leases and the default root - and the `exchange` package that moves the chunks of objects and + refs between peers over `net` (asks with a budget, wants as objects, priorities, several + sources, lying peers, blocking fetch). Use when storing or sharing heavy content between + projects or peers, building an asset library or a derived data cache on top of it, fetching an + object from another machine, or diagnosing a corrupt, missing, slow or growing store. +--- + +# Mirai store + +`store` (layer 1) imports only `base`, `core`, `std` and `compressor:data`: a plain library a game +binary may link. +It knows chunks, objects and refs and interprets none of them: the transport, the library and the +derived data cache that use it decide what a ref means and what is live. + +## Map + +- Root, layout, errors, file operations and their test hooks: `store/store.odin`; what differs per + system (sharing and deleting on Windows, mapping, a full volume, whether a process runs): + `store/system_*.odin`. +- Cutting bytes into chunks: `store/chunk.odin`; node records, building an object's tree into a + sink and resolving a byte or a group offset to a leaf: `store/node.odin`. +- Packs, frames, the writer, commit and seal, scanning a pack: `store/pack.odin`. +- What a `Store` value knows (generation, table of chunks, folded refs) and reading the root again: + `store/index.odin`. +- Putting and reading objects and chunks: `store/object.odin`; refs: `store/refs.odin`. +- The codecs of frames, what a value decodes and writes with, and the hook a test gives it to act + as a build of another compressor: `store/codec.odin`. +- Sweep, compaction, generations, scrub and the sweep's claims: `store/sweep.odin`; leases: + `store/leases.odin`. The sweep a long-running owner calls, with what is live on the machine: + `asset.store_sweep` (`asset/collect.odin`). +- Moving objects and refs between peers (`exchange`): what its owner calls → + [exchange](references/exchange.md#what-the-owner-calls), its rules → + [invariants](references/exchange.md#invariants), its tests → + [verification](references/exchange.md#verification). +- Sharing a whole project over the exchange (`mirai share`, `mirai join`): + [sharing](../mirai-projects/references/projects.md#sharing-a-project). + +## Workflow + +1. A writer puts objects (bytes, or a native asset's container chunks as groups) and refs into a + batch, and commits once when the batch is done; a batch that grows past its bound commits on + its own. A large stream is put with a worker of a job system: the caller cuts, jobs hash and + append. +2. A reader opens an object by its root and reads a group from an offset; a chunk of a sealed pack + is a slice of its mapping. +3. A long-running owner sweeps with what is live on the machine and scrubs at a byte budget. + +## Invariants + +- A chunk is 1 to `CHUNK_MAX` bytes named by the standard BLAKE3 of its bytes; an object is a tree + of nodes over its chunks and its address is its root node's. What is hashed never depends on a + compressor: a put encodes a chunk's frame with this build's data codec when that shrinks it and + records the codec in the frame (nodes stay as they are, so a sweep of any build walks them), and + a read decodes into the caller's buffer; a frame stored as it is is a slice of the mapped pack. + The root's `LAYOUT` carries the layout of packs, frames, generations, nodes and the cut: a root + of another layout, or one of the store before packs, is refused, never read or migrated. +- A frame of a codec this value does not decode is, to it, a chunk that is not here to read or + serve: a lookup misses it, a put of its address appends a frame it reads, and its table keeps + the copy it reads over it. For what is kept it is here: a ref, a lease and a commit find an + object whole whatever the codecs of its frames, and a sweep keeps such frames as they are and, + of several copies, copies and indexes one every build reads, so builds with the open and the + private compressor share a root. A frame a caller offers (`chunk_put_frame`) is decoded and + checked against its address before it is kept. A frame whose codec id, sizes + or bytes lie is damage: refused when read, tombstoned by a scrub. +- Every chunk lives in a pack, appended only by the writer that made it, whose lease (`w-` and its + id) says it runs. A put reserves its range of the pack under the lock and writes outside it, so + threads write at once; a small object is one write, and nothing is created, renamed or flushed + per put, because a file and a flush per object cost far more than an append. Cuts stay in order + on one thread while jobs hash and append the chunks, and leaves join the tree in the order of + the cuts, so an object is the same put on one thread or many. +- A commit writes the batch's frames and the commit frame that makes them visible, then flushes + once: a commit that returns success is durable. Readers take a pack's frames only up to its + last commit whose count and sum of heads match, so a torn tail is never read; a ref is a frame + of the same pack written after the chunks it reaches. What a batch names in another writer's + open pack is flushed before its commit frame is written, so a crash never keeps a ref and loses + what it names. +- A pack is written only by its writer: it is sealed (a trailer of its chunks sorted by address) + at its size bound or when its writer closes. A sweep never writes into another's pack: one whose + writer's lease is gone is folded up to its last commit and retired, and a writer that is in fact + alive (another machine's lease not refreshed in time) finds its lease gone after its commit + frame and writes the batch again in a new pack under a new lease before the commit returns. +- Duplicates are allowed and harmless: a writer names a chunk it finds already stored instead of + writing it, and two writers may store the same chunk at once; compaction keeps one copy. A + chunk found with a copy older than a quarter of `GRACE` gets a Touch frame, so a sweep keeps it. +- A `Store` value reads the root again when a lookup misses and before `has`, `chunk_has`, + `ref_get` and listing refs, which must see what other values committed; nothing polls. The + state of a value, shared by its copies and the threads that use it, lives on the heap: it + outlives the calls and crosses threads, so the allocator a store is opened with, also an + arena, serves only its root and what a call returns. +- Only a sweep writes a generation, one sweep at a time by the claim it creates in `tmp/sweep`; a + claim of a holder that ended is taken by moving it aside and checking it is the one judged, and + a generation is published only where none of that number is, so exactly one sweep owns it. It + marks what refs, leases and its caller name, every chunk under them at every level, and what + was written or touched within `GRACE`; + copies the live chunks of sealed packs that are mostly dead, or small, into its own pack; writes + the next generation of every sealed pack, refs folded, synced and renamed into `index/`; and + retires the replaced packs only after that. Active packs of live writers are never touched. +- A sweep never loses a committed put beside writers of other processes, which it cannot stop as + an in-process collector would. A retired pack lives in `retired/` under the generation that + dropped it, and a reader that misses everywhere else reads it there. A writer looks what its + batch found stored up again when a newer generation appeared, before its commit frame and after + it, and copies back what was retired; a lease update does the same for the addresses it adds. + Each lease carries the generation its holder last checked against; a sweep copies back what a + live root reaches that only a retired pack holds, and deletes a retired pack only once every + other writer's lease carries its generation and every other lease does or is what the sweep + marked from, deleting before it writes anything so it frees space on a full volume. +- What another process's open pack holds stays until that pack is sealed: a sweep folds only + sealed packs. +- A ref is folded into a generation by the newest frame of its name; a removed ref stays there as + a record while a pack still being written could hold an older frame of it. +- Nothing is checked on a local read or a serve: a scrub checks the chunks of sealed packs against + their addresses at a byte budget and a Tombstone frame stops every value reading a damaged copy, + which a sweep then drops; a receiver checks what it receives. A read never touches a mapping + beyond the file's size now: a pack cut below a chunk is damaged, an error. +- A full volume is `No_Space`, never `Io`; a pack that will not delete (one a process maps on + Windows) is skipped, counted and deleted by a later sweep. +- Slices of a mapping handed to a caller have no lifetime, so a pack a value lent stays mapped + until it closes: a sweep reports what it deleted that its value lent as held, apart from what it + freed, and a value closes the gone packs it never lent when it reads a new generation. What + other processes map is invisible to it and stays on disk until they close it. +- The store never decides what is live: refs, leases and what its caller names are, and what was + written or touched within `GRACE`. A process that uses addresses holds a lease that names its + machine, pid and process start: on that machine it holds while the process runs, whatever the + clock or a suspend did, and stops at once when the process ends; one of another machine on a + shared root holds while it is younger than `LEASE_MAX_AGE`. +- On Windows every file of the store is opened sharing delete, and replacing or removing one uses + POSIX semantics, so a reader, another process or an antivirus holding it does not stop it; + sharing violations are retried briefly. +- Ref namespaces and names are lowercase ASCII, digits, `.`, `_` and `-`, the same on every + system. +- External bytes are cut with FastCDC 2020, normalized at level 1 (`CHUNK_MIN`, `CHUNK_AVG`, + `CHUNK_MAX`); the cut function and its gear table are part of the layout, since changing either + changes every address. A cut is decided only with `CHUNK_MAX` bytes ahead or at the end, so a + stream fed in any split cuts as the whole does, holding two `CHUNK_MAX` windows and allocating + nothing per chunk. A container chunk of a native asset is cut where its writer says, never on + content: consecutive pieces merge up to `CHUNK_MAX`, and a larger piece is cut at `CHUNK_MAX` + offsets from its start, its remainder merging with the pieces after it. +- A node is a chunk: a `Node` record, its entries (address and data bytes under it), and in the + root only its groups (runs of leaves, one per container chunk; a plain file has one) and meta, + which the store does not read. Every object has a root, also an empty one. Each node has one + encoding: reserved bytes are zero and the parser checks every count, size and sum against the + bytes, the level against `NODE_LEVEL_MAX`, and each child against its parent's level and entry, + so hostile nodes return `Corrupt` and a walk is bounded by the root's level. +- Nodes are grouped with content-defined boundaries over their entries' addresses, so an edit + rewrites one node per level. The level that ends as one node is the root; when the groups and + meta do not fit beside its entries, a root of one entry goes above it. A builder holds one open + node per level and hands every chunk, data before the nodes naming it and the root last, to the + sink its caller gives. A root's groups follow each other from leaf 0 and byte 0, each with the + leaf and the byte where it begins; above level 0 only the leaves say where those are, so opening + such an object walks its leaf nodes once and refuses a group that does not begin on its leaf. +- No input terminates the process: damaged, truncated or hostile packs and generations, random + tables and pieces, out-of-range indices and memory that runs out return an `Error`; a pack that + does not read is skipped and a generation that does not read refuses the root. + +## Verification + +- `mirai check store` (it runs the allocation lint) and `mirai test store`. +- Focal tests: `writers_of_several_processes_and_threads_share_one_root`, + `a_stream_put_on_workers_is_the_same_object`, + `a_commit_paused_across_a_sweep_keeps_what_it_names`, + `a_live_writer_whose_lease_went_stale_keeps_its_commits`, + `two_sweeps_taking_a_stale_claim_publish_once`, + `an_address_added_to_a_lease_during_a_sweep_is_kept`, + `a_pack_cut_anywhere_in_its_tail_opens_to_its_last_commit`, + `a_commit_that_returns_is_durable`, + `a_small_put_writes_once_and_a_batch_flushes_once`, + `an_object_of_several_groups_reads_by_group_and_offset`, + `a_sweep_beside_writers_loses_no_committed_put`, + `compaction_keeps_every_live_chunk_and_drops_the_dead`, + `a_full_volume_is_no_space_and_a_sweep_frees_it`, `a_scrub_tombstones_a_damaged_chunk`, + `hostile_packs_and_generations_are_refused`, `hostile_inputs_return_errors` and + `fresh_leases_keep_their_current_addresses`, `a_chunk_check_tombstones_a_damaged_copy`, + `frames_compress_and_addresses_are_of_the_bytes`, + `a_codec_this_build_lacks_reads_as_missing_and_a_put_repairs_it`, + `a_damaged_encoded_frame_is_a_damaged_chunk`, `hostile_chunk_frames_are_refused`, + `a_sweep_of_copies_of_two_codecs_keeps_one_every_build_reads`, + `a_ref_on_an_object_of_a_codec_this_build_lacks_is_kept`; for + chunks and nodes + `chunks_are_the_same_streamed_in_any_split_and_whole`, + `an_edit_changes_two_data_chunks_and_one_node_per_level`, `hostile_nodes_return_errors` and + `groups_resolve_at_every_boundary`. diff --git a/.agents/skills/mirai-store/references/exchange.md b/.agents/skills/mirai-store/references/exchange.md new file mode 100644 index 000000000..13d103aa9 --- /dev/null +++ b/.agents/skills/mirai-store/references/exchange.md @@ -0,0 +1,132 @@ +# Exchange: objects and refs between peers + +`exchange` (layer 3) moves the chunks of a `store`'s objects and its refs between peers over the +connections of `net` hosts its owner already runs. It imports `store` and `net` and nothing +engine-time, so a headless peer and the editor link the same code. Its messages (protocol 5) are +in `exchange/wire.odin`. + +## What the owner calls + +- `channels()` gives the two reliable channels the exchange needs; the owner puts them in the + `Host_Desc.channels` of every host at the same index and passes that index as `Desc.channel`. +- `init` sizes every buffer, table and job slot from the `Desc`; nothing is sized by an object, so + an object of any size moves. `Desc.worker` is the job system and worker of the thread that + pumps: reading, hashing, storing, walking, serving and committing run as its jobs; without one + they run on the pump. `destroy` waits for them, so it is called from that thread. +- Per host poll: `peer_add` on `Connected`, `peer_remove` on `Disconnected`, `receive` on every + `Message` (it says whether the message was its own). Then `update(now)` before `host_update`. +- A connection is given from its start: a message that began before `peer_add` is not read from + its middle, so a restarted exchange takes new connections. +- `want` takes an object's root; asking again raises the priority. `forget`, `progress`, + `peer_progress`; `ref_ask`, `names_ask`, `query_result`, `query_end`. +- `want_addresses` gives the root of every want, in progress or kept: the owner puts + them in its store lease so a sweep keeps them and everything already received under them; the + exchange holds no lease. +- `fetch` and `fetch_ref` block a job thread until what they name is complete in the store, or + return `Missing` once no connected peer has it; the owner keeps pumping meanwhile, since every + procedure takes the exchange's lock. `fetch_all` and `fetch_refs` do the same for a set, asked + together: a caller that needs many objects or refs waits on the set, since each blocking call + costs at least a round trip. Success is what the store says now: a kept want whose object was + swept is asked again. `destroy` wakes every waiter with `Closed` and frees once none is left. + +## Invariants + +- Asks are stateless, by chunk address with a byte budget; there is no session or fetch id. Any + peer that holds a chunk answers, also one still receiving the object, and a chunk is atomic, so + a peer has it or lacks it. The answer to an entry is its chunk, then its descendants depth first + while their bytes fit the budget, stopping at the first one missing; then `Done` ends the ask. +- The receiver walks the tree of each want over what its store holds and asks only what it lacks: + a chunk already stored is never asked, so a changed object costs its new chunks and the nodes + above them. A root not here is asked with `ROOT_BUDGET`, so a small object comes whole in one + round trip; the chunks and nodes the walk finds missing are asked with no budget, since the + receiver may hold what is under them. A missing node stops the walk of its want until it + arrives, and the next missing nodes are asked meanwhile. +- Priority belongs to the receiver: it asks for the most important wants first and nothing about + priority goes on the wire beyond the order of the entries. At most `BATCHES` asks are in flight + per peer, each sized to what the peer delivers in a round trip, so they keep twice that + outstanding without queueing a low priority want ahead of a later high one. A root not here + counts only against the entries of an ask, since most objects are small. A want keeps at most + `WALK_AHEAD` chunks found and not asked, so many wants share the table of needs. +- Chunks are spread among the connected peers; a peer that lacks one is not asked for it again for + a while, and a want that no peer can be asked for, or one of whose chunks no peer has, is + `Missing`, asked again when a peer joins. What a peer was asked stays reserved to it only while + it makes progress on that want; otherwise it goes to others, that peer is not asked for the + want for a while, and its late answers release nothing a second time. +- A chunk travels in a codec its receiver decodes: before its first ask a receiver says which + frame codecs its store decodes (`store.codecs_read`), and a server sends a frame as stored when + the receiver decodes its codec, else decoded and encoded again in an open codec the receiver + decodes, or as it is (`store.chunk_read_in`); until a receiver says, chunks go as they are. The + receiver's store decodes what it gets, checks it against its address and keeps the frame as it + came (`store.chunk_put_frame`), one decode per chunk asked. Peers of builds + with different compressors exchange every object, and each ends reading all of them. +- Nothing is checked on serve. Nothing unverified reaches the store: a chunk is taken only as the + answer to an entry asked of that peer or as a descendant of the entry it is answering within its + budget, and only once it hashes to the address the ask or its parent named. A node is parsed and + checked against its parent's level and entry before its entries are asked. A chunk that fails + its address is reported with `Bad` and marks that peer wrong for that want, never the want; the + peer checks its copy and tombstones it if damaged, so it lacks it from then on and fetches it + again when it wants it. A tree whose chunks make their addresses but do not hold together fails + the want, since every peer would send the same. Marks are per peer and leave with it, never + passing to the next peer in its slot. +- A want is complete only after the commit that holds its last chunk: received chunks join the + store's batch and the exchange commits once for the wants that finished, at most every + `COMMIT_SECONDS`, and every `COMMIT_IDLE` for what arrived meanwhile, so a restarted receiver + walks again from what it committed without asking for it. +- A want names an object by its root, which is always a node: a root that hashes right but is no + node fails the want, since every peer would send the same. +- What a peer exposes is the owner's `Desc.expose`; without one no namespace is read. Names come + sorted, a page at a time after the last one seen. A ref is asked of every connected peer, or of + one when only that peer may say it (the snapshot of an invitation, from the host it names): + refs name content, and only content is verified by address. +- Hostile input: a malformed message, an ask beyond the limits, descendants past a budget or an + answer nobody could have asked for in that form marks the peer bad, and it is neither asked nor + served again; a well formed message about something not asked for is ignored. An ask that does + not fit the room for `BATCHES` asks is refused, and answers to questions find room only while + the asker keeps a few waiting, so a peer cannot make another send without bound. A peer that + does not answer a question within `QUERY_SECONDS` is taken to know nothing of it and its credit + comes back, so a silent or overloaded peer never holds a question or its slot; a question its + owner ended stays until every peer asked answered, gave up or left. +- The pump keeps the bookkeeping: wants, needs, flights, peers and the order of answers. Each job + owns a slot sized at `init` (a ring of received chunks and their bytes, walks, the commit, a + ring of answers per peer, checks of reported damage) and touches nothing else; the pump hands + work to a slot and applies its result under the lock, so a job never takes the lock and the + pump may wait for one, helping, when a ring is full. Received chunks are applied in the order + they came, so a node is parsed and its entries are needs before a descendant streamed after it + is taken; a descendant that came first is hashed and waits for its parent. A received node is + walked by its own job, as far as it names chunks, and the walk goes on with the store's nodes + in a walk job. +- Serving and receiving do not allocate in the exchange after `init`, except listing a namespace, + for which the store reads its root. Answers to ref and name questions are read on the pump. + +## Verification + +- `mirai check exchange` and `mirai test exchange` (no GPU): every test runs peers on `net`'s + `Sim_Network` in virtual time with temporary stores. The rate over UDP on loopback is the opt-in + `objects_over_udp_loopback` with `-define:EXCHANGE_BENCHMARK=true`; it prints the CPU of the + exchange (pump and jobs) apart from that of `net`'s worker. Tests run the jobs on the pump, so + virtual time steps fast; `-define:EXCHANGE_TEST_JOBS=true` runs them on a job system, each step + waiting for them. The memory bound of + `a_ten_gigabyte_object_moves_within_bounded_memory` is the process's footprint, checked when it + runs alone with `-define:ODIN_TEST_THREADS=1`. +- Focal tests: + `objects_of_every_size_arrive_verified_under_loss_and_jitter`, + `a_high_priority_want_asked_later_completes_before_a_large_low_one`, + `a_transfer_resumes_after_the_connection_drops_and_after_the_store_reopens`, + `a_liar_is_marked_and_the_object_still_arrives`, + `a_receivers_bad_makes_the_server_repair_or_lack`, + `a_peer_still_receiving_an_object_serves_its_chunks_to_a_third`, + `refs_are_asked_and_listed_and_a_namespace_not_exposed_gives_nothing`, + `the_blocking_form_returns_for_a_hit_a_miss_and_a_disconnect`, + `the_batch_blocking_forms_return_for_a_hit_a_partial_miss_and_a_disconnect`, + `many_small_objects_wanted_together_take_round_trips_that_do_not_grow_with_their_count`, + `a_one_byte_insert_sends_only_the_new_chunks`, + `a_ten_gigabyte_object_moves_within_bounded_memory`, + `hostile_messages_are_refused_without_a_crash`, + `hostile_trees_fail_the_want_without_a_crash`, `a_peer_past_the_budget_of_an_ask_is_bad`, + `a_game_channel_keeps_its_delay_small_during_a_transfer`, + `a_peer_that_holds_a_want_without_progress_loses_it`, + `a_peer_in_a_freed_slot_is_not_blamed_for_its_predecessor`, + `late_ref_answers_return_their_credits`, + `a_late_done_does_not_release_what_another_peer_was_asked`, + `peers_of_different_codecs_exchange_objects_both_read` and + `destroy_wakes_a_waiting_fetch`. diff --git a/README.md b/README.md index 75be61840..55afa91d7 100644 --- a/README.md +++ b/README.md @@ -162,6 +162,7 @@ with a map from tasks to references. | Projects and build | [mirai-projects](.agents/skills/mirai-projects/SKILL.md) | | Standard library | [mirai-std](.agents/skills/mirai-std/SKILL.md) | | Platform and audio | [mirai-platform](.agents/skills/mirai-platform/SKILL.md) | +| Content-addressed store | [mirai-store](.agents/skills/mirai-store/SKILL.md) | The layers and contracts are in [ARCHITECTURE.md](docs/ARCHITECTURE.md) and the project rules in [AGENTS.md](AGENTS.md). diff --git a/THIRD_PARTY_NOTICES.md b/THIRD_PARTY_NOTICES.md index e22eba65e..0adfb5135 100644 --- a/THIRD_PARTY_NOTICES.md +++ b/THIRD_PARTY_NOTICES.md @@ -42,6 +42,10 @@ folders, which include the notices of the work they carry. Parts of Mirai are rewrites in Odin or Slang of the algorithms of these projects. Their notices are kept here and, where the license asks for it, in the header of the file. +- **BLAKE3** (CC0 1.0, Apache-2.0, or Apache-2.0 with LLVM exceptions; + © 2019 Jack O'Connor and Samuel Neves), . + `std/blake3`, translated from the reference implementation under CC0 1.0, with test vectors. + Game. - **Clay v0.14** (zlib; © 2024 Nic Barker), . `ui/layout_tree.odin`, altered. Game. - **MikkTSpace** (zlib; © 2011 Morten S. Mikkelsen), . @@ -66,6 +70,11 @@ are kept here and, where the license asks for it, in the header of the file. of islands of `cluster_islands.odin`. Game. That retry where whole levels stop in `cook/import/levels.odin`, the vertex codec and the quaternion and exponential filters of `cook/import/gltf/meshopt.odin`. Editor. +- **Lore** (MIT; © 2026 Epic Games, Inc.), . + The streaming windows of the chunker of `store/chunk.odin`, and the reservation of a range of a + pack under its lock, written outside it, a failed write giving its range back unless another + was reserved after it, of `stage_write`, `reserve_locked` and `land_locked` in + `store/pack.odin`, altered. Game. - **Mesh2Motion** (MIT; © 2025 Scott Petrovic), . The skinning solvers of `cook/rig/weights.odin`, altered. Editor. - **ACL 2.1, Animation Compression Library** (MIT; © 2017 Nicholas Frechette), diff --git a/asset/collect.odin b/asset/collect.odin new file mode 100644 index 000000000..e975973fd --- /dev/null +++ b/asset/collect.odin @@ -0,0 +1,33 @@ +package asset + +import "core:time" +import "mirai:store" + +// Sweeps the machine's store: the one procedure a long-running owner of it +// (a share, an editor) calls now and then. What stays is what live names +// (what the owner holds open), what every fresh lease names, every ref of +// every namespace, and the outputs and thumbnails of the builds those refs +// name. A project reaches its pins through these refs while it is refreshed +// (REF_BUILDS, REF_PACKAGES), and its root carries the text of each manifest +// it pins (PACKAGE_FILES), so a manifest swept is taken from there again. +store_sweep :: proc(s:^store.Store,live:[]store.Hash,now:Maybe(time.Time)=nil) -> (result:store.Sweep_Result,error:store.Error) { + held:=make([dynamic]store.Hash,0,len(live)+256,context.temp_allocator) + append(&held,..live) + namespaces:=store.ref_namespaces(s,context.temp_allocator) or_return + for namespace in namespaces { + refs:=store.ref_list(s,namespace,context.temp_allocator) or_return + builds:=namespace==REF_IMPORTS || namespace==REF_IMPORTS_FAST || namespace==REF_BUILDS + for ref in refs { + append(&held,ref.hash) + if !builds {continue} + // A build that does not read keeps only itself. + b,build_error:=build_get(s,ref.hash,context.temp_allocator) + if build_error!=.None {continue} + for o in b.outputs { + append(&held,o.object) + if o.thumbnail!={} {append(&held,o.thumbnail)} + } + } + } + return store.sweep(s,held[:],now) +} diff --git a/asset/compress.odin b/asset/compress.odin index bbd388614..11351aa93 100644 --- a/asset/compress.odin +++ b/asset/compress.odin @@ -9,18 +9,20 @@ import "mirai:project" import "mirai:std/base" import "mirai:std/jobs" -// What a write may compress with, as the project of what it writes allows (build.compression). -// Types.codecs says it for the documents made with them. +// What a write may compress with. Types.codecs says it for the documents made with them. Only +// the files a cook ships are compressed, as their project says (project_codecs); what the store +// holds and the documents of a project are written with None, so what is hashed and shared never +// depends on a compressor. Codecs :: enum u8 { + None, // no stage of a compressor: each kind keeps the modeling of its data, chosen without one Open, // the formats of the open compressor, which every build reads Best, // the best the compressor of this build has - None, // nothing is compressed } -// What the assets of the project at root, built with the engine at engine_root, are compressed -// with (build.compression): Best only where the games of that engine have the private compressor, -// since what a game does not decode it cannot ship. A folder without a project, as the assets of -// the engine and of its plugins: Open. +// What the files a cook ships of the project at root, built with the engine at engine_root, are +// compressed with (build.compression): Best only where the games of that engine have the private +// compressor, since what a game does not decode it cannot ship. A folder without a project, as +// the assets of the engine and of its plugins: Open. project_codecs :: proc(root,engine_root:string) -> Codecs { m,error:=project.manifest_read(project.path(root,project.PROJECT_FILE,allocator=context.temp_allocator)) if error.code!=.None {return .Open} diff --git a/asset/container.odin b/asset/container.odin index cb2125718..db900dd1e 100644 --- a/asset/container.odin +++ b/asset/container.odin @@ -131,6 +131,7 @@ Error :: enum u8 { IO, // a file could not be read or written Invalid, // an argument that the format cannot hold Codec_Missing, // a stream of a format the compressor of this build does not decode + Not_Here, // a part of an asset of the store that the store does not hold yet: asked again once it arrives } Blob :: struct { @@ -168,7 +169,7 @@ file_owner_set :: proc(path:string,owner:base.Asset_Id) -> Error { } // One entry of the directory, checked against the file: inside it, aligned, -// and with a decompressed size a stream holds. +// and with a decompressed size the pieces its stored bytes hold can make. @(private) chunk_entry_check :: proc(header:Header,entry:Chunk_Entry) -> bool { directory_end:=header.directory_offset+u64(header.chunk_count)*CHUNK_ENTRY_SIZE @@ -177,7 +178,8 @@ chunk_entry_check :: proc(header:Header,entry:Chunk_Entry) -> bool { if entry.offset&(u64(1)<=STREAM_HEAD + // Each piece of a stream takes a word of its table: what it decodes to is bounded by what it stores. + case .Stream: return entry.uncompressed_size=u64(STREAM_HEAD+4*stream_pieces(int(entry.uncompressed_size))) } return false } @@ -239,8 +241,11 @@ blob_stored :: proc(blob:^Blob,entry:Chunk_Entry) -> []u8 { } // The content of a chunk, into dst of its uncompressed size, with the decoder of the reader. blob_read :: proc(blob:^Blob,entry:Chunk_Entry,dst:[]u8,decoder:^Decoder) -> Error { - if u64(len(dst))!=entry.uncompressed_size {return .Invalid} - stored:=blob_stored(blob,entry) + return chunk_decode(entry,blob_stored(blob,entry),dst,decoder) +} +// The content of a chunk from its stored bytes, into dst of its uncompressed size. +chunk_decode :: proc(entry:Chunk_Entry,stored:[]u8,dst:[]u8,decoder:^Decoder) -> Error { + if u64(len(dst))!=entry.uncompressed_size || u64(len(stored))!=entry.size {return .Invalid} switch entry.compression { case .None: copy(dst,stored) @@ -268,7 +273,7 @@ Build_Chunk :: struct { // Writes a whole file into w, which has to be empty, compressed with profile as codecs allow. // scratch holds a chunk while it is compressed. -blob_build :: proc(w:^serial.Writer,kind:Fourcc,kind_version:u16,id,owner:base.Asset_Id,flags:Header_Flags,chunks:[]Build_Chunk,scratch:^[dynamic]u8,profile:=data.Profile.Fast,codecs:=Codecs.Open,worker:=jobs.Worker_Ref{}) -> Error { +blob_build :: proc(w:^serial.Writer,kind:Fourcc,kind_version:u16,id,owner:base.Asset_Id,flags:Header_Flags,chunks:[]Build_Chunk,scratch:^[dynamic]u8,profile:=data.Profile.Fast,codecs:=Codecs.None,worker:=jobs.Worker_Ref{}) -> Error { if w.count!=0 || len(chunks)>MAX_CHUNKS {return .Invalid} encoder:Stream_Encoder if codecs!=.None {stream_encoder_init(&encoder,scratch.allocator if scratch.allocator.procedure!=nil else context.allocator) or_return} diff --git a/asset/container_test.odin b/asset/container_test.odin index aeddf99cd..9ec7f1f06 100644 --- a/asset/container_test.odin +++ b/asset/container_test.odin @@ -61,7 +61,7 @@ blob_round_trips_chunks_with_alignment_and_compression :: proc(t:^testing.T) { {tag=fourcc("BBBB"),index=2,alignment=64,compression=.Stream,data=noise}, {tag=fourcc("EMPT"),alignment=8}, } - testing.expect_value(t,blob_build(&w,KIND_TEST,7,{1,2},{3,4},{.Cooked},chunks[:],&scratch),Error.None) + testing.expect_value(t,blob_build(&w,KIND_TEST,7,{1,2},{3,4},{.Cooked},chunks[:],&scratch,codecs=.Open),Error.None) blob,error:=blob_open(serial.writer_bytes(&w)) testing.expect_value(t,error,Error.None) testing.expect_value(t,blob.header.kind,KIND_TEST) @@ -111,7 +111,7 @@ blob_open_rejects_files_that_contradict_themselves :: proc(t:^testing.T) { payload:=make([]u8,400);defer delete(payload) for &b,i in payload {b=u8(i%7)} chunks:=[?]Build_Chunk{{tag=fourcc("AAAA"),alignment=16,data=payload[:100]},{tag=fourcc("BBBB"),alignment=8,compression=.Stream,data=payload}} - blob_build(&w,KIND_TEST,0,{},{},{},chunks[:],&scratch) + blob_build(&w,KIND_TEST,0,{},{},{},chunks[:],&scratch,codecs=.Open) good:=serial.writer_bytes(&w) for cut in 0.. Animation_Desc { animation_round_trips_and_fills_a_library :: proc(t:^testing.T) { defer free_all(context.temp_allocator) types:Types;types_init(&types);defer types_destroy(&types) + types.codecs=.Open // what a cook ships: the streams under test storage:Test_Animation desc:=test_animation(&storage) path:=fmt.tprintf("%s/lift.anim",base.test_folder(t,"asset")) diff --git a/asset/document.odin b/asset/document.odin index 798291cb5..1ea31b57b 100644 --- a/asset/document.odin +++ b/asset/document.odin @@ -390,11 +390,13 @@ document_write :: proc(doc:^Document,w:^serial.Writer,profile:=data.Profile.Fast return blob_build(w,doc.kind,doc.kind_version,doc.id,doc.owner,doc.flags,chunks[:],&scratch,profile,doc.codecs,worker) } -// Writes the document to a file, whole or not at all. +// Writes the document to a file, whole or not at all; a file that already +// holds its bytes is left untouched. document_save :: proc(doc:^Document,path:string,sync:=false,profile:=data.Profile.Fast) -> Error { bytes:=document_bytes(doc,doc.allocator,profile) or_return defer mem.free_bytes(bytes,doc.allocator) - return serial.file_write_atomic(path,bytes,sync)==.None?.None:.IO + _,error:=serial.file_write_changed(path,bytes,sync) + return .None if error==.None else .IO } // The file the document is, as bytes from allocator, aligned for its chunks. @@ -436,9 +438,7 @@ document_from_asset :: proc(doc:^Document,a:^Asset,allocator:=context.allocator) error:=column_size(doc,&column,source.rows*from.size) append(&target.columns,column) if no_column_name!=nil {error=.Out_Of_Memory} - decoder:^Decoder - if error==.None && entry.compression!=.None {decoder,error=asset_decoder(a)} - if error==.None {error=blob_read(&a.blob,entry,target.columns[at].data,decoder)} + if error==.None {error=asset_chunk_read(a,from.chunk,target.columns[at].data)} if error!=.None { document_destroy(doc) return error @@ -449,13 +449,22 @@ document_from_asset :: proc(doc:^Document,a:^Asset,allocator:=context.allocator) } // The file bytes are, written again with profile as codecs allow: the same content with its -// chunks, the pages of a mesh and the tables of blocks compressed again. With a worker, the -// pieces of a large chunk are compressed on the workers of its job system. +// chunks, the pages of a mesh and the tables of blocks compressed again. A file written without +// a compressor (Codecs.None) does not say which chunks its writer meant to compress: with codecs, +// every column becomes a stream where that shrinks it, but those that hold streams of their own +// (the page_data of a mesh, the blocks of a table of blocks), which are read by offset. With a +// worker, the pieces of a large chunk are compressed on the workers of its job system. document_repack :: proc(bytes:[]u8,profile:data.Profile,codecs:Codecs,allocator:=context.allocator,worker:=jobs.Worker_Ref{}) -> (repacked:[]u8,error:Error) { doc:Document document_read(&doc,bytes,allocator) or_return defer document_destroy(&doc) doc.codecs=codecs + if codecs!=.None { + for &table in doc.tables { + if doc.kind==KIND_MESH && table.name=="page_data" || strings.has_suffix(table.name,"_blocks") {continue} + for &column in table.columns {column.compression=.Stream} + } + } if doc.kind==KIND_MESH {mesh_pages_repack(&doc,profile) or_return} document_blocks_repack(&doc,profile,codecs) or_return return document_bytes(&doc,allocator,profile,worker) diff --git a/asset/kinds.odin b/asset/kinds.odin index dd99b4c82..3b8b68f95 100644 --- a/asset/kinds.odin +++ b/asset/kinds.odin @@ -141,7 +141,7 @@ version_message :: proc(a:^Asset,types:^Types,path:string,current:u16=0,allocato // Where an asset built from a source file came from: what an importer needs // to know whether it is out of date. It is the row of the table "import". Import_Info :: struct { - source:[260]u8 `mirai:"text"`, // path of the source file, relative to the project + source:[260]u8 `mirai:"text"`, // path of the source file, relative to its package name:[64]u8 `mirai:"text"`, // what was taken from the source: a mesh, a material, an image index:i32, // its position among those of its kind in the source source_size:u64, // bytes diff --git a/asset/kinds_test.odin b/asset/kinds_test.odin index eb18ae18a..4c82a8e63 100644 --- a/asset/kinds_test.odin +++ b/asset/kinds_test.odin @@ -43,6 +43,7 @@ inside :: proc(a:^Asset,p:rawptr) -> bool { @(test) mesh_round_trips_its_clusters_compressed :: proc(t:^testing.T) { types:Types;types_init(&types);defer types_destroy(&types) + types.codecs=.Open // what a cook ships: the streams under test vertices,indices:=test_grid(8);defer delete(vertices);defer delete(indices) for &vertex,i in vertices {vertex.material=u32(i/16)} skin:=make([]Skin_Vertex,len(vertices));defer delete(skin) @@ -281,6 +282,7 @@ mesh_refuses_descriptions_and_files_with_ranges_outside_their_tables :: proc(t:^ @(test) texture_round_trips_every_format_and_checks_level_sizes :: proc(t:^testing.T) { types:Types;types_init(&types);defer types_destroy(&types) + types.codecs=.Open // what a cook ships: the streams under test testing.expect_value(t,texture_level_size(.RGBA8,5,3),60) testing.expect_value(t,texture_level_size(.BC1,5,3),16) testing.expect_value(t,texture_level_size(.BC3,8,8),64) @@ -344,6 +346,7 @@ texture_round_trips_every_format_and_checks_level_sizes :: proc(t:^testing.T) { @(test) a_read_of_the_small_levels_decodes_only_the_start_of_the_texels :: proc(t:^testing.T) { types:Types;types_init(&types);defer types_destroy(&types) + types.codecs=.Open // what a cook ships: the streams under test // BC1 whose endpoints change slowly and whose indices are noise: as planes // of fields it compresses smaller. Its levels take several pieces of a stream. SIDE :: 4096 diff --git a/asset/mesh.odin b/asset/mesh.odin index 1f595561a..f816158e7 100644 --- a/asset/mesh.odin +++ b/asset/mesh.odin @@ -486,7 +486,7 @@ mesh_tables :: proc(doc:^Document,types:^Types,info:^Mesh_Info,view:Mesh_View,de // root is not every page as it does one of a file. In allocator, with // scratch from the temporary allocator; false without memory. mesh_pages_pack :: proc(g:^Mesh_Geometry,allocator:=context.allocator) -> bool { - pages,data,error:=pages_compress(g^,.Fast,.Open,allocator) + pages,data,error:=pages_compress(g^,.Fast,.None,allocator) if error!=.None {return false} delete(g.pages,allocator) g.pages,g.page_data=pages,data[:] diff --git a/asset/mesh_page_code.odin b/asset/mesh_page_code.odin index d460fa42f..f0cfdc28b 100644 --- a/asset/mesh_page_code.odin +++ b/asset/mesh_page_code.odin @@ -570,7 +570,8 @@ page_pack :: proc(coder:^Page_Coder,encoder:^Stream_Encoder,g:Mesh_Geometry,page if ok { tried:=at+packed smaller:=pack(encoder,profile,codecs,coded[:size],blocks[tried:]) or_return - if smaller names. +// Its id, where the source was, what the source and its settings are +// (identity) and each asset with its id. Nothing in it depends on the +// version of the engine. The import that ingests the source writes it and +// names it with the ref REF_PACKAGES/. +// - Build: the object of each asset of one build of a package. The import that +// builds it writes it, names it with REF_IMPORTS/ (Final) or +// REF_IMPORTS_FAST/, and the refresh of a project names the one +// it chose with REF_BUILDS/, which a program that has no +// importer finds. +// - Packages: the list of a project (PACKAGES_FILE, beside its manifest), +// versioned: the packages it uses by the hash of their manifest, the folder +// of its assets each is listed in, the assets it uses and what its file of +// each document held when an import or a refresh last wrote it. Beside it, in +// PACKAGE_FILES, the text of each manifest it pins (package_file), so the +// root carries what its pins name: a store without a manifest takes it from +// there. The library of cook/import/sources writes both. +// +// What a parse gives lives in its allocator, an arena: nothing frees it piece +// by piece. A text that is not a record of its version is an error, never a +// default. + +PACKAGES_FILE :: "packages.mirai" +PACKAGE_FILES :: "packages" // the folder of a root, beside PACKAGES_FILE, with .mirai of each manifest it pins +PACKAGES_VERSION :: 2 +PACKAGE_MANIFEST_VERSION :: 1 +BUILD_MANIFEST_VERSION :: 4 +RECORD_BYTES :: 16<<20 // the most a record takes as text: a larger one is refused unread + +// Namespaces of the refs of the records. +REF_PACKAGES :: "packages" // : its manifest, on a machine that ingested it +REF_MANIFESTS :: "manifests" // : its object, what a pin names +REF_IMPORTS :: "imports" // : a build with Final textures, which peers share +REF_IMPORTS_FAST :: "imports-fast" // : a build with Fast textures, which peers share and a cook never takes +REF_BUILDS :: "builds" // : the build the refresh of this machine chose + +REF_CACHE_PREFIX :: "cache-" // and the name of a folder of cook/cache: - +REF_ORIGINS :: "origins" // : the manifest of the package its source was ingested as on this machine +REF_SOURCES :: "sources" // : what the files of that source were when last read, never shared +IMPORT_KEY_VERSIONS :: 32 // the most versions an import key combines + +// Where the source of a package was when it was ingested: a hint to ingest it +// again, not part of what it is. +Package_Origin :: struct { + path:string, // relative to the root whose assets hold it, alike in every copy of it; else absolute + folder:bool, // the source is a folder (an Unreal package), not a file + importer:string, // the format or the importer of a plugin that reads it +} + +Package_Asset :: struct { + kind:Fourcc, + name:string, // in the source: the mesh, the material, the image + index:int, // among those of its kind in the source + id:base.Asset_Id, + part:bool, // a part of the root of the package, decided when it was first ingested + owner:base.Asset_Id, + dependencies:[dynamic]base.Asset_Id, +} + +Package :: struct { + id:base.Asset_Id, + name:string, + origin:Package_Origin, + identity:store.Hash, // of the bytes of the source and its inputs, its settings and the code of the plugin that reads it + settings:string, // the text of the settings object; "" without one + inputs:[dynamic]string, // the files the import read besides the source, from its folder + assets:[dynamic]Package_Asset, +} + +Build_Output :: struct { + id:base.Asset_Id, + kind:Fourcc, + kind_version:u16, + object:store.Hash, // of the asset (object_put) + thumbnail:store.Hash, // an object of its bytes; zero: none +} + +Build :: struct { + package_id:base.Asset_Id, + key:store.Hash, // import_key of what built it + final:bool, // its textures are Final, else Fast + outputs:[dynamic]Build_Output, +} + +Project_Package :: struct { + id:base.Asset_Id, + manifest:store.Hash, + folder:string, // of the assets of the project, where its assets are listed; "": the top + assets:[dynamic]base.Asset_Id, // those the project uses, with what they depend on; none: all + written:[dynamic]Package_Written, +} + +// A document of a package whose file in the project the import or the +// refresh writes over: the content_hash of the header of that file as it last +// wrote it, so a file that holds anything else was changed since. A document +// followed field by field (a material) has none: what a person changed in it +// stays. +Package_Written :: struct { + id:base.Asset_Id, + hash:u64, +} + +Packages :: struct { + packages:[dynamic]Project_Package, +} + +// What a program reads, as one value: the identity of a package and the +// versions of what imports it and writes its assets. Any machine that has the +// manifest computes it, without the source. ok is false with more than +// IMPORT_KEY_VERSIONS versions. +import_key :: proc(identity:store.Hash,versions:[]u64) -> (key:store.Hash,ok:bool) { + TAG :: "mirai import key" + if len(versions)>IMPORT_KEY_VERSIONS {return} + bytes:[len(TAG)+1+size_of(store.Hash)+8*IMPORT_KEY_VERSIONS]u8 + n:=copy(bytes[:],TAG) + bytes[n]=u8(len(versions)) + n+=1 + identity:=identity + n+=copy(bytes[n:],identity[:]) + for version in versions {n+=word_put(bytes[n:],version,8)} + return store.address_of_bytes(bytes[:n]),true +} + +// The name of the build of the manifest with hash manifest that a program +// reads: the version of each kind of the cache its assets are of, as this +// program reads it. The refresh of a project names its choice so; a program +// without importers finds it by the same name. +build_select :: proc(manifest:store.Hash,p:^Package) -> store.Hash { + TAG :: "mirai build select" + bytes:[len(TAG)+size_of(store.Hash)+6*len(KINDS)]u8 + n:=copy(bytes[:],TAG) + manifest:=manifest + n+=copy(bytes[n:],manifest[:]) + for info in KINDS { + if !import_cached(info.kind) {continue} + for &a in p.assets { + if a.kind!=info.kind {continue} + n+=word_put(bytes[n:],u64(info.kind),4) + n+=word_put(bytes[n:],u64(info.version),2) + break + } + } + return store.address_of_bytes(bytes[:n]) +} + +// The namespace of the refs of a folder of cook/cache, in buffer; "" for a +// name that does not fit. +cache_namespace :: proc(name:string,buffer:^[store.NAME_MAX]u8) -> string { + if len(REF_CACHE_PREFIX)+len(name)>store.NAME_MAX {return ""} + n:=copy(buffer[:],REF_CACHE_PREFIX) + n+=copy(buffer[n:],name) + return string(buffer[:n]) +} + +// The name of the refs REF_ORIGINS and REF_SOURCES of the source at origin +// (an absolute path with forward slashes), in buffer. +origin_name :: proc(origin:string,buffer:^[64]u8) -> string { + return store.hex(store.address_of_bytes(transmute([]u8)origin),buffer) +} + +@(private="file") +word_put :: proc(out:[]u8,value:u64,size:int) -> int { + for i in 0..>uint(8*i))} + return size +} + +// The text of a manifest; .Invalid for one that would not read back as it is. +package_text :: proc(p:^Package,allocator:=context.allocator) -> (text:string,error:Error) { + if !package_valid(p) {return "",.Invalid} + w:serial.Writer + serial.writer_init(&w,1024,allocator) + serial.text_write_raw(&w,"{\n \"version\": ") + serial.text_write_int(&w,PACKAGE_MANIFEST_VERSION) + serial.text_write_raw(&w,",\n \"id\": ") + id_write(&w,p.id) + serial.text_write_raw(&w,",\n \"name\": ") + serial.text_write_string(&w,p.name) + serial.text_write_raw(&w,",\n \"origin\": {\"path\": ") + serial.text_write_string(&w,p.origin.path) + serial.text_write_raw(&w,p.origin.folder?", \"folder\": true, \"importer\": ":", \"folder\": false, \"importer\": ") + serial.text_write_string(&w,p.origin.importer) + serial.text_write_raw(&w,"},\n \"identity\": ") + hash_write(&w,p.identity) + if p.settings!="" { + serial.text_write_raw(&w,",\n \"settings\": ") + serial.text_write_raw(&w,p.settings) + } + serial.text_write_raw(&w,",\n \"inputs\": [") + for input,at in p.inputs { + if at>0 {serial.text_write_raw(&w,", ")} + serial.text_write_string(&w,input) + } + serial.text_write_raw(&w,"],\n \"assets\": [") + for &a,at in p.assets { + serial.text_write_raw(&w,at>0?",\n {\"kind\": ":"\n {\"kind\": ") + kind_write(&w,a.kind) + serial.text_write_raw(&w,", \"name\": ") + serial.text_write_string(&w,a.name) + serial.text_write_raw(&w,", \"index\": ") + serial.text_write_int(&w,i64(a.index)) + serial.text_write_raw(&w,", \"id\": ") + id_write(&w,a.id) + if a.part {serial.text_write_raw(&w,", \"part\": true")} + if a.owner!={} { + serial.text_write_raw(&w,", \"owner\": ") + id_write(&w,a.owner) + } + if len(a.dependencies)>0 { + serial.text_write_raw(&w,", \"dependencies\": [") + for d,i in a.dependencies { + if i>0 {serial.text_write_raw(&w,", ")} + id_write(&w,d) + } + serial.text_write_raw(&w,"]") + } + serial.text_write_raw(&w,"}") + } + serial.text_write_raw(&w,len(p.assets)>0?"\n ]\n}\n":"]\n}\n") + return writer_text(&w) +} + +package_parse :: proc(text:string,allocator:=context.allocator) -> (p:Package,error:Error) { + if len(text)>RECORD_BYTES {return {},.Too_Large} + r:serial.Text_Reader + serial.text_reader_init(&r,text,context.temp_allocator) + defer serial.text_reader_destroy(&r) + p.inputs=make([dynamic]string,allocator) + p.assets=make([dynamic]Package_Asset,allocator) + Field :: enum {Version,Id,Name,Origin,Identity,Settings,Inputs,Assets} + seen:bit_set[Field] + version:u64 + expect(&r,'{') or_return + for first:=true;!serial.text_accept(&r,'}');first=false { + if !first {expect(&r,',') or_return} + name:=key(&r) or_return + field:Field + switch name { + case "version": field=.Version + case "id": field=.Id + case "name": field=.Name + case "origin": field=.Origin + case "identity": field=.Identity + case "settings": field=.Settings + case "inputs": field=.Inputs + case "assets": field=.Assets + case: return {},.Corrupt + } + if field in seen {return {},.Corrupt} + seen+={field} + switch field { + case .Version: version=count_read(&r,max(u64)) or_return + case .Id: p.id=id_read(&r) or_return + case .Name: p.name=string_read(&r,allocator) or_return + case .Origin: p.origin=origin_read(&r,allocator) or_return + case .Identity: p.identity=hash_read(&r) or_return + case .Settings: + serial.text_space(&r) + start:=r.offset + if serial.text_peek(&r)!='{' || !serial.text_skip(&r) {return {},.Corrupt} + p.settings=clone(text[start:r.offset],allocator) or_return + case .Inputs: + expect(&r,'[') or_return + for begun:=false;!serial.text_accept(&r,']');begun=true { + if begun {expect(&r,',') or_return} + input:=string_read(&r,allocator) or_return + if _,no_room:=append(&p.inputs,input);no_room!=nil {return {},.Out_Of_Memory} + } + case .Assets: + expect(&r,'[') or_return + for begun:=false;!serial.text_accept(&r,']');begun=true { + if begun {expect(&r,',') or_return} + a:=asset_read(&r,allocator) or_return + if _,no_room:=append(&p.assets,a);no_room!=nil {return {},.Out_Of_Memory} + } + } + } + if !serial.text_end(&r) {return {},.Corrupt} + if version!=PACKAGE_MANIFEST_VERSION {return {},.Version} + if seen!={.Version,.Id,.Name,.Origin,.Identity,.Inputs,.Assets} && seen!={.Version,.Id,.Name,.Origin,.Identity,.Settings,.Inputs,.Assets} {return {},.Corrupt} + if !package_valid(&p) {return {},.Corrupt} + return p,.None +} + +// The text of a build manifest; .Invalid for one that would not read back as it is. +build_text :: proc(b:^Build,allocator:=context.allocator) -> (text:string,error:Error) { + if !build_valid(b) {return "",.Invalid} + w:serial.Writer + serial.writer_init(&w,512,allocator) + serial.text_write_raw(&w,"{\n \"version\": ") + serial.text_write_int(&w,BUILD_MANIFEST_VERSION) + serial.text_write_raw(&w,",\n \"package\": ") + id_write(&w,b.package_id) + serial.text_write_raw(&w,",\n \"key\": ") + hash_write(&w,b.key) + serial.text_write_raw(&w,b.final?",\n \"final\": true":",\n \"final\": false") + serial.text_write_raw(&w,",\n \"outputs\": [") + for o,at in b.outputs { + serial.text_write_raw(&w,at>0?",\n {\"id\": ":"\n {\"id\": ") + id_write(&w,o.id) + serial.text_write_raw(&w,", \"kind\": ") + kind_write(&w,o.kind) + serial.text_write_raw(&w,", \"kind_version\": ") + serial.text_write_int(&w,i64(o.kind_version)) + serial.text_write_raw(&w,", \"object\": ") + hash_write(&w,o.object) + if o.thumbnail!={} { + serial.text_write_raw(&w,", \"thumbnail\": ") + hash_write(&w,o.thumbnail) + } + serial.text_write_raw(&w,"}") + } + serial.text_write_raw(&w,len(b.outputs)>0?"\n ]\n}\n":"]\n}\n") + return writer_text(&w) +} + +build_parse :: proc(text:string,allocator:=context.allocator) -> (b:Build,error:Error) { + if len(text)>RECORD_BYTES {return {},.Too_Large} + r:serial.Text_Reader + serial.text_reader_init(&r,text,context.temp_allocator) + defer serial.text_reader_destroy(&r) + b.outputs=make([dynamic]Build_Output,allocator) + Field :: enum {Version,Package,Key,Final,Outputs} + seen:bit_set[Field] + version:u64 + expect(&r,'{') or_return + for first:=true;!serial.text_accept(&r,'}');first=false { + if !first {expect(&r,',') or_return} + name:=key(&r) or_return + field:Field + switch name { + case "version": field=.Version + case "package": field=.Package + case "key": field=.Key + case "final": field=.Final + case "outputs": field=.Outputs + case: return {},.Corrupt + } + if field in seen {return {},.Corrupt} + seen+={field} + switch field { + case .Version: version=count_read(&r,max(u64)) or_return + case .Package: b.package_id=id_read(&r) or_return + case .Key: b.key=hash_read(&r) or_return + case .Final: b.final=bool_read(&r) or_return + case .Outputs: + expect(&r,'[') or_return + for begun:=false;!serial.text_accept(&r,']');begun=true { + if begun {expect(&r,',') or_return} + o:=output_read(&r) or_return + if _,no_room:=append(&b.outputs,o);no_room!=nil {return {},.Out_Of_Memory} + } + } + } + if !serial.text_end(&r) {return {},.Corrupt} + if version!=BUILD_MANIFEST_VERSION {return {},.Version} + if seen!={.Version,.Package,.Key,.Final,.Outputs} {return {},.Corrupt} + if !build_valid(&b) {return {},.Corrupt} + return b,.None +} + +// The text of the list of a project; .Invalid for one that would not read back as it is. +packages_text :: proc(list:^Packages,allocator:=context.allocator) -> (text:string,error:Error) { + if !packages_valid(list) {return "",.Invalid} + w:serial.Writer + serial.writer_init(&w,256,allocator) + serial.text_write_raw(&w,"{\n \"version\": ") + serial.text_write_int(&w,PACKAGES_VERSION) + serial.text_write_raw(&w,",\n \"packages\": [") + for &entry,at in list.packages { + serial.text_write_raw(&w,at>0?",\n {\"id\": ":"\n {\"id\": ") + id_write(&w,entry.id) + serial.text_write_raw(&w,", \"manifest\": ") + hash_write(&w,entry.manifest) + serial.text_write_raw(&w,", \"folder\": ") + serial.text_write_string(&w,entry.folder) + serial.text_write_raw(&w,", \"assets\": [") + for id,i in entry.assets { + if i>0 {serial.text_write_raw(&w,", ")} + id_write(&w,id) + } + serial.text_write_raw(&w,"], \"written\": [") + for written,i in entry.written { + serial.text_write_raw(&w,i>0?", {\"id\": ":"{\"id\": ") + id_write(&w,written.id) + serial.text_write_raw(&w,", \"hash\": \"") + serial.text_write_hex(&w,written.hash,16,false) + serial.text_write_raw(&w,"\"}") + } + serial.text_write_raw(&w,"]}") + } + serial.text_write_raw(&w,len(list.packages)>0?"\n ]\n}\n":"]\n}\n") + return writer_text(&w) +} + +packages_parse :: proc(text:string,allocator:=context.allocator) -> (list:Packages,error:Error) { + if len(text)>RECORD_BYTES {return {},.Too_Large} + r:serial.Text_Reader + serial.text_reader_init(&r,text,context.temp_allocator) + defer serial.text_reader_destroy(&r) + list.packages=make([dynamic]Project_Package,allocator) + Field :: enum {Version,Packages} + seen:bit_set[Field] + version:u64 + expect(&r,'{') or_return + for first:=true;!serial.text_accept(&r,'}');first=false { + if !first {expect(&r,',') or_return} + name:=key(&r) or_return + field:Field + switch name { + case "version": field=.Version + case "packages": field=.Packages + case: return {},.Corrupt + } + if field in seen {return {},.Corrupt} + seen+={field} + switch field { + case .Version: version=count_read(&r,max(u64)) or_return + case .Packages: + expect(&r,'[') or_return + for begun:=false;!serial.text_accept(&r,']');begun=true { + if begun {expect(&r,',') or_return} + entry:=entry_read(&r,allocator) or_return + if _,no_room:=append(&list.packages,entry);no_room!=nil {return {},.Out_Of_Memory} + } + } + } + if !serial.text_end(&r) {return {},.Corrupt} + if version!=PACKAGES_VERSION {return {},.Version} + if seen!={.Version,.Packages} || !packages_valid(&list) {return {},.Corrupt} + return list,.None +} + +// The list of the project at root, in allocator. found is false, with an +// empty list, when the project has none. +packages_load :: proc(root:string,allocator:=context.allocator) -> (list:Packages,found:bool,error:Error) { + path:=fmt.tprintf("%s/%s",root,PACKAGES_FILE) + data,read_error:=serial.file_read(path,context.temp_allocator,limit=RECORD_BYTES) + if read_error==.Not_Found { + list.packages=make([dynamic]Project_Package,allocator) + return list,false,.None + } + if read_error!=.None {return {},false,file_error(read_error)} + list,error=packages_parse(string(data),allocator) + return list,error==.None,error +} + +// Writes the list of the project at root, unless its file already says the same. +packages_save :: proc(root:string,list:^Packages) -> Error { + text:=packages_text(list,context.temp_allocator) or_return + path:=fmt.tprintf("%s/%s",root,PACKAGES_FILE) + _,write_error:=serial.file_write_changed(path,transmute([]u8)text) + return file_error(write_error) +} + +// The file of root that holds the text of the manifest of the package id it +// pins, in the temp allocator. +package_file :: proc(root:string,id:base.Asset_Id) -> string { + buffer:[base.ASSET_ID_TEXT]u8 + return fmt.tprintf("%s/%s/%s.mirai",root,PACKAGE_FILES,base.asset_id_format(id,buffer[:])) +} + +// Writes the manifest the store keeps under entry.manifest to the file of +// root for its package, unless the file already holds it. .Corrupt when that +// object is not a manifest of the package entry names. +package_file_save :: proc(s:^store.Store,root:string,entry:^Project_Package) -> Error { + text:=manifest_get(s,entry.manifest) or_return + p:=package_parse(text,context.temp_allocator) or_return + if p.id!=entry.id {return .Corrupt} + path:=package_file(root,entry.id) + if folder:=fmt.tprintf("%s/%s",root,PACKAGE_FILES);!os.is_dir(folder) && os.make_directory_all(folder)!=nil {return .IO} + _,write_error:=serial.file_write_changed(path,transmute([]u8)text) + return file_error(write_error) +} + +// Removes the file of root for the package id; .None also when it has none. +package_file_remove :: proc(root:string,id:base.Asset_Id) -> Error { + path:=package_file(root,id) + if !os.exists(path) {return .None} + return .None if os.remove(path)==nil else .IO +} + +// The manifest a root pins with entry: the one the store keeps, else the file +// of root for it (package_file), which then goes into the store. A file that +// is there must hold the bytes of the address entry pins: .Hash when it does +// not, also when the store has the manifest. .Not_Found when neither has it. +// root "": the store alone. In allocator, an arena. +package_load :: proc(s:^store.Store,root:string,entry:^Project_Package,allocator:=context.allocator) -> (p:Package,error:Error) { + file:[]u8 + kept:=false + if root!="" { + read_error:serial.File_Error + file,read_error=serial.file_read(package_file(root,entry.id),context.temp_allocator,limit=RECORD_BYTES) + #partial switch read_error { + case .None: + if store.address_of_bytes(file)!=entry.manifest {return {},.Hash} + kept=true + case .Not_Found: + case: return {},file_error(read_error) + } + } + p,error=package_get(s,entry.manifest,allocator) + if error==.Not_Found && kept { + p,error=package_parse(string(file),allocator) + if error!=.None {return {},error} + manifest_put(s,file) or_return + } + if error==.None && p.id!=entry.id {return {},.Corrupt} + return +} + +// Keeps a manifest in the store: hash, what pins it, is the BLAKE3 of its text. +package_put :: proc(s:^store.Store,p:^Package) -> (hash:store.Hash,error:Error) { + text:=package_text(p,context.temp_allocator) or_return + return manifest_put(s,transmute([]u8)text) +} + +// The manifest the store keeps under the hash of its text, in allocator. .Not_Found without it. +package_get :: proc(s:^store.Store,hash:store.Hash,allocator:=context.allocator) -> (p:Package,error:Error) { + text:=manifest_get(s,hash) or_return + return package_parse(text,allocator) +} + +// Keeps the text of a manifest as an object named by the ref REF_MANIFESTS/, committed. +@(private="file") +manifest_put :: proc(s:^store.Store,text:[]u8) -> (hash:store.Hash,error:Error) { + hash=store.address_of_bytes(text) + object:=record_put(s,text) or_return + digits:[64]u8 + return hash,store_error(store.ref_set(s,REF_MANIFESTS,store.hex(hash,&digits),object)) +} + +// The text of the manifest pinned by hash, only when it still hashes to it, in the temp allocator. +@(private="file") +manifest_get :: proc(s:^store.Store,hash:store.Hash) -> (text:string,error:Error) { + digits:[64]u8 + object,ref_error:=store.ref_get(s,REF_MANIFESTS,store.hex(hash,&digits)) + if ref_error!=.None {return "",store_error(ref_error)} + text=record_get(s,object) or_return + if store.address_of_bytes(transmute([]u8)text)!=hash {return "",.Hash} + return text,.None +} + +// Keeps a build in the store: the address of its object, which refs name. +build_put :: proc(s:^store.Store,b:^Build) -> (object:store.Hash,error:Error) { + text:=build_text(b,context.temp_allocator) or_return + return record_put(s,transmute([]u8)text) +} + +// The root of a package: the prefab of a model or the scene of a source a +// plugin imports whole, else its one asset that is no part of another (the +// texture of an image, the sound of a sound file). Zero when it has none. +package_root :: proc(p:^Package) -> base.Asset_Id { + for &a in p.assets { + if a.kind==KIND_SCENE && a.owner=={} {return a.id} + } + root:base.Asset_Id + for &a in p.assets { + if a.owner!={} || a.part {continue} + if root!={} {return {}} + root=a.id + } + return root +} + +// The object of a build its package shows: that of its root (package_root) +// when the build holds it, else of its first mesh, else of its first asset; +// a model's prefab is a document of the project, not of its build. Zero: none. +build_shown :: proc(b:^Build,p:^Package) -> (object:store.Hash,id:base.Asset_Id) { + root:=package_root(p) + for &o in b.outputs { + if o.id==root {return o.object,o.id} + } + for &o in b.outputs { + if o.kind==KIND_MESH {return o.object,o.id} + } + if len(b.outputs)==0 {return} + return b.outputs[0].object,b.outputs[0].id +} + +build_get :: proc(s:^store.Store,object:store.Hash,allocator:=context.allocator) -> (b:Build,error:Error) { + text:=record_get(s,object) or_return + return build_parse(text,allocator) +} + +// Puts the text of a record as an object, without committing: the ref that names it commits it. +record_put :: proc(s:^store.Store,text:[]u8) -> (object:store.Hash,error:Error) { + if len(text)>RECORD_BYTES {return {},.Too_Large} + put_error:store.Error + object,put_error=store.put(s,text) + return object,store_error(put_error) +} + +Listed_State :: enum u8 { + Built, + Not_Built, // no build this program reads: a machine with the source, or a peer that built it, gives it +} + +// An asset of the cache a project lists from its packages. +Package_Listed :: struct { + path:string, // in the registry: /, as listed_path makes it + object:store.Hash, // the object of the store that holds it; zero when not built + id:base.Asset_Id, + kind:Fourcc, + package_id:base.Asset_Id, + state:Listed_State, + header:Header, // of the object when built: zero when it does not read + dependencies:[]base.Asset_Id, // what its header and directory name, when built +} + +Package_Problem_Code :: enum u8 { + Manifest_Missing, // neither the store nor the root has the manifest the project names + Manifest_Unreadable, // the one there is not a manifest of this version, or of another package + Manifest_Mismatch, // the file of the root for it (package_file) is not the manifest it pins + Build_Unreadable, // the build its ref names is missing or does not read: its assets are not built + Asset_Unknown, // the project uses an asset its manifest does not have +} + +Package_Problem :: struct { + package_id:base.Asset_Id, + asset:base.Asset_Id, // Asset_Unknown: the one the project names + code:Package_Problem_Code, +} + +Packages_Resolved :: struct { + listed:[dynamic]Package_Listed, + problems:[dynamic]Package_Problem, + packages:[dynamic]Resolved_Package, // each one whose manifest read, in the order of the list +} + +// A package a root uses, by what its manifest says of it. +Resolved_Package :: struct { + id:base.Asset_Id, + name:string, + origin:Package_Origin, + inputs:[]string, // the files the import read besides the source, from its folder +} + +// What a project lists from the packages it uses: each asset of the cache it +// uses, with what they depend on, at its path, and the object of the build +// REF_BUILDS names for its manifest, or Not_Built. Documents are files of the +// project and are not listed. The manifests come from the store or from the +// files of root (package_load; "": the store alone). A manifest or a build +// that is not there or does not read is a problem of the result, never an +// error; nothing is removed, built or fetched. Two packages that list one path take the name of +// their package after it, " ()". In allocator, an arena; error +// only when memory runs out. +packages_resolve :: proc(s:^store.Store,root:string,list:^Packages,allocator:=context.allocator) -> (resolved:Packages_Resolved,error:Error) { + resolved.listed=make([dynamic]Package_Listed,allocator) + resolved.problems=make([dynamic]Package_Problem,allocator) + resolved.packages=make([dynamic]Resolved_Package,allocator) + names:=make([dynamic]string,context.temp_allocator) + for &entry in list.packages { + p,manifest_error:=package_load(s,root,&entry,context.temp_allocator) + #partial switch manifest_error { + case .None: + case .Out_Of_Memory: return {},.Out_Of_Memory + case .Not_Found: problem_add(&resolved,{entry.id,{},.Manifest_Missing}) or_return + case .Hash: problem_add(&resolved,{entry.id,{},.Manifest_Mismatch}) or_return + case: problem_add(&resolved,{entry.id,{},.Manifest_Unreadable}) or_return + } + if manifest_error!=.None {continue} + named:=Resolved_Package{id=p.id,origin={folder=p.origin.folder}} + named.name=clone(p.name,allocator) or_return + named.origin.path=clone(p.origin.path,allocator) or_return + named.origin.importer=clone(p.origin.importer,allocator) or_return + inputs,no_inputs:=make([]string,len(p.inputs),allocator) + if no_inputs!=nil {return {},.Out_Of_Memory} + for input,at in p.inputs {inputs[at]=clone(input,allocator) or_return} + named.inputs=inputs + if _,no_room:=append(&resolved.packages,named);no_room!=nil {return {},.Out_Of_Memory} + used,no_used:=make([]bool,len(p.assets),context.temp_allocator) + by_id:=make(map[base.Asset_Id]int,context.temp_allocator) + if no_used!=nil || reserve(&by_id,len(p.assets))!=nil {return {},.Out_Of_Memory} + for &a,at in p.assets {by_id[a.id]=at} + if len(entry.assets)==0 { + for &u in used {u=true} + } else { + pending:=make([dynamic]int,context.temp_allocator) + for id in entry.assets { + at,known:=by_id[id] + if !known { + problem_add(&resolved,{entry.id,id,.Asset_Unknown}) or_return + continue + } + if _,no_room:=append(&pending,at);no_room!=nil {return {},.Out_Of_Memory} + } + for len(pending)>0 { + at:=pop(&pending) + if used[at] {continue} + used[at]=true + for d in p.assets[at].dependencies { + if next,known:=by_id[d];known && !used[next] { + if _,no_room:=append(&pending,next);no_room!=nil {return {},.Out_Of_Memory} + } + } + } + } + outputs:=make(map[base.Asset_Id]Build_Output,context.temp_allocator) + digits:[64]u8 + built,ref_error:=store.ref_get(s,REF_BUILDS,store.hex(build_select(entry.manifest,&p),&digits)) + switch ref_error { + case .None: + b,build_error:=build_get(s,built,context.temp_allocator) + if build_error==.Out_Of_Memory {return {},.Out_Of_Memory} + if build_error==.None && b.package_id!=p.id {build_error=.Corrupt} + if build_error!=.None { + problem_add(&resolved,{entry.id,{},.Build_Unreadable}) or_return + break + } + if reserve(&outputs,len(b.outputs))!=nil {return {},.Out_Of_Memory} + for o in b.outputs {outputs[o.id]=o} + case .Missing: + case .Out_Of_Memory: return {},.Out_Of_Memory + case .Corrupt,.Invalid_Argument,.Io,.No_Space,.Unsupported_Layout: + problem_add(&resolved,{entry.id,{},.Build_Unreadable}) or_return + } + folder:=entry.folder!=""?fmt.tprintf("%s/",entry.folder):"" + for &a,at in p.assets { + info,known:=kind_info(a.kind) + if !used[at] || !known || !import_cached(a.kind) {continue} + repeated:=false + for &other in p.assets {repeated||=other.kind==a.kind && other.name==a.name && other.index +// with the name as safe_path gives it, and " ()" before the extension +// when an output of its kind and name has a lower index (repeated). folder: +// "" or ending with '/'. In the temp allocator. +@(private) +listed_path :: proc(folder,name:string,index:int,repeated:bool,extension:string) -> string { + if repeated {return fmt.tprintf("%s%s (%d)%s",folder,safe_path(name),index,extension)} + return fmt.tprintf("%s%s%s",folder,safe_path(name),extension) +} + +// A folder under another one: relative, with forward slashes, never out of it. +@(private) +folder_valid :: proc(folder:string) -> bool { + if folder=="" {return true} + if folder[0]=='/' || strings.contains_any(folder,"\\:") {return false} + rest:=folder + for part in strings.split_iterator(&rest,"/") { + if part=="" || part=="." || part==".." {return false} + } + return true +} + +@(private="file") +package_valid :: proc(p:^Package) -> bool { + if p.id=={} || p.settings!="" && !object_valid(p.settings) {return false} + for &a,at in p.assets { + if a.id=={} || a.id==p.id || !kind_valid(a.kind) || a.index<0 || a.index>int(max(i32)) || a.part && !import_cached(a.kind) {return false} + for &other in p.assets[:at] { + if other.id==a.id || other.kind==a.kind && other.index==a.index && other.name==a.name {return false} + } + for d in a.dependencies { + if d=={} {return false} + } + } + return true +} + +@(private="file") +build_valid :: proc(b:^Build) -> bool { + if b.package_id=={} {return false} + for o,at in b.outputs { + if o.id=={} || !kind_valid(o.kind) {return false} + for other in b.outputs[:at] { + if other.id==o.id {return false} + } + } + return true +} + +@(private="file") +packages_valid :: proc(list:^Packages) -> bool { + for &entry,at in list.packages { + if entry.id=={} || !folder_valid(entry.folder) {return false} + for &other in list.packages[:at] { + if other.id==entry.id {return false} + } + for id,i in entry.assets { + if id=={} {return false} + for other in entry.assets[:i] { + if other==id {return false} + } + } + for written,i in entry.written { + if written.id=={} || written.hash==0 {return false} + for other in entry.written[:i] { + if other.id==written.id {return false} + } + } + } + return true +} + +// Four bytes a person can read, not all spaces: what the text of a kind keeps. +@(private="file") +kind_valid :: proc(kind:Fourcc) -> bool { + if kind==fourcc(" ") {return false} + for i in 0..<4 { + c:=u8(u32(kind)>>uint(8*i)) + if c<0x20 || c>0x7e {return false} + } + return true +} + +// One JSON object and nothing after it. +@(private="file") +object_valid :: proc(text:string) -> bool { + r:serial.Text_Reader + serial.text_reader_init(&r,text,context.temp_allocator) + defer serial.text_reader_destroy(&r) + return serial.text_peek(&r)=='{' && serial.text_skip(&r) && serial.text_end(&r) +} + +@(private="file") +writer_text :: proc(w:^serial.Writer) -> (text:string,error:Error) { + if w.overflow>0 { + serial.writer_destroy(w) + return "",.Out_Of_Memory + } + return string(serial.writer_take(w)),.None +} + +@(private="file") +id_write :: proc(w:^serial.Writer,id:base.Asset_Id) { + serial.text_write_raw(w,"\"") + serial.text_write_asset_id(w,id) + serial.text_write_raw(w,"\"") +} + +@(private="file") +hash_write :: proc(w:^serial.Writer,hash:store.Hash) { + digits:[64]u8 + serial.text_write_raw(w,"\"") + serial.text_write_raw(w,store.hex(hash,&digits)) + serial.text_write_raw(w,"\"") +} + +@(private="file") +kind_write :: proc(w:^serial.Writer,kind:Fourcc) { + buffer:[4]u8 + serial.text_write_string(w,strings.trim_right(fourcc_text(kind,&buffer)," ")) +} + +@(private="file") +expect :: proc(r:^serial.Text_Reader,c:u8) -> Error { + return .None if serial.text_expect(r,c) else .Corrupt +} + +// The name of a field and its colon: a view valid until the next string. +@(private="file") +key :: proc(r:^serial.Text_Reader) -> (name:string,error:Error) { + text,ok:=serial.text_string(r) + if !ok || !serial.text_expect(r,':') {return "",.Corrupt} + return text,.None +} + +@(private="file") +clone :: proc(text:string,allocator:runtime.Allocator) -> (cloned:string,error:Error) { + if text=="" {return "",.None} + copied,no_room:=strings.clone(text,allocator) + if no_room!=nil {return "",.Out_Of_Memory} + return copied,.None +} + +@(private="file") +string_read :: proc(r:^serial.Text_Reader,allocator:runtime.Allocator) -> (text:string,error:Error) { + value,ok:=serial.text_string(r) + if !ok {return "",.Corrupt} + return clone(value,allocator) +} + +@(private="file") +count_read :: proc(r:^serial.Text_Reader,most:u64) -> (value:u64,error:Error) { + negative,magnitude,ok:=serial.text_integer(r) + if !ok || negative || magnitude>most {return 0,.Corrupt} + return magnitude,.None +} + +@(private="file") +bool_read :: proc(r:^serial.Text_Reader) -> (value:bool,error:Error) { + if serial.text_keyword(r,"true") {return true,.None} + if serial.text_keyword(r,"false") {return false,.None} + return false,.Corrupt +} + +@(private="file") +id_read :: proc(r:^serial.Text_Reader) -> (id:base.Asset_Id,error:Error) { + text,ok:=serial.text_string(r) + if !ok {return {},.Corrupt} + parsed,valid:=base.asset_id_parse(text) + if !valid {return {},.Corrupt} + return parsed,.None +} + +@(private="file") +hash_read :: proc(r:^serial.Text_Reader) -> (hash:store.Hash,error:Error) { + text,ok:=serial.text_string(r) + if !ok {return {},.Corrupt} + parsed,valid:=store.parse_hex(text) + if !valid {return {},.Corrupt} + return parsed,.None +} + +@(private="file") +kind_read :: proc(r:^serial.Text_Reader) -> (kind:Fourcc,error:Error) { + text,ok:=serial.text_string(r) + if !ok || len(text)==0 || len(text)>4 {return 0,.Corrupt} + kind=fourcc(text) + if !kind_valid(kind) {return 0,.Corrupt} + return kind,.None +} + +@(private="file") +ids_read :: proc(r:^serial.Text_Reader,ids:^[dynamic]base.Asset_Id) -> Error { + expect(r,'[') or_return + for begun:=false;!serial.text_accept(r,']');begun=true { + if begun {expect(r,',') or_return} + id:=id_read(r) or_return + if _,no_room:=append(ids,id);no_room!=nil {return .Out_Of_Memory} + } + return .None +} + +@(private="file") +origin_read :: proc(r:^serial.Text_Reader,allocator:runtime.Allocator) -> (origin:Package_Origin,error:Error) { + Field :: enum {Path,Folder,Importer} + seen:bit_set[Field] + expect(r,'{') or_return + for first:=true;!serial.text_accept(r,'}');first=false { + if !first {expect(r,',') or_return} + name:=key(r) or_return + field:Field + switch name { + case "path": field=.Path + case "folder": field=.Folder + case "importer": field=.Importer + case: return {},.Corrupt + } + if field in seen {return {},.Corrupt} + seen+={field} + switch field { + case .Path: origin.path=string_read(r,allocator) or_return + case .Folder: origin.folder=bool_read(r) or_return + case .Importer: origin.importer=string_read(r,allocator) or_return + } + } + if seen!={.Path,.Folder,.Importer} {return {},.Corrupt} + return origin,.None +} + +@(private="file") +asset_read :: proc(r:^serial.Text_Reader,allocator:runtime.Allocator) -> (a:Package_Asset,error:Error) { + Field :: enum {Kind,Name,Index,Id,Part,Owner,Dependencies} + seen:bit_set[Field] + a.dependencies=make([dynamic]base.Asset_Id,allocator) + expect(r,'{') or_return + for first:=true;!serial.text_accept(r,'}');first=false { + if !first {expect(r,',') or_return} + name:=key(r) or_return + field:Field + switch name { + case "kind": field=.Kind + case "name": field=.Name + case "index": field=.Index + case "id": field=.Id + case "part": field=.Part + case "owner": field=.Owner + case "dependencies": field=.Dependencies + case: return {},.Corrupt + } + if field in seen {return {},.Corrupt} + seen+={field} + switch field { + case .Kind: a.kind=kind_read(r) or_return + case .Name: a.name=string_read(r,allocator) or_return + case .Index: a.index=int(count_read(r,u64(max(i32))) or_return) + case .Id: a.id=id_read(r) or_return + case .Part: a.part=bool_read(r) or_return + case .Owner: + a.owner=id_read(r) or_return + if a.owner=={} {return {},.Corrupt} + case .Dependencies: + ids_read(r,&a.dependencies) or_return + if len(a.dependencies)==0 {return {},.Corrupt} + } + } + if seen&{.Kind,.Name,.Index,.Id}!={.Kind,.Name,.Index,.Id} || .Part in seen && !a.part {return {},.Corrupt} + return a,.None +} + +@(private="file") +output_read :: proc(r:^serial.Text_Reader) -> (o:Build_Output,error:Error) { + Field :: enum {Id,Kind,Kind_Version,Object,Thumbnail} + seen:bit_set[Field] + expect(r,'{') or_return + for first:=true;!serial.text_accept(r,'}');first=false { + if !first {expect(r,',') or_return} + name:=key(r) or_return + field:Field + switch name { + case "id": field=.Id + case "kind": field=.Kind + case "kind_version": field=.Kind_Version + case "object": field=.Object + case "thumbnail": field=.Thumbnail + case: return {},.Corrupt + } + if field in seen {return {},.Corrupt} + seen+={field} + switch field { + case .Id: o.id=id_read(r) or_return + case .Kind: o.kind=kind_read(r) or_return + case .Kind_Version: o.kind_version=u16(count_read(r,u64(max(u16))) or_return) + case .Object: o.object=hash_read(r) or_return + case .Thumbnail: + o.thumbnail=hash_read(r) or_return + if o.thumbnail=={} {return {},.Corrupt} + } + } + if seen&{.Id,.Kind,.Kind_Version,.Object}!={.Id,.Kind,.Kind_Version,.Object} {return {},.Corrupt} + return o,.None +} + +@(private="file") +entry_read :: proc(r:^serial.Text_Reader,allocator:runtime.Allocator) -> (entry:Project_Package,error:Error) { + Field :: enum {Id,Manifest,Folder,Assets,Written} + seen:bit_set[Field] + entry.assets=make([dynamic]base.Asset_Id,allocator) + entry.written=make([dynamic]Package_Written,allocator) + expect(r,'{') or_return + for first:=true;!serial.text_accept(r,'}');first=false { + if !first {expect(r,',') or_return} + name:=key(r) or_return + field:Field + switch name { + case "id": field=.Id + case "manifest": field=.Manifest + case "folder": field=.Folder + case "assets": field=.Assets + case "written": field=.Written + case: return {},.Corrupt + } + if field in seen {return {},.Corrupt} + seen+={field} + switch field { + case .Id: entry.id=id_read(r) or_return + case .Manifest: entry.manifest=hash_read(r) or_return + case .Folder: entry.folder=string_read(r,allocator) or_return + case .Assets: ids_read(r,&entry.assets) or_return + case .Written: + expect(r,'[') or_return + for begun:=false;!serial.text_accept(r,']');begun=true { + if begun {expect(r,',') or_return} + written:=written_read(r) or_return + if _,no_room:=append(&entry.written,written);no_room!=nil {return {},.Out_Of_Memory} + } + } + } + if seen!={.Id,.Manifest,.Folder,.Assets,.Written} {return {},.Corrupt} + return entry,.None +} + +@(private="file") +written_read :: proc(r:^serial.Text_Reader) -> (written:Package_Written,error:Error) { + Field :: enum {Id,Hash} + seen:bit_set[Field] + expect(r,'{') or_return + for first:=true;!serial.text_accept(r,'}');first=false { + if !first {expect(r,',') or_return} + name:=key(r) or_return + field:Field + switch name { + case "id": field=.Id + case "hash": field=.Hash + case: return {},.Corrupt + } + if field in seen {return {},.Corrupt} + seen+={field} + switch field { + case .Id: written.id=id_read(r) or_return + case .Hash: + digits,ok:=serial.text_string(r) + if !ok || len(digits)!=16 {return {},.Corrupt} + parsed,parsed_ok:=base.hex_parse_u64(digits) + if !parsed_ok {return {},.Corrupt} + written.hash=parsed + } + } + if seen!={.Id,.Hash} {return {},.Corrupt} + return written,.None +} + +// The text of the record the store keeps as the object at address, in the temp allocator. +record_get :: proc(s:^store.Store,address:store.Hash) -> (text:string,error:Error) { + o,open_error:=store.object_open(s,address) + if open_error!=.None {return "",store_error(open_error)} + defer store.object_close(&o) + size:=store.object_size(&o) + if size>RECORD_BYTES {return "",.Too_Large} + bytes,no_room:=make([]u8,int(size),context.temp_allocator) + if no_room!=nil {return "",.Out_Of_Memory} + if read_error:=store.object_read(&o,0,bytes);read_error!=.None {return "",store_error(read_error)} + return string(bytes),.None +} + +@(private="file") +file_error :: proc(error:serial.File_Error) -> Error { + switch error { + case .None: return .None + case .Too_Large: return .Too_Large + case .Out_Of_Memory: return .Out_Of_Memory + case .Not_Found: return .Not_Found + case .Open,.Read,.Write,.Rename,.Invalid_Alignment: return .IO + } + return .IO +} + +@(private) +store_error :: proc(error:store.Error) -> Error { + switch error { + case .None: return .None + case .Missing: return .Not_Found + case .Corrupt,.Unsupported_Layout: return .Corrupt + case .Invalid_Argument: return .Invalid + case .Out_Of_Memory: return .Out_Of_Memory + case .Io,.No_Space: return .IO + } + return .IO +} + +@(private="file") +problem_add :: proc(resolved:^Packages_Resolved,problem:Package_Problem) -> Error { + if _,no_room:=append(&resolved.problems,problem);no_room!=nil {return .Out_Of_Memory} + return .None +} + +// Whether what an import builds of a kind is of the cache: nothing edits it +// after the import, so its build gives it back whole. +import_cached :: proc "contextless" (kind:Fourcc) -> bool { + return kind==KIND_MESH || kind==KIND_TEXTURE || kind==KIND_SOUND || kind==KIND_FONT +} + +// A name a file can have: what no file system takes becomes _, and the +// spaces and dots of its ends go. "unnamed" when nothing is left. In the +// temp allocator. +safe_name :: proc(name:string) -> string { + cleaned:=name_clean(name) + return cleaned if cleaned!="" else "unnamed" +} + +// Where under the folder of an import the output named name goes: each part +// between '/' as safe_name makes it, without the parts that are empty, "." +// or "..". "unnamed" when no part is left. In the temp allocator. +safe_path :: proc(name:string) -> string { + b:=strings.builder_make(context.temp_allocator) + rest:=name + for part in strings.split_iterator(&rest,"/") { + cleaned:=name_clean(part) + if cleaned=="" {continue} + if strings.builder_len(b)>0 {strings.write_byte(&b,'/')} + strings.write_string(&b,cleaned) + } + return strings.to_string(b) if strings.builder_len(b)>0 else "unnamed" +} + +@(private="file") +name_clean :: proc(name:string) -> string { + b:=strings.builder_make(context.temp_allocator) + for r in name { + switch { + case r<32 || r==127,r=='<',r=='>',r==':',r=='"',r=='/',r=='\\',r=='|',r=='?',r=='*',r=='#': + strings.write_byte(&b,'_') + case: + strings.write_rune(&b,r) + } + } + return strings.trim(strings.to_string(b)," .") +} diff --git a/asset/packages_root.odin b/asset/packages_root.odin new file mode 100644 index 000000000..d669c26bc --- /dev/null +++ b/asset/packages_root.odin @@ -0,0 +1,125 @@ +package asset + +import "core:fmt" +import "core:strings" +import "mirai:std/base" +import "mirai:store" + +// What a root lists from the packages it uses: a project, or a plugin, which +// keeps its packages as a project does. Every program that sets a registry up +// for a root that is not shipped gives it these entries. +Root_Packages :: struct { + listed:[]Package_Listed, // for registry_init or registry_mount of the root + packages:[]Resolved_Package, // whose manifest read: their names and origins + problems:[]string, // one line per package it cannot list whole: which one, and how to fix it +} + +// packages_resolve of the list at root (PACKAGES_FILE) against the store at +// store_root, the machine's (store.default_root) when "", which takes the +// manifests it lacks from the files of the root. A root without the +// list lists nothing and opens no store. A list or a store that does not read +// is a problem, as is a package that is not here or not built here: none is an +// error, which is only that memory ran out. In allocator, an arena. +root_packages :: proc(root,store_root:string,allocator:=context.allocator) -> (resolved:Root_Packages,error:Error) { + list,problems,listed:=packages_list(root,allocator) or_return + if !listed {return {problems=problems[:]},.None} + folder:=store_root!=""?store_root:store.default_root(context.temp_allocator) + if folder=="" { + problem_line(&problems,"this machine has no folder for the store of Mirai, so the packages of %s are not listed",root) or_return + return {problems=problems[:]},.None + } + s,open_error:=store.open(folder,context.temp_allocator) + switch open_error { + case .None: + case .Out_Of_Memory: return {},.Out_Of_Memory + case .Unsupported_Layout: + problem_line(&problems,"the store at %s was written by another version of Mirai, so the packages of %s are not listed: remove that folder and build again, which fills it",folder,root) or_return + return {problems=problems[:]},.None + case .Missing,.Corrupt,.Invalid_Argument,.Io,.No_Space: + problem_line(&problems,"the store at %s does not open (%v), so the packages of %s are not listed",folder,open_error,root) or_return + return {problems=problems[:]},.None + } + defer store.close(&s) + return packages_listed(&s,root,&list,problems,allocator) +} + +// root_packages against the store s, already open; a root without the list +// lists nothing. In allocator, an arena. +root_packages_in :: proc(s:^store.Store,root:string,allocator:=context.allocator) -> (resolved:Root_Packages,error:Error) { + list,problems,listed:=packages_list(root,allocator) or_return + if !listed {return {problems=problems[:]},.None} + return packages_listed(s,root,&list,problems,allocator) +} + +// The list at root, with a problem when it does not read; listed: it names packages. +@(private="file") +packages_list :: proc(root:string,allocator:=context.allocator) -> (list:Packages,problems:[dynamic]string,listed:bool,error:Error) { + problems=make([dynamic]string,allocator) + load_error:Error + found:bool + list,found,load_error=packages_load(root,context.temp_allocator) + switch load_error { + case .None: + case .Out_Of_Memory: return {},problems,false,.Out_Of_Memory + case .Truncated,.Not_An_Asset,.Version,.Corrupt,.Hash,.Too_Large,.Schema,.Not_Found,.IO,.Invalid,.Codec_Missing,.Not_Here: + problem_line(&problems,"%s/%s does not read (%v), so its packages are not listed: restore it from version control",root,PACKAGES_FILE,load_error) or_return + return + } + return list,problems,found && len(list.packages)>0,.None +} + +@(private="file") +packages_listed :: proc(s:^store.Store,root:string,list:^Packages,problems:[dynamic]string,allocator:=context.allocator) -> (resolved:Root_Packages,error:Error) { + problems:=problems + folder:=s.root + packages:=packages_resolve(s,root,list,allocator) or_return + unreadable:=make(map[base.Asset_Id]bool,context.temp_allocator) + for problem in packages.problems { + named:=package_named(s,list,problem.package_id) + switch problem.code { + case .Manifest_Missing: + problem_line(&problems,"%s is neither in the store at %s nor in %s: restore that file from version control, or fetch the manifest from a peer that has it",named,folder,package_file(root,problem.package_id)) or_return + case .Manifest_Unreadable: + problem_line(&problems,"the manifest of %s is not one this engine reads: ingest its source again",named) or_return + case .Manifest_Mismatch: + problem_line(&problems,"%s is not the manifest %s/%s pins for %s: restore both from version control, or ingest its source again",package_file(root,problem.package_id),root,PACKAGES_FILE,named) or_return + case .Build_Unreadable: + unreadable[problem.package_id]=true + problem_line(&problems,"the build of %s on this machine does not read, so its assets are not built: refresh it on a machine with its source",named) or_return + case .Asset_Unknown: + buffer:[base.ASSET_ID_TEXT]u8 + problem_line(&problems,"%s has no asset %s, which %s/%s uses: remove it from the list, or ingest the package again",named,base.asset_id_format(problem.asset,buffer[:]),root,PACKAGES_FILE) or_return + } + } + for &l in packages.listed { + if l.state!=.Not_Built || unreadable[l.package_id] {continue} + unreadable[l.package_id]=true + problem_line(&problems,"%s has no build on this machine that this engine reads, so its assets do not load: refresh it on a machine with its source, or fetch it from a peer that built it",package_named(s,list,l.package_id)) or_return + } + return {listed=packages.listed[:],packages=packages.packages[:],problems=problems[:]},.None +} + +// "the package () of /": by its manifest when the store +// has it, else by its id. In the temp allocator. +@(private="file") +package_named :: proc(s:^store.Store,list:^Packages,id:base.Asset_Id) -> string { + buffer:[base.ASSET_ID_TEXT]u8 + text:=base.asset_id_format(id,buffer[:]) + for &entry in list.packages { + if entry.id!=id {continue} + folder:=entry.folder!=""?entry.folder:"the top of the assets" + if p,error:=package_get(s,entry.manifest,context.temp_allocator);error==.None && p.id==id { + return fmt.tprintf("the package %s (%s) listed in %s",p.name,text,folder) + } + return fmt.tprintf("the package %s listed in %s",text,folder) + } + return fmt.tprintf("the package %s",text) +} + +@(private="file") +problem_line :: proc(problems:^[dynamic]string,format:string,args:..any) -> Error { + line,no_line:=strings.clone(fmt.tprintf(format,..args),problems.allocator) + if no_line!=nil {return .Out_Of_Memory} + if _,no_room:=append(problems,line);no_room!=nil {return .Out_Of_Memory} + return .None +} diff --git a/asset/packages_test.odin b/asset/packages_test.odin new file mode 100644 index 000000000..9cd462546 --- /dev/null +++ b/asset/packages_test.odin @@ -0,0 +1,425 @@ +package asset + +import "base:runtime" +import "core:fmt" +import "core:os" +import "core:strings" +import "core:testing" +import "core:time" +import "mirai:std/base" +import "mirai:store" + +@(private="file") +hash_of :: proc(text:string) -> store.Hash {return store.address_of_bytes(transmute([]u8)text)} + +@(private="file") +ids :: proc(list:..base.Asset_Id) -> [dynamic]base.Asset_Id { + out:=make([dynamic]base.Asset_Id,context.temp_allocator) + append(&out,..list) + return out +} + +@(private="file") +sample_package :: proc() -> Package { + p:=Package{id={1,1},name="crate \"kit\"/ü",origin={path="C:/Downloads/Kit\\crate.glb",importer="gltf"},identity=hash_of("crate"),settings=`{"lods": "None", "nested": {"a": [1, 2]}}`} + p.inputs=make([dynamic]string,context.temp_allocator) + append(&p.inputs,"crate.bin","textures/wood.png") + p.assets=make([dynamic]Package_Asset,context.temp_allocator) + append(&p.assets, + Package_Asset{kind=KIND_SCENE,name="crate",id={2,2},dependencies=ids({3,3},{6,6})}, + Package_Asset{kind=KIND_MESH,name="body",id={3,3},part=true,owner={2,2},dependencies=ids({4,4})}, + Package_Asset{kind=KIND_MATERIAL,name="wood",id={4,4},dependencies=ids({5,5})}, + Package_Asset{kind=KIND_TEXTURE,name="wood",id={5,5}}, + Package_Asset{kind=KIND_MESH,name="body",index=1,id={6,6}}, + Package_Asset{kind=KIND_TEXTURE,name="spare",id={7,7}}, + ) + return p +} + +// The object of a small texture whose id is id. +@(private="file") +texture_object :: proc(t:^testing.T,s:^store.Store,types:^Types,id:base.Asset_Id) -> store.Hash { + pixels:[16]u8 + doc:Document + if !testing.expect_value(t,texture_document(&doc,types,{id=id,width=2,height=2,format=.RGBA8,mips={pixels[:]}},context.temp_allocator),Error.None) {return {}} + defer document_destroy(&doc) + bytes,_:=document_bytes(&doc,context.temp_allocator) + object,error:=object_put(s,bytes) + testing.expect_value(t,error,Error.None) + testing.expect_value(t,store.commit(s),store.Error.None) + return object +} + +// Whether the store keeps the manifest pinned by hash. +@(private="file") +manifest_kept :: proc(s:^store.Store,hash:store.Hash) -> bool { + digits:[64]u8 + object,error:=store.ref_get(s,REF_MANIFESTS,store.hex(hash,&digits)) + return error==.None && object_whole(s,object) +} + +@(private="file") +sample_build :: proc(p:^Package,blobs:[]store.Hash) -> Build { + b:=Build{package_id=p.id,key=hash_of("key"),final=true} + b.outputs=make([dynamic]Build_Output,context.temp_allocator) + for &a,at in p.assets { + info,_:=kind_info(a.kind) + append(&b.outputs,Build_Output{id=a.id,kind=a.kind,kind_version=info.version,object=blobs[at%len(blobs)],thumbnail=at==0?hash_of("thumb"):{}}) + } + return b +} + +@(private="file") +sample_list :: proc() -> Packages { + list:Packages + list.packages=make([dynamic]Project_Package,context.temp_allocator) + crate:=Project_Package{id={1,1},manifest=hash_of("m1"),folder="props/crate",assets=ids({3,3})} + crate.written=make([dynamic]Package_Written,context.temp_allocator) + append(&crate.written,Package_Written{{3,3},0xfedcba9876543210},Package_Written{{4,4},1}) + append(&list.packages,crate,Project_Package{id={9,9},manifest=hash_of("m2"),assets=ids()}) + return list +} + +// Each record reads back as it was written, field by field, and writes the +// same text again; a record that could not do so is refused when written. +@(test) +package_records_read_back_what_they_write :: proc(t:^testing.T) { + p:=sample_package() + text,error:=package_text(&p,context.temp_allocator) + if !testing.expect_value(t,error,Error.None) {return} + back,parse_error:=package_parse(text,context.temp_allocator) + if !testing.expect_value(t,parse_error,Error.None) {return} + testing.expect(t,back.id==p.id && back.name==p.name && back.origin==p.origin && back.identity==p.identity && back.settings==p.settings && len(back.inputs)==2 && back.inputs[1]==p.inputs[1] && len(back.assets)==len(p.assets),"the manifest reads back") + for &a,at in back.assets { + want:=p.assets[at] + testing.expectf(t,a.kind==want.kind && a.name==want.name && a.index==want.index && a.id==want.id && a.part==want.part && a.owner==want.owner && len(a.dependencies)==len(want.dependencies),"asset %d reads back",at) + } + again,_:=package_text(&back,context.temp_allocator) + testing.expect_value(t,again,text) + + b:=sample_build(&p,{hash_of("a"),hash_of("b")}) + build,build_error:=build_text(&b,context.temp_allocator) + if !testing.expect_value(t,build_error,Error.None) {return} + built,built_error:=build_parse(build,context.temp_allocator) + if !testing.expect_value(t,built_error,Error.None) {return} + testing.expect(t,built.package_id==b.package_id && built.key==b.key && built.final && len(built.outputs)==len(b.outputs),"the build reads back") + for o,at in built.outputs {testing.expectf(t,o==b.outputs[at],"output %d reads back",at)} + rebuilt,_:=build_text(&built,context.temp_allocator) + testing.expect_value(t,rebuilt,build) + + list:=sample_list() + listed,list_error:=packages_text(&list,context.temp_allocator) + if !testing.expect_value(t,list_error,Error.None) {return} + read,read_error:=packages_parse(listed,context.temp_allocator) + if !testing.expect_value(t,read_error,Error.None) {return} + testing.expect(t,len(read.packages)==2 && read.packages[0].id==list.packages[0].id && read.packages[0].manifest==list.packages[0].manifest && read.packages[0].folder=="props/crate" && len(read.packages[0].assets)==1 && len(read.packages[1].assets)==0 && len(read.packages[0].written)==2 && read.packages[0].written[0]==list.packages[0].written[0] && len(read.packages[1].written)==0,"the list reads back") + relisted,_:=packages_text(&read,context.temp_allocator) + testing.expect_value(t,relisted,listed) + + // Through the store and the file of a project. + folder:=base.test_folder(t,"packages-records");if folder=="" {return} + s,open_error:=store.open(fmt.tprintf("%s/store",folder),context.temp_allocator) + if !testing.expect_value(t,open_error,store.Error.None) {return} + put,put_error:=package_put(&s,&p) + testing.expect(t,put_error==.None && put==hash_of(text),"a manifest is pinned by the hash of its text") + got,get_error:=package_get(&s,put,context.temp_allocator) + testing.expect(t,get_error==.None && got.id==p.id,"and reads back from the store") + _,missing:=package_get(&s,hash_of("nothing"),context.temp_allocator) + testing.expect_value(t,missing,Error.Not_Found) + empty,found,load_error:=packages_load(folder,context.temp_allocator) + testing.expect(t,load_error==.None && !found && len(empty.packages)==0,"a project without a list uses no package") + testing.expect_value(t,packages_save(folder,&list),Error.None) + loaded,saved,_:=packages_load(folder,context.temp_allocator) + testing.expect(t,saved && len(loaded.packages)==2,"the list of a project reads back from its file") + + // What would not read back is refused. + bad:=p + bad.settings=`{"open": ` + _,refused:=package_text(&bad,context.temp_allocator) + testing.expect_value(t,refused,Error.Invalid) + folded:=list + folded.packages[0].folder="../outside" + _,refused=packages_text(&folded,context.temp_allocator) + testing.expect_value(t,refused,Error.Invalid) +} + +// The listing of a project comes from its list, the manifests and the build +// each one's ref names, with the objects and their headers. A package without a +// build is listed not built; a manifest the store lacks, a build that does not +// read or an asset the manifest lacks is a problem of the result; nothing is +// removed. +@(test) +packages_resolve_lists_the_objects_of_what_a_project_uses :: proc(t:^testing.T) { + folder:=base.test_folder(t,"packages-resolve");if folder=="" {return} + s,open_error:=store.open(fmt.tprintf("%s/store",folder),context.temp_allocator) + if !testing.expect_value(t,open_error,store.Error.None) {return} + types:Types;types_init(&types);defer types_destroy(&types) + mesh,texture:=texture_object(t,&s,&types,{3,3}),texture_object(t,&s,&types,{5,5}) + p:=sample_package() + manifest,_:=package_put(&s,&p) + blobs:=[6]store.Hash{mesh,mesh,mesh,texture,mesh,texture} + b:=sample_build(&p,blobs[:]) + // The second mesh is of a version this program does not read. + b.outputs[4].kind_version+=1 + build,_:=build_put(&s,&b) + digits:[64]u8 + testing.expect_value(t,store.ref_set(&s,REF_BUILDS,store.hex(build_select(manifest,&p),&digits),build),store.Error.None) + + other:=Package{id={20,20},name="rock",identity=hash_of("rock")} + other.assets=make([dynamic]Package_Asset,context.temp_allocator) + append(&other.assets,Package_Asset{kind=KIND_TEXTURE,name="wood",id={21,21}}) + unbuilt,_:=package_put(&s,&other) + broken:=Package{id={30,30},name="broken",identity=hash_of("broken")} + broken.assets=make([dynamic]Package_Asset,context.temp_allocator) + append(&broken.assets,Package_Asset{kind=KIND_MESH,name="m",id={31,31}}) + broken_manifest,_:=package_put(&s,&broken) + not_a_build,_:=store.put(&s,transmute([]u8)string("not a build")) + testing.expect_value(t,store.ref_set(&s,REF_BUILDS,store.hex(build_select(broken_manifest,&broken),&digits),not_a_build),store.Error.None) + + list:Packages + list.packages=make([dynamic]Project_Package,context.temp_allocator) + append(&list.packages, + Project_Package{id=p.id,manifest=manifest,folder="kit",assets=ids({2,2},{99,99})}, + Project_Package{id=other.id,manifest=unbuilt,folder="kit",assets=ids()}, + Project_Package{id={40,40},manifest=hash_of("absent")}, + Project_Package{id=broken.id,manifest=broken_manifest}, + ) + resolved,error:=packages_resolve(&s,"",&list,context.temp_allocator) + if !testing.expect_value(t,error,Error.None) {return} + Want :: struct {id:base.Asset_Id,path:string,object:store.Hash,state:Listed_State,header:base.Asset_Id} + wanted:=[]Want{ + {{3,3},"kit/body.mesh",mesh,.Built,{3,3}}, + {{5,5},"kit/wood (crate _kit__ü).tex",texture,.Built,{5,5}}, + {{6,6},"kit/body (1).mesh",{},.Not_Built,{}}, + {{21,21},"kit/wood (rock).tex",{},.Not_Built,{}}, + {{31,31},"m.mesh",{},.Not_Built,{}}, + } + testing.expectf(t,len(resolved.listed)==len(wanted),"the cache assets used, with what they depend on: %v",resolved.listed) + for w in wanted { + found:=false + for l in resolved.listed { + if l.id!=w.id {continue} + found=true + testing.expectf(t,l.path==w.path && l.object==w.object && l.state==w.state && l.header.id==w.header,"%v is %s (%v), not %s (%v)",w.id,w.path,w.state,l.path,l.state) + } + testing.expectf(t,found,"%v is listed",w.id) + } + Problem :: struct {package_id,asset:base.Asset_Id,code:Package_Problem_Code} + problems:=[]Problem{{p.id,{99,99},.Asset_Unknown},{{40,40},{},.Manifest_Missing},{broken.id,{},.Build_Unreadable}} + testing.expectf(t,len(resolved.problems)==len(problems),"the problems: %v",resolved.problems) + for want in problems { + seen:=false + for got in resolved.problems {seen||=got.package_id==want.package_id && got.asset==want.asset && got.code==want.code} + testing.expectf(t,seen,"%v is reported",want) + } + for object in ([]store.Hash{mesh,texture,build,not_a_build}) {testing.expect(t,object_whole(&s,object),"resolving removes no object")} + for pin in ([]store.Hash{manifest,unbuilt}) {testing.expect(t,manifest_kept(&s,pin),"nor a manifest")} + _,ref_error:=store.ref_get(&s,REF_BUILDS,store.hex(build_select(manifest,&p),&digits)) + testing.expect_value(t,ref_error,store.Error.None) + + // The name of a build changes with the versions of the kinds a program reads. + _,ok:=import_key(p.identity,make([]u64,IMPORT_KEY_VERSIONS+1,context.temp_allocator)) + testing.expect(t,!ok,"an import key takes at most IMPORT_KEY_VERSIONS versions") + first,_:=import_key(p.identity,{1,2}) + second,_:=import_key(p.identity,{1,3}) + testing.expect(t,first!=second && first!=build_select(manifest,&p),"keys of other versions differ") + buffer:[store.NAME_MAX]u8 + testing.expect_value(t,cache_namespace("textures",&buffer),"cache-textures") + testing.expect_value(t,cache_namespace(strings.repeat("x",store.NAME_MAX,context.temp_allocator),&buffer),"") +} + +// text with the first of before replaced; "" when text does not have it. +@(private="file") +swap :: proc(text,before,after:string) -> string { + at:=strings.index(text,before) + if at<0 {return ""} + return strings.concatenate({text[:at],after,text[at+len(before):]},context.temp_allocator) +} + +@(private="file") +failing_allocator :: proc() -> runtime.Allocator { + return {procedure=proc(data:rawptr,mode:runtime.Allocator_Mode,size,alignment:int,old_memory:rawptr,old_size:int,loc:=#caller_location) -> ([]byte,runtime.Allocator_Error) { + #partial switch mode { + case .Free,.Free_All: return nil,.None + } + return nil,.Out_Of_Memory + }} +} + +// No text makes a parser of the records stop the program: random bytes, +// every cut of a valid text, oversized, deep or contradictory text, and +// memory that runs out give an error. +@(test) +package_records_refuse_hostile_text :: proc(t:^testing.T) { + Parser :: enum {Package,Build,Packages} + parse :: proc(parser:Parser,text:string,allocator:runtime.Allocator) -> Error { + switch parser { + case .Package: _,error:=package_parse(text,allocator);return error + case .Build: _,error:=build_parse(text,allocator);return error + case .Packages: _,error:=packages_parse(text,allocator);return error + } + return .None + } + p:=sample_package() + b:=sample_build(&p,{hash_of("a")}) + list:=sample_list() + valid:[Parser]string + valid[.Package],_=package_text(&p,context.temp_allocator) + valid[.Build],_=build_text(&b,context.temp_allocator) + valid[.Packages],_=packages_text(&list,context.temp_allocator) + id:=`"00000000000000010000000000000001"` + Case :: struct {parser:Parser,text:string,want:Error} + cases:=[]Case{ + {.Package,"",.Corrupt}, + {.Package,strings.repeat(" ",RECORD_BYTES+1,context.temp_allocator),.Too_Large}, + {.Package,strings.concatenate({`{"version": 1, "settings": `,strings.repeat("[",200,context.temp_allocator)},context.temp_allocator),.Corrupt}, + {.Package,swap(valid[.Package],`"version": 1`,`"version": 2`),.Version}, + {.Package,swap(valid[.Package],`"id": "00000000000000070000000000000007"`,`"id": "00000000000000050000000000000005"`),.Corrupt}, + {.Package,swap(valid[.Package],`"version": 1`,`"version": 1, "version": 1`),.Corrupt}, + {.Package,swap(valid[.Package],`"version": 1`,`"version": 1, "extra": 1`),.Corrupt}, + {.Package,swap(valid[.Package],`"kind": "TEX"`,`"kind": "TOOLONG"`),.Corrupt}, + {.Package,swap(valid[.Package],`"index": 1`,`"index": -1`),.Corrupt}, + {.Package,swap(valid[.Package],`"part": true`,`"part": true, "part": false`),.Corrupt}, + {.Build,swap(valid[.Build],`"kind_version": `,`"kind_version": 70000`),.Corrupt}, + {.Build,swap(valid[.Build],`"final": true`,`"final": 1`),.Corrupt}, + {.Build,swap(valid[.Build],`"package": `,`"package": "00000000000000000000000000000000", "x": `),.Corrupt}, + {.Build,fmt.tprintf(`{{"version": %d}}`,BUILD_MANIFEST_VERSION),.Corrupt}, + {.Packages,fmt.tprintf(`{{"version": 2, "packages": [{{"id": %s, "manifest": "nothex", "folder": "", "assets": [], "written": []}}]}}`,id),.Corrupt}, + {.Packages,fmt.tprintf(`{{"version": 2, "packages": [{{"id": %s, "manifest": "%s", "folder": "a/../b", "assets": [], "written": []}}]}}`,id,strings.repeat("0",64,context.temp_allocator)),.Corrupt}, + {.Packages,fmt.tprintf(`{{"version": 2, "packages": [{{"id": %s, "manifest": "%s", "folder": "", "assets": [%s, %s], "written": []}}]}}`,id,strings.repeat("0",64,context.temp_allocator),id,id),.Corrupt}, + {.Packages,fmt.tprintf(`{{"version": 2, "packages": [{{"id": %s, "manifest": "%s", "folder": "", "assets": []}}]}}`,id,strings.repeat("0",64,context.temp_allocator)),.Corrupt}, + {.Packages,fmt.tprintf(`{{"version": 2, "packages": [{{"id": %s, "manifest": "%s", "folder": "", "assets": [], "written": [{{"id": %s, "hash": "0000000000000000"}}]}}]}}`,id,strings.repeat("0",64,context.temp_allocator),id),.Corrupt}, + {.Packages,fmt.tprintf(`{{"version": 2, "packages": [{{"id": %s, "manifest": "%s", "folder": "", "assets": [], "written": [{{"id": %s, "hash": "01"}}]}}]}}`,id,strings.repeat("0",64,context.temp_allocator),id),.Corrupt}, + {.Packages,fmt.tprintf(`{{"version": 2, "packages": [{{"id": %s, "manifest": "%s", "folder": "", "assets": [], "written": [{{"id": %s, "hash": "0000000000000001"}}, {{"id": %s, "hash": "0000000000000002"}}]}}]}}`,id,strings.repeat("0",64,context.temp_allocator),id,id),.Corrupt}, + {.Packages,swap(valid[.Packages],`"version": 2`,`"version": 1`),.Version}, + {.Packages,`{"version": 2, "packages": []} {}`,.Corrupt}, + {.Packages,`{"version": 18446744073709551616, "packages": []}`,.Corrupt}, + } + for c,at in cases { + testing.expectf(t,at==0 || c.text!="","case %d changes a text it has",at) + got:=parse(c.parser,c.text,context.temp_allocator) + testing.expectf(t,got==c.want,"case %d (%v): %v, not %v",at,c.parser,got,c.want) + } + h:=base.Hostile{state=7} + random:=make([]u8,4096,context.temp_allocator) + for parser in Parser { + text:=strings.trim_right(valid[parser],"\n") + for cut in 0.. store.Hash { + hash,_:=store.put(s,transmute([]u8)text,at) + store.commit(s,at) + return hash + } + // What nothing reaches is most of the pack, which a sweep compacts. + noise:=make([]u8,1<<20,context.temp_allocator) + for &n,at in noise {n=u8((at*2654435761)>>13)} + mesh,texture,held,loose,cached:=put(&s,"mesh",old),put(&s,"texture",old),put(&s,"held",old),put(&s,string(noise),old),put(&s,"cached",old) + p:=sample_package() + manifest,_:=package_put(&s,&p) + blobs:=[2]store.Hash{mesh,texture} + b:=sample_build(&p,blobs[:]) + for &o in b.outputs {o.thumbnail={}} + build,_:=build_put(&s,&b) + digits:[64]u8 + store.ref_set(&s,REF_IMPORTS,store.hex(hash_of("key"),&digits),build) + store.ref_set(&s,REF_PACKAGES,"crate",manifest) + store.ref_set(&s,"cache-thumbs","1",cached) + swept,error:=store_sweep(&s,{held},now) + testing.expect_value(t,error,store.Error.None) + testing.expect(t,swept.chunks>0) + for hash in ([]store.Hash{mesh,texture,held,cached,build}) {testing.expect(t,object_whole(&s,hash))} + testing.expect(t,manifest_kept(&s,manifest)) + testing.expect(t,!store.chunk_has(&s,loose),"what nothing reaches goes") +} diff --git a/asset/parts.odin b/asset/parts.odin new file mode 100644 index 000000000..7d950411b --- /dev/null +++ b/asset/parts.odin @@ -0,0 +1,464 @@ +package asset + +import "base:intrinsics" +import "core:mem" +import "core:slice" +import "core:strings" +import "mirai:std/base" +import "mirai:std/serial" +import "mirai:store" + +// A native asset of a package lives in the store as an object: one group per +// container chunk in the order of the directory, cut where the format has +// pieces it reads alone, and the header and the directory as the meta of the +// root. The object holds its file as a write with Codecs.None makes it: no +// chunk, block or page goes through a stage of a compressor, and each kind's +// own modeling (BC blocks, planes of their fields, the coding of mesh pages and +// of terrain pages) is chosen without one; the store compresses each chunk +// where it rests. So the same asset is the same object for every compressor. +// The file is the meta, then each group at the offset of its chunk with zeros +// between them: loading the object and saving a file gives the bytes that were +// put once written so. + +// The most chunks an asset of the store has: its header and directory fit in the meta of a root. +OBJECT_CHUNKS_MAX :: (int(max(u16))-HEADER_SIZE)/CHUNK_ENTRY_SIZE + +// The object an Asset reads its chunks from, each the first time it is asked for. +@(private) +Asset_Object :: struct { + object:store.Object, + stored:[][]u8, // of each chunk once read whole: a slice of a mapped pack, or memory of the asset + here:[]bool, +} + +// Puts the file of an asset held in data as an object into the batch of s, +// without committing. The file is checked whole, its hash too, before any +// of it goes in. +object_put :: proc(s:^store.Store,original:[]u8) -> (address:store.Hash,error:Error) { + if s==nil {return {},.Invalid} + data:=original + // The file as it arrived is checked: what is written again from it carries a hash of its own. + { + arrived:Blob + arrived,error=blob_open(original) + if error!=.None {return} + if !blob_verify_hash(&arrived) {return {},.Hash} + } + if !(object_raw(data) or_return) { + data=object_raw_write(data) or_return + } + defer if raw_data(data)!=raw_data(original) {mem.free_bytes(data,context.temp_allocator)} + a:Asset + asset_open(&a,data,context.temp_allocator) or_return + defer asset_close(&a) + if !blob_verify_hash(&a.blob) {return {},.Hash} + count:=int(a.blob.header.chunk_count) + if count>OBJECT_CHUNKS_MAX {return {},.Too_Large} + types:Types + types_init(&types,context.temp_allocator) + defer types_destroy(&types) + cuts:=make([dynamic]int,context.temp_allocator) + p,begin_error:=store.put_begin(s) + if begin_error!=.None {return {},store_error(begin_error)} + for i in 0.. (raw:bool,error:Error) { + a:Asset + asset_open(&a,data,context.temp_allocator) or_return + defer asset_close(&a) + for i in 0..u64(len(page_data)) {return false,.Corrupt} + if page.data_bytes>0 && !pack_matches(page_data[page.data_offset:][:page.data_bytes],.Fast,.None) {return false,.None} + } + } + for &table in a.tables { + if !strings.has_suffix(table.name,"_pages") || len(table.columns)!=1 || table.columns[0].size!=size_of(Block_Record) {continue} + blocks_table,found:=asset_table(&a,strings.concatenate({table.name[:len(table.name)-len("_pages")],"_blocks"},context.temp_allocator)) + if !found || len(blocks_table.columns)!=1 {continue} + records:=asset_column_data(&a,&table,0) or_return + blocks:=asset_column_data(&a,blocks_table,0) or_return + for at:=0;at+size_of(Block_Record)<=len(records);at+=size_of(Block_Record) { + record:=intrinsics.unaligned_load((^Block_Record)(raw_data(records[at:]))) + if record.size==0 {continue} + if record.offset>u64(len(blocks)) || record.size>u64(len(blocks))-record.offset {return false,.Corrupt} + if !pack_matches(blocks[record.offset:][:record.size],.Fast,.None) {return false,.None} + } + } + return true,.None +} + +// A file written again as Codecs.None writes it, in the temporary allocator. +@(private="file") +object_raw_write :: proc(data:[]u8) -> (raw:[]u8,error:Error) { + header:=header_read(data) or_return + if header.kind==KIND_TEXTURE { + types:Types + types_init(&types,context.temp_allocator) + defer types_destroy(&types) + return texture_repack(data,&types,.Fast,context.temp_allocator) + } + return document_repack(data,.Fast,.None,context.temp_allocator) +} + +// object_put of the file at path. +object_put_file :: proc(s:^store.Store,path:string) -> (address:store.Hash,error:Error) { + data:=asset_file_read(path,context.temp_allocator) or_return + defer mem.free_bytes(data,context.temp_allocator) + return object_put(s,data) +} + +// Where a container chunk is cut, as offsets of its stored bytes: between the +// pieces of a stream, which decode alone; between the levels of the texels of +// a texture stored as they are, smallest first; between the blocks of the +// pages of a mesh and of a table of blocks (Block_Record). +@(private="file") +object_cuts :: proc(a:^Asset,types:^Types,index:int,entry:Chunk_Entry,stored:[]u8,cuts:^[dynamic]int) -> Error { + add :: proc(cuts:^[dynamic]int,at:u64) -> Error { + if _,no_room:=append(cuts,int(at));no_room!=nil {return .Out_Of_Memory} + return .None + } + if entry.compression==.Stream { + s:=stream_read(stored,int(entry.uncompressed_size)) or_return + at:=u64(s.first) + add(cuts,at) or_return + for piece in 0..len("_blocks") && table.name[len(table.name)-len("_blocks"):]=="_blocks": + name:=table.name[:len(table.name)-len("_blocks")] + records,found:=asset_table(a,strings.concatenate({name,"_pages"},context.temp_allocator)) + if !found || len(records.columns)!=1 || records.columns[0].size!=size_of(Block_Record) {break} + bytes:=asset_column_data(a,records,0) or_return + if len(bytes)%size_of(Block_Record)!=0 {return .Corrupt} + for at:=0;at0 {add(cuts,record.offset) or_return} + } + } + } + // What the store takes: offsets inside the bytes, ascending, each once. + slice.sort(cuts[:]) + kept:=0 + for cut in cuts { + if cut<=0 || cut>=len(stored) || kept>0 && cuts[kept-1]==cut {continue} + cuts[kept]=cut + kept+=1 + } + base.truncate(cuts,kept) + return .None +} + +// The table one of whose columns is the chunk index; nil for another chunk. +@(private="file") +object_table_of :: proc(a:^Asset,index:int) -> ^Asset_Table { + for &t in a.tables { + for c in t.columns { + if c.chunk==index {return &t} + } + } + return nil +} + +// Opens the asset of the store at address: its header, directory and tables +// from the meta and the chunks of the front, and every other chunk when it is +// first read, so a read of a part reads only the chunks that hold it. The store +// is borrowed and outlives the Asset; so is decoder. A chunk the store does not +// hold yet is Not_Here to whoever reads it. +asset_open_object :: proc(a:^Asset,s:^store.Store,address:store.Hash,allocator:=context.allocator,limit:=u64(DEFAULT_MEMORY_LIMIT),decoder:^Decoder=nil) -> Error { + a^={allocator=allocator,decoder=decoder} + a.owned=make([dynamic][]u8,allocator) + a.hashes=make([dynamic]u64,allocator) + error:=object_attach(a,s,address) + if error==.None {error=asset_parse(a,store.object_meta(&a.object.object),limit,object=true)} + if error!=.None {asset_close(a)} + return error +} + +@(private="file") +object_attach :: proc(a:^Asset,s:^store.Store,address:store.Hash) -> Error { + if s==nil {return .Invalid} + allocation:mem.Allocator_Error + if a.object,allocation=new(Asset_Object,a.allocator);allocation!=nil {return .Out_Of_Memory} + o,open_error:=store.object_open(s,address) + if open_error!=.None {return object_error(open_error)} + a.object.object=o + return .None +} + +// What a read of the store meets, as an asset error: a chunk it does not hold is Not_Here. +@(private) +object_error :: proc(error:store.Error) -> Error { + switch error { + case .None: return .None + case .Missing: return .Not_Here + case .Corrupt,.Invalid_Argument,.Unsupported_Layout: return .Corrupt + case .Out_Of_Memory: return .Out_Of_Memory + case .Io,.No_Space: return .IO + } + return .IO +} + +// The meta and the groups of an object against each other: a directory of +// entries inside the file, ascending and apart, each with the group of its +// tag and index holding its stored bytes, and a file that ends with its last +// chunk. Every size the meta declares is bounded by the groups before anything +// is allocated. +@(private) +object_check :: proc(a:^Asset,limit:u64) -> Error { + header:=a.blob.header + count:=int(header.chunk_count) + if len(a.blob.data)!=HEADER_SIZE+count*CHUNK_ENTRY_SIZE {return .Corrupt} + o:=&a.object.object + groups:=store.object_groups(o) + if len(groups)!=count {return .Corrupt} + total:=u64(0) + end:=u64(HEADER_SIZE+count*CHUNK_ENTRY_SIZE) + for i in 0..limit {return .Too_Large} + } + // The file ends where its last chunk does: its size is what the groups hold. + if header.total_size!=end {return .Corrupt} + allocation:mem.Allocator_Error + if a.object.stored,allocation=make([][]u8,count,a.allocator);allocation!=nil {return .Out_Of_Memory} + if a.object.here,allocation=make([]bool,count,a.allocator);allocation!=nil {return .Out_Of_Memory} + return .None +} + +// Reads every chunk of an asset of the store that is not read yet and lets go +// of its object: the Asset reads nothing more of the store, whose mapped packs +// its chunks may still be slices of while the store is open. Nothing for a file. +asset_object_settle :: proc(a:^Asset) -> Error { + if a.object==nil || a.object.object.s==nil {return .None} + for i in 0.. (stored:[]u8,error:Error) { + entry:=blob_chunk(&a.blob,index) + if a.object==nil { + // The front of a file (asset_open_front) holds only the chunks it was read for. + if entry.offset+entry.size>u64(len(a.blob.data)) {return nil,.Truncated} + return blob_stored(&a.blob,entry),.None + } + o:=a.object + if o.here[index] || entry.size==0 {return o.stored[index],.None} + if o.object.s==nil {return nil,.Invalid} + groups:=store.object_groups(&o.object) + if groups[index].count==1 { + bytes,at,chunk_error:=store.group_chunk(&o.object,index,0) + if chunk_error!=.None {return nil,object_error(chunk_error)} + if at!=0 || u64(len(bytes))!=entry.size {return nil,.Corrupt} + // A chunk of a pack not sealed yet is read into a buffer of the object the next read reuses. + if inside(bytes,o.object.chunk) || inside(bytes,o.object.nodes[0]) || inside(bytes,o.object.nodes[1]) { + copied:=asset_alloc(a,len(bytes),1< bool { + if len(buffer)==0 {return false} + at,start:=uintptr(raw_data(bytes)),uintptr(raw_data(buffer)) + return at>=start && at (stored:[]u8,error:Error) { + entry:=blob_chunk(&a.blob,index) + if a.object==nil || a.object.here[index] || size>=int(entry.size) {return asset_stored(a,index)} + stored=asset_alloc(a,int(entry.size),max(1< (stored:[]u8,error:Error) { + entry:=blob_chunk(&a.blob,index) + if a.object==nil || a.object.here[index] {return asset_stored(a,index)} + head:[STREAM_HEAD]u8 + if entry.sizeentry.size {return nil,.Corrupt} + table:=asset_stored_prefix(a,index,first) or_return + need:=u64(first) + for piece in 0..=raw {break} + need+=u64(u32_read(&table[STREAM_HEAD+piece*4])) + } + if need>entry.size {return nil,.Corrupt} + if e:=store.group_read(&a.object.object,index,u64(first),table[first:need]);e!=.None {return nil,object_error(e)} + return table,.None +} + +// The file of the asset of the store at address, as loading it and saving it +// gives, in memory aligned for its chunks. +object_file :: proc(s:^store.Store,address:store.Hash,allocator:=context.allocator) -> (data:[]u8,error:Error) { + a:Asset + asset_open_object(&a,s,address,context.temp_allocator) or_return + defer asset_close(&a) + header:=a.blob.header + allocation:mem.Allocator_Error + data,allocation=mem.alloc_bytes(int(header.total_size),asset_alignment(header),allocator) + if allocation!=nil {return nil,.Out_Of_Memory} + copy(data,a.blob.data) + for i in 0.. Error { + data:=object_file(s,address,context.temp_allocator) or_return + defer mem.free_bytes(data,context.temp_allocator) + return serial.file_write_atomic(path,data)==.None?.None:.IO +} + +// What the front of the asset of the store at address says, as import_peek of a file. +object_peek :: proc(s:^store.Store,address:store.Hash,types:^Types,dependencies:^[dynamic]base.Asset_Id=nil) -> (header:Header,info:Import_Info,found:bool,error:Error) { + a:Asset + asset_open_object(&a,s,address,context.temp_allocator) or_return + defer asset_close(&a) + header=a.blob.header + if dependencies!=nil {append(dependencies,..a.dependencies)} + if .Cooked not_in header.flags {return header,{},false,.None} + row:=import_info(&a,types) or_return + if row==nil {return header,{},false,.None} + return header,row^,true,.None +} + +// The header of the asset of the store at address, and into dependencies what +// its directory names: what a registry lists of it, from its meta and front. +object_header :: proc(s:^store.Store,address:store.Hash,dependencies:^[dynamic]base.Asset_Id) -> (header:Header,error:Error) { + a:Asset + asset_open_object(&a,s,address,context.temp_allocator) or_return + defer asset_close(&a) + if _,no_room:=append(dependencies,..a.dependencies);no_room!=nil {return {},.Out_Of_Memory} + return a.blob.header,.None +} + +// Whether the store holds every chunk of the object at address. +object_whole :: proc(s:^store.Store,address:store.Hash) -> bool { + if s==nil || !store.chunk_has(s,address) {return false} + o,error:=store.object_open(s,address) + if error!=.None {return false} + defer store.object_close(&o) + walk :: proc(s:^store.Store,n:store.Node_View,buffer:[]u8,depth:int) -> bool { + if n.level==0 { + for e in n.entries { + if !store.chunk_here(s,e.address) {return false} + } + return true + } + if depth>store.NODE_LEVEL_MAX {return false} + entries:=slice.clone(n.entries,context.temp_allocator) + defer delete(entries,context.temp_allocator) + for e in entries { + bytes,read_error:=store.chunk_read(s,e.address,buffer) + if read_error!=.None {return false} + child,parse_error:=store.node_parse(bytes) + if parse_error!=.None || child.level!=n.level-1 {return false} + child_buffer,no_room:=make([]u8,store.CHUNK_MAX,context.temp_allocator) + if no_room!=nil {return false} + defer delete(child_buffer,context.temp_allocator) + if !walk(s,child,child_buffer,depth+1) {return false} + } + return true + } + buffer,no_room:=make([]u8,store.CHUNK_MAX,context.temp_allocator) + if no_room!=nil {return false} + defer delete(buffer,context.temp_allocator) + return walk(s,o.root,buffer,0) +} diff --git a/asset/parts_test.odin b/asset/parts_test.odin new file mode 100644 index 000000000..2e7457355 --- /dev/null +++ b/asset/parts_test.odin @@ -0,0 +1,488 @@ +package asset + +import "core:fmt" +import "core:mem" +import "core:slice" +import "core:strings" +import "core:testing" +import "compressor:data" +import "mirai:std/base" +import "mirai:std/serial" +import "mirai:store" + +@(private="file") PARTS_DEMO :: #directory+"/../demo/assets/" + +@(private="file") +parts_store :: proc(t:^testing.T,name:string) -> (s:store.Store,ok:bool) { + folder:=base.test_folder(t,name) + if folder=="" {return} + error:store.Error + s,error=store.open(fmt.tprintf("%s/store",folder),context.temp_allocator) + return s,testing.expect_value(t,error,store.Error.None) +} + +@(private="file") +document_file :: proc(doc:^Document) -> []u8 { + defer document_destroy(doc) + bytes,_:=document_bytes(doc,context.temp_allocator) + return bytes +} + +// The file of one asset of each kind the store keeps, written from the same +// input each time it is called. +@(private="file") +kind_files :: proc(t:^testing.T,types:^Types) -> (names:[dynamic]string,files:[dynamic][]u8) { + names=make([dynamic]string,context.temp_allocator) + files=make([dynamic][]u8,context.temp_allocator) + add :: proc(names:^[dynamic]string,files:^[dynamic][]u8,name:string,bytes:[]u8) { + append(names,name) + append(files,bytes) + } + for codecs in ([2]Codecs{.None,.Open}) { + vertices,indices:=test_grid(300,context.temp_allocator) + doc:Document + types.codecs=codecs + defer types.codecs=.None + testing.expect_value(t,mesh_document(&doc,types,{id={1,1},geometry=test_geometry(vertices,indices)},context.temp_allocator),Error.None) + add(&names,&files,fmt.tprintf("mesh %v",codecs),document_file(&doc)) + } + for codecs in ([2]Codecs{.Open,.None}) { + SIDE :: 256 + levels:=make([][]u8,texture_mip_count(SIDE,SIDE),context.temp_allocator) + state:=u64(9) + for &level,i in levels { + side:=max(SIDE>>uint(i),1) + level=make([]u8,texture_level_size(.RGBA8,side,side),context.temp_allocator) + for &b in level {b=u8(test_random(&state)>>20)} + } + doc:Document + types.codecs=codecs + defer types.codecs=.None + testing.expect_value(t,texture_document(&doc,types,{id={2,2},width=SIDE,height=SIDE,format=.RGBA8,mips=levels},context.temp_allocator),Error.None) + add(&names,&files,fmt.tprintf("texture %v",codecs),document_file(&doc)) + } + { + lossless:=make([]u8,300000,context.temp_allocator) + copy(lossless,"MLPC") + for &b,i in lossless[4:] {b=u8(i*37)} + doc:Document + testing.expect_value(t,sound_document(&doc,types,{id={3,3},channels=2,sample_rate=48000,frames=480,lossless=lossless},context.temp_allocator),Error.None) + add(&names,&files,"sound",document_file(&doc)) + } + { + doc:Document + testing.expect_value(t,font_document(&doc,types,{id={4,4},file=#load("../assets/fixtures/cff.otf"),compression=.Stream},context.temp_allocator),Error.None) + add(&names,&files,"font",document_file(&doc)) + } + for file in ([]string{"blockout/column.mesh","gallery/plant/Leaves.mat","showcase/hud_badge.tex","showcase/showcase.scene","showcase/hud.ui","showcase/lift.anim","characters/spider/spider.skeleton","characters/jelly/jelly_moods.character","vfx_gallery/smoke_plume.particle","interface/merchant_answers.records","physics_gallery/bricks.fracture","forest/Hills.terrain","texts.strings","forest/Woods.foliage","physics_gallery/recorded.physics_cache","../../plugins/prompts_kenney/assets/kenney.prompts"}) { + bytes,error:=serial.file_read(strings.concatenate({PARTS_DEMO,file},context.temp_allocator),context.temp_allocator) + if testing.expectf(t,error==.None,"%s reads",file) {add(&names,&files,file,bytes)} + } + return +} + +// Every table and column of two opened assets, decoded. +@(private="file") +assets_equal :: proc(t:^testing.T,name:string,file,object:^Asset) -> bool { + // Sizes and the hash are of how the chunks are stored, which the store's object does not compress. + a,b:=file.blob.header,object.blob.header + a.total_size,a.content_hash,b.total_size,b.content_hash=0,0,0,0 + if !testing.expectf(t,a==b && len(file.tables)==len(object.tables) && slice.equal(file.dependencies,object.dependencies),"%s: the same header, tables and dependencies",name) {return false} + for &table,at in file.tables { + other:=&object.tables[at] + // A table of blocks is compared by what its blocks decode to, where its pages say. + blocks:=strings.has_suffix(table.name,"_blocks") + if !testing.expectf(t,table.name==other.name && (blocks || table.rows==other.rows) && len(table.columns)==len(other.columns),"%s: table %s",name,table.name) {return false} + if blocks {continue} + for _,column in table.columns { + a,a_error:=asset_column_data(file,&table,column) + b,b_error:=asset_column_data(object,other,column) + if strings.has_suffix(table.name,"_pages") && table.columns[column].size==size_of(Block_Record) && a_error==.None && b_error==.None { + if !blocks_equal(t,name,file,object,table.name,a,b) {return false} + continue + } + if !testing.expectf(t,a_error==.None && b_error==.None && mem.compare(a,b)==0,"%s: %s column %d (%v %v)",name,table.name,column,a_error,b_error) {return false} + } + } + return true +} + +@(private="file") +blocks_equal :: proc(t:^testing.T,name:string,file,object:^Asset,pages:string,a,b:[]u8) -> bool { + blocks_name:=strings.concatenate({pages[:len(pages)-len("_pages")],"_blocks"},context.temp_allocator) + file_table,_:=asset_table(file,blocks_name) + object_table,_:=asset_table(object,blocks_name) + file_blocks,_:=asset_column_data(file,file_table,0) + object_blocks,_:=asset_column_data(object,object_table,0) + ra,rb:=mem.slice_data_cast([]Block_Record,a),mem.slice_data_cast([]Block_Record,b) + if !testing.expectf(t,len(ra)==len(rb),"%s: %s",name,pages) {return false} + d:Stream_Decoder + stream_decoder_init(&d,context.temp_allocator) + defer stream_decoder_destroy(&d) + for x,i in ra { + y:=rb[i] + if !testing.expectf(t,x.count==y.count && x.fill==y.fill && x.raw==y.raw && (x.size==0)==(y.size==0),"%s: %s run %d",name,pages,i) {return false} + if x.size==0 {continue} + p,q:=make([]u8,x.raw,context.temp_allocator),make([]u8,y.raw,context.temp_allocator) + testing.expect_value(t,unpack(&d,file_blocks[x.offset:][:x.size],p),Error.None) + testing.expect_value(t,unpack(&d,object_blocks[y.offset:][:y.size],q),Error.None) + if !testing.expectf(t,mem.compare(p,q)==0,"%s: %s block %d",name,pages,i) {return false} + } + return true +} + +// Frames as a store of each build holds them: of this build's codec, kept as they are (a +// codec that never shrinks), and of a codec this build lacks (another build's). +@(private="file") +Frames :: enum {Build,Raw,Other} + +@(private="file") +frames_store :: proc(t:^testing.T,name:string,frames:Frames) -> (s:store.Store,ok:bool) { + s,ok=parts_store(t,fmt.tprintf("%s-%v",name,frames)) + if !ok {return} + switch frames { + case .Build: + case .Raw: store.codec_hook(&s,{codec=14,encode=proc(src,dst:[]u8) -> int {return 0},decode=proc(src,dst:[]u8) -> bool {return false}}) + case .Other: store.codec_hook(&s,{codec=15,encode=proc(src,dst:[]u8) -> int { + e:data.Encoder + data.encoder_init(&e) + defer data.encoder_destroy(&e) + size,error:=data.encode(&e,.Fast,src,dst) + return size if error==.None else 0 + },decode=proc(src,dst:[]u8) -> bool { + d:data.Decoder + if data.decoder_init(&d)!=.None {return false} + defer data.decoder_destroy(&d) + return data.decode(&d,data.ZSTD,src,dst)==.None + }}) + } + return +} + +// Every kind opens from its object as from its file, from a store of each kind of frames, and +// is the same object in each and whatever compressor wrote its file: an object holds its file +// as Codecs.None writes it, which it gives back byte for byte. +@(test) +every_kind_loads_equal_from_a_file_and_from_the_store :: proc(t:^testing.T) { + types:Types;types_init(&types);defer types_destroy(&types) + names,files:=kind_files(t,&types) + _,again:=kind_files(t,&types) + objects:=make([]store.Hash,len(files),context.temp_allocator) + for frames in Frames { + s,ok:=frames_store(t,"asset-parts-kinds",frames);if !ok {return} + defer store.close(&s) + for bytes,at in files { + name:=names[at] + object,put_error:=object_put(&s,bytes) + if !testing.expectf(t,put_error==.None,"%s is put: %v",name,put_error) {continue} + twice,_:=object_put(&s,again[at]) + testing.expectf(t,twice==object,"%s: the same input encodes into the same object",name) + if frames==.Build {objects[at]=object} + testing.expectf(t,object==objects[at],"%s: the same object in a store of %v frames",name,frames) + testing.expect_value(t,store.commit(&s),store.Error.None) + file,stored:Asset + if !testing.expect_value(t,asset_open(&file,bytes,context.temp_allocator),Error.None) {continue} + defer asset_close(&file) + if !testing.expect_value(t,asset_open_object(&stored,&s,object,context.temp_allocator),Error.None) {continue} + defer asset_close(&stored) + if file.blob.header.kind!=KIND_TEXTURE && file.blob.header.kind!=KIND_MESH {assets_equal(t,name,&file,&stored)} + switch stored.blob.header.kind { + case KIND_MESH: + a,a_error:=mesh_view(&file,&types) + b,b_error:=mesh_view(&stored,&types) + testing.expectf(t,a_error==.None && b_error==.None && a.info^==b.info^ && slice.equal(a.levels,b.levels) && slice.equal(a.clusters,b.clusters) && len(a.pages)==len(b.pages),"%s: the same mesh",name) + if a_error==.None && b_error==.None {testing.expectf(t,slice.equal(a.vertices,b.vertices) && slice.equal(a.triangles,b.triangles),"%s: the same vertices and triangles",name)} + case KIND_TEXTURE: + a,a_error:=texture_view(&file,&types) + b,b_error:=texture_view(&stored,&types) + testing.expectf(t,a_error==.None && b_error==.None && a.info.width==b.info.width && a.info.format==b.info.format && len(a.mips)==len(b.mips),"%s: the same texture",name) + if a_error==.None && b_error==.None { + for _,level in a.mips {testing.expectf(t,mem.compare(texture_mip(a,level),texture_mip(b,level))==0,"%s: level %d",name,level)} + } + case KIND_SOUND: + a,a_error:=sound_view(&file,&types) + b,b_error:=sound_view(&stored,&types) + testing.expectf(t,a_error==.None && b_error==.None && mem.compare(a.lossless,b.lossless)==0,"%s: the same sound",name) + case KIND_FONT: + a,a_error:=font_view(&file,&types) + b,b_error:=font_view(&stored,&types) + testing.expectf(t,a_error==.None && b_error==.None && mem.compare(a.file,b.file)==0,"%s: the same font",name) + } + back,back_error:=object_file(&s,object,context.temp_allocator) + if !testing.expectf(t,back_error==.None,"%s: the object gives a file back",name) {continue} + raw,_:=document_repack(bytes,.Fast,.None,context.temp_allocator) + if header,_:=header_read(bytes);header.kind!=KIND_TEXTURE && mem.compare(raw,bytes)==0 { + testing.expectf(t,mem.compare(back,bytes)==0,"%s: a file written without a compressor comes back byte for byte",name) + } + again_object,_:=object_put(&s,back) + testing.expectf(t,again_object==object,"%s: the file it gives back is the same object",name) + } + } + for name,at in names { + for other,k in names { + kind:=name[:strings.index_byte(name,' ')+1] + if k<=at || (kind!="texture " && kind!="mesh ") || !strings.has_prefix(other,kind) {continue} + testing.expectf(t,objects[at]==objects[k],"%s and %s, of one content, are one object",name,other) + } + } +} + +// The leaves of the object at root and where each begins in it. +@(private="file") +leaves_of :: proc(s:^store.Store,root:store.Hash) -> (nodes:[dynamic]store.Hash,leaves:[dynamic]store.Leaf) { + nodes=make([dynamic]store.Hash,context.temp_allocator) + leaves=make([dynamic]store.Leaf,context.temp_allocator) + walk :: proc(s:^store.Store,address:store.Hash,at:^u64,nodes:^[dynamic]store.Hash,leaves:^[dynamic]store.Leaf) { + buffer:=make([]u8,store.CHUNK_MAX,context.temp_allocator) + bytes,_:=store.chunk_read(s,address,buffer) + n,error:=store.node_parse(bytes) + if error!=.None {return} + append(nodes,address) + for e in n.entries { + if n.level>0 {walk(s,e.address,at,nodes,leaves);continue} + append(leaves,store.Leaf{address=e.address,size=e.size,offset=at^}) + at^+=e.size + } + } + at:u64 + walk(s,root,&at,&nodes,&leaves) + return +} + +@(private="file") +chunk_copy :: proc(from,into:^store.Store,address:store.Hash) { + buffer:=make([]u8,store.CHUNK_MAX,context.temp_allocator) + bytes,_:=store.chunk_read(from,address,buffer) + store.chunk_put(into,address,bytes) +} + +// The small levels of a texture are the first leaves of its texels: they load +// while the large ones are not in the store, which are Not_Here, never a +// failure of the asset, and load once they arrive. +@(test) +the_small_levels_of_a_texture_load_before_the_large_ones_arrive :: proc(t:^testing.T) { + types:Types;types_init(&types);defer types_destroy(&types) + full,ok:=parts_store(t,"asset-parts-levels");if !ok {return} + defer store.close(&full) + partial,partial_ok:=parts_store(t,"asset-parts-levels-partial");if !partial_ok {return} + defer store.close(&partial) + _,files:=kind_files(t,&types) + bytes:=files[3] // the texture stored as it is + object,_:=object_put(&full,bytes) + store.commit(&full) + file:Asset + if !testing.expect_value(t,asset_open(&file,bytes,context.temp_allocator),Error.None) {return} + defer asset_close(&file) + table,_:=asset_table(&file,"texels") + chunk:=table.columns[0].chunk + entry:=blob_chunk(&file.blob,chunk) + if !testing.expect(t,entry.compression==.None) {return} + mips,_:=asset_rows(&file,&types,"mips","",Texture_Mip) + // The object begins with the groups before texels; its levels from 2 end where level 1 begins. + groups_before:=u64(0) + { + o,_:=store.object_open(&full,object) + groups_before=store.object_groups(&o)[chunk].offset + store.object_close(&o) + } + small_end:=groups_before+u64(mips[1].offset) + nodes,leaves:=leaves_of(&full,object) + for n in nodes {chunk_copy(&full,&partial,n)} + held,lacking:=0,0 + for leaf in leaves { + if leaf.offset>=small_end {lacking+=1;continue} + chunk_copy(&full,&partial,leaf.address) + held+=1 + } + store.commit(&partial) + testing.expectf(t,lacking>0 && held>0,"the large levels are leaves of their own (%d held, %d lacking)",held,lacking) + a:Asset + if !testing.expect_value(t,asset_open_object(&a,&partial,object,context.temp_allocator),Error.None) {return} + view,error:=texture_open(&a,&types) + testing.expect_value(t,error,Error.None) + testing.expect_value(t,texture_decode(&a,&types,&view,2),Error.None) + reference,_:=texture_view(&file,&types) + for level in 2..=small_end {chunk_copy(&full,&partial,leaf.address)} + } + store.commit(&partial) + testing.expect_value(t,asset_open_object(&a,&partial,object,context.temp_allocator),Error.None) + view,error=texture_open(&a,&types) + testing.expect(t,error==.None && texture_decode(&a,&types,&view,0)==.None && mem.compare(texture_mip(view,0),texture_mip(reference,0))==0,"the large levels load once they arrive") + asset_close(&a) + _,absent:=store.object_open(&partial,store.address_of_bytes(transmute([]u8)string("absent"))) + testing.expect_value(t,absent,store.Error.Missing) + testing.expect_value(t,asset_open_object(&a,&partial,store.address_of_bytes(transmute([]u8)string("absent")),context.temp_allocator),Error.Not_Here) +} + +// Meta and groups that lie about the asset fail it with an error: a directory +// whose sizes are not those of the groups, groups of other tags, too few or +// beyond the object, a meta that is no header, a root that is no node. +@(test) +hostile_object_meta_and_groups_fail_the_asset :: proc(t:^testing.T) { + types:Types;types_init(&types);defer types_destroy(&types) + s,ok:=parts_store(t,"asset-parts-hostile");if !ok {return} + defer store.close(&s) + _,files:=kind_files(t,&types) + bytes:=files[0] + good:Asset + if !testing.expect_value(t,asset_open(&good,bytes,context.temp_allocator),Error.None) {return} + defer asset_close(&good) + count:=int(good.blob.header.chunk_count) + meta_size:=HEADER_SIZE+count*CHUNK_ENTRY_SIZE + // Puts the chunks of the file as groups with meta, after change edits the copy. + put :: proc(s:^store.Store,good:^Asset,bytes:[]u8,meta:[]u8,skip:int,tag:u32) -> store.Hash { + p,_:=store.put_begin(s) + for i in 0..=size_of(store.Node) { + (^u32)(raw_data(junk))^=store.NODE_MAGIC + junk[4]=store.NODE_VERSION + } + address:=store.address_of_bytes(junk) + store.chunk_put(&s,address,junk) + a:Asset + if asset_open_object(&a,&s,address,context.temp_allocator)==.None {asset_close(&a)} + } + store.commit(&s) +} + +// A chunk whose frame keeps it as it is, one chunk of a sealed pack, is read in place, +// a slice of the mapped pack: no copy, and no read of anything else. +@(test) +a_chunk_stored_as_it_is_is_read_in_place :: proc(t:^testing.T) { + types:Types;types_init(&types);defer types_destroy(&types) + folder:=base.test_folder(t,"asset-parts-in-place");if folder=="" {return} + root:=fmt.tprintf("%s/store",folder) + // Bytes that do not compress: the store keeps their frame as it is. + pixels:=make([]u8,64*64*4,context.temp_allocator) + state:=u64(5) + for &p in pixels {p=u8(test_random(&state)>>20)} + doc:Document + types.codecs=.None + testing.expect_value(t,texture_document(&doc,&types,{id={8,8},width=64,height=64,format=.RGBA8,mips={pixels}},context.temp_allocator),Error.None) + bytes:=document_file(&doc) + object:store.Hash + { + s,_:=store.open(root,context.temp_allocator) + object,_=object_put(&s,bytes) + store.commit(&s) + store.close(&s) // seals its pack + } + s,_:=store.open(root,context.temp_allocator) + defer store.close(&s) + a:Asset + if !testing.expect_value(t,asset_open_object(&a,&s,object,context.temp_allocator),Error.None) {return} + defer asset_close(&a) + view,error:=texture_view(&a,&types) + testing.expect_value(t,error,Error.None) + at:=uintptr(raw_data(texture_mip(view,0))) + for block in a.owned {testing.expect(t,at=uintptr(raw_data(block))+uintptr(len(block)),"the level is no memory of the asset")} + testing.expect(t,mem.compare(texture_mip(view,0),pixels)==0) +} + +// A file whose content does not match its hash is refused as it arrived, also one a compressor +// wrote, which the store's object holds written again without it. +@(test) +a_damaged_file_is_refused_before_it_is_written_again :: proc(t:^testing.T) { + types:Types;types_init(&types);defer types_destroy(&types) + s,ok:=parts_store(t,"asset-parts-damaged");if !ok {return} + defer store.close(&s) + names,files:=kind_files(t,&types) + for bytes,at in files { + if names[at]!="mesh Open" {continue} + a:Asset + if !testing.expect_value(t,asset_open(&a,bytes,context.temp_allocator),Error.None) {return} + streams,flipped:=0,false + for i in 0..8 { + bytes[entry.offset+entry.size/2]~=0x40 + flipped=true + } + } + asset_close(&a) + if !testing.expect(t,streams>0 && flipped,"a file of a compressed and an uncompressed chunk") {return} + _,error:=object_put(&s,bytes) + testing.expect_value(t,error,Error.Hash) + } +} diff --git a/asset/record.odin b/asset/record.odin index 9d613b773..920107d3f 100644 --- a/asset/record.odin +++ b/asset/record.odin @@ -92,6 +92,7 @@ Asset :: struct { view_types:^Types, decoder:^Decoder, // lent by whoever reads, else made by asset_decoder and owned lacking:[2]string, // the table and the field of a row a whole read refused: the file does not store that field + object:^Asset_Object, // of an asset of the store (asset_open_object): blob.data is its header and directory decoder_owned:bool, } @@ -163,6 +164,7 @@ asset_close :: proc(a:^Asset) { delete(a.owned);delete(a.hashes) if a.schema.allocator.procedure!=nil {serial.schema_destroy(&a.schema)} mem.free_bytes(a.file,a.allocator) + object_detach(a) if a.decoder_owned { decoder_destroy(a.decoder) free(a.decoder,a.allocator) @@ -198,19 +200,24 @@ asset_alloc :: proc(a:^Asset,size:int,alignment:int) -> (block:[]u8,error:Error) @(private) asset_chunk :: proc(a:^Asset,index:int,alignment:int) -> (data:[]u8,error:Error) { entry:=blob_chunk(&a.blob,index) - // The front of a file (asset_open_front) holds only the chunks it was read for. - if entry.offset+entry.size>u64(len(a.blob.data)) {return nil,.Truncated} - if entry.compression==.None { - stored:=blob_stored(&a.blob,entry) - if len(stored)==0 || uintptr(raw_data(stored))&uintptr(alignment-1)==0 {return stored,.None} - } + stored:=asset_stored(a,index) or_return + if entry.compression==.None && (len(stored)==0 || uintptr(raw_data(stored))&uintptr(alignment-1)==0) {return stored,.None} data=asset_alloc(a,int(entry.uncompressed_size),alignment) or_return decoder:^Decoder if entry.compression!=.None {decoder=asset_decoder(a) or_return} - blob_read(&a.blob,entry,data,decoder) or_return + chunk_decode(entry,stored,data,decoder) or_return return data,.None } +// The content of a chunk into dst of its uncompressed size, whether the asset is a file or of the store. +asset_chunk_read :: proc(a:^Asset,index:int,dst:[]u8) -> Error { + entry:=blob_chunk(&a.blob,index) + stored:=asset_stored(a,index) or_return + decoder:^Decoder + if entry.compression!=.None {decoder=asset_decoder(a) or_return} + return chunk_decode(entry,stored,dst,decoder) +} + // Opens the front of an asset file: everything up to the chunks a caller // reads, which the bulk data follows. Reaching a chunk past it is Truncated. @(private) @@ -262,8 +269,12 @@ front_read :: proc(path:string,head:[]u8,header:Header,all:bool,allocator:=conte } @(private) -asset_parse :: proc(a:^Asset,data:[]u8,limit:u64,front:=false) -> Error { - if front { +asset_parse :: proc(a:^Asset,data:[]u8,limit:u64,front:=false,object:=false) -> Error { + if object { + header:=header_read(data) or_return + a.blob={data=data,header=header} + object_check(a,limit) or_return + } else if front { header:=header_read(data) or_return if u64(len(data))>header.total_size || len(data) bool {return len( Registry :: struct { root:string, - imported:string, // the imported folder that goes with the root; "" for none + listed:[]Package_Listed, // the outputs of the packages of the root, listed with its files + listed_files:map[string]store.Hash, // the path where each listed output is, to the object of its entry (zero when not built) + store:^store.Store, // where the listed outputs are read (registry_read); nil: they are not cache:string, // "" for none mounts:[dynamic]Registry_Mount, files:[dynamic]Registry_File, // every file, sorted by path @@ -126,20 +130,24 @@ registry_slashed :: proc(path:string,allocator:runtime.Allocator) -> (slashed:st return string(bytes),nil } -// False, with a registry of no folder that registry_destroy takes, without memory for it. -registry_init :: proc(r:^Registry,root:string,cache:="",allocator:=context.allocator,imported:="") -> bool { +// listed: the outputs of the packages of the root (packages_resolve), which +// scans list beside its files. False, with a +// registry of no folder that registry_destroy takes, without memory for it. +registry_init :: proc(r:^Registry,root:string,cache:="",allocator:=context.allocator,listed:[]Package_Listed=nil) -> bool { r^={allocator=allocator} - no_root,no_cache,no_imported:runtime.Allocator_Error + no_root,no_cache:runtime.Allocator_Error r.root,no_root=registry_slashed(root,allocator) r.cache,no_cache=registry_slashed(cache,allocator) - r.imported,no_imported=registry_slashed(imported,allocator) - if no_root!=nil || no_cache!=nil || no_imported!=nil { + kept,no_listed:=registry_listed_clone(listed,allocator) + if no_root!=nil || no_cache!=nil || !no_listed { delete(r.root,allocator) delete(r.cache,allocator) - delete(r.imported,allocator) - r.root,r.cache,r.imported="","","" + registry_listed_free(kept,allocator) + r.root,r.cache,kept="","",nil no_root=.Out_Of_Memory } + r.listed=kept + r.listed_files=make(map[string]store.Hash,allocator) r.files=make([dynamic]Registry_File,allocator) r.folders=make([dynamic]string,allocator) r.dependencies=make([dynamic]base.Asset_Id,allocator) @@ -157,24 +165,75 @@ registry_init :: proc(r:^Registry,root:string,cache:="",allocator:=context.alloc } @(private="file") -registry_mount_add :: proc(r:^Registry,folder,prefix:string,guest:bool,imported:="") -> bool { +registry_mount_add :: proc(r:^Registry,folder,prefix:string,guest:bool,listed:[]Package_Listed=nil) -> bool { kept_folder,no_folder:=strings.clone(folder,r.allocator) kept_prefix,no_prefix:=strings.clone(prefix,r.allocator) - kept_imported,no_imported:=strings.clone(imported,r.allocator) - if no_folder==nil && no_prefix==nil && no_imported==nil { - if _,no_room:=append(&r.mounts,Registry_Mount{kept_folder,kept_prefix,guest,kept_imported});no_room==nil {return true} + kept_listed,listed_ok:=registry_listed_clone(listed,r.allocator) + if no_folder==nil && no_prefix==nil && listed_ok { + if _,no_room:=append(&r.mounts,Registry_Mount{kept_folder,kept_prefix,guest,kept_listed});no_room==nil {return true} } delete(kept_folder,r.allocator) delete(kept_prefix,r.allocator) - delete(kept_imported,r.allocator) + registry_listed_free(kept_listed,r.allocator) return false } -// Adds folder to what the next scan sees, its files under prefix/. The prefix +// Adds folder to what the next scan sees, its files under prefix/, and the +// outputs of the packages of its root, listed under prefix/ too. The prefix // is a relative path with '/' that names no folder of the root. False without // memory for it. -registry_mount :: proc(r:^Registry,folder,prefix:string,imported:="") -> bool { - return registry_mount_add(r,folder,prefix,false,imported) +registry_mount :: proc(r:^Registry,folder,prefix:string,listed:[]Package_Listed=nil) -> bool { + return registry_mount_add(r,folder,prefix,false,listed) +} + +// Replaces the outputs of packages the next scan lists: those of the root +// ("") or of the mount of prefix. False without memory for them, or with no +// mount of prefix, which keep what they had. +registry_listed_set :: proc(r:^Registry,prefix:string,listed:[]Package_Listed) -> bool { + kept,ok:=registry_listed_clone(listed,r.allocator) + if !ok {return false} + if prefix=="" { + registry_listed_free(r.listed,r.allocator) + r.listed=kept + return true + } + for &mount in r.mounts { + if mount.prefix!=prefix {continue} + registry_listed_free(mount.listed,r.allocator) + mount.listed=kept + return true + } + registry_listed_free(kept,r.allocator) + return false +} + +// A copy of entries whose strings live in allocator. False without memory for it. +@(private="file") +registry_listed_clone :: proc(entries:[]Package_Listed,allocator:runtime.Allocator) -> (kept:[]Package_Listed,ok:bool) { + if len(entries)==0 {return nil,true} + no_room:runtime.Allocator_Error + if kept,no_room=make([]Package_Listed,len(entries),allocator);no_room!=nil {return nil,false} + for entry,at in entries { + no_path,no_dependencies:runtime.Allocator_Error + copied:=entry + copied.path,no_path=strings.clone(entry.path,allocator) + copied.dependencies,no_dependencies=slice.clone(entry.dependencies,allocator) + kept[at]=copied + if no_path!=nil || no_dependencies!=nil { + registry_listed_free(kept,allocator) + return nil,false + } + } + return kept,true +} + +@(private="file") +registry_listed_free :: proc(entries:[]Package_Listed,allocator:runtime.Allocator) { + for entry in entries { + delete(entry.path,allocator) + delete(entry.dependencies,allocator) + } + delete(entries,allocator) } // Mounts now a folder the program did not ship (what a player saved or @@ -197,7 +256,7 @@ registry_mount_guest :: proc(r:^Registry,folder,prefix:string) -> (refused:int,e if !registry_mount_add(r,folder,prefix,true) {return 0,.Out_Of_Memory} r.revision+=1 mount:=r.mounts[len(r.mounts)-1] - if error=registry_walk(r,mount,nil,nil,nil,nil);error!=.None { + if error=registry_walk(r,mount,nil,nil,nil);error!=.None { registry_unmount(r,prefix) return } @@ -243,7 +302,7 @@ registry_unmount :: proc(r:^Registry,prefix:string) -> Error { } base.truncate(&r.folders,kept) registry_refused_clear(r,prefix) - delete(r.mounts[at].folder,r.allocator);delete(r.mounts[at].prefix,r.allocator);delete(r.mounts[at].imported,r.allocator) + delete(r.mounts[at].folder,r.allocator);delete(r.mounts[at].prefix,r.allocator);registry_listed_free(r.mounts[at].listed,r.allocator) ordered_remove(&r.mounts,at) indexed:=registry_index(r) if r.cache!="" {registry_cache_write(r)} @@ -289,11 +348,25 @@ registry_refused_clear :: proc(r:^Registry,prefix:string) { base.truncate(&r.refused,kept) } -// The file a path of the registry is: under the root, under the folder of -// the mount whose prefix it starts with, or itself when it is outside them. -// "" without memory for it. +// The file a path of the registry is: under the root, under the folder of the +// mount whose prefix it starts with, or itself when it is outside them. "" +// for a listed output, which is an object of the store and no file +// (registry_listed_object), and without memory for it. registry_file_path :: proc(r:^Registry,path:string,allocator:=context.allocator) -> string { - if at,known:=r.by_path[path];known && r.files[at].imported!=0 {return registry_imported_file(r,path,r.files[at].id,r.files[at].imported,allocator)} + if path in r.listed_files {return ""} + return registry_disk_path(r,path,allocator) +} + +// The object of the store a listed output at path is; false for a path that +// is no listed output, or one not built. +registry_listed_object :: proc(r:^Registry,path:string) -> (object:store.Hash,ok:bool) { + object,ok=r.listed_files[path] + return object,ok && object!={} +} + +// The file of the folders at a path of the registry, whatever is listed there. +@(private="file") +registry_disk_path :: proc(r:^Registry,path:string,allocator:=context.allocator) -> string { parts:=[3]string{r.root,"/",path} if registry_outside(path) { parts={path,"",""} @@ -325,14 +398,17 @@ registry_destroy :: proc(r:^Registry) { for entry in r.reassigned {registry_reassigned_free(r,entry)} delete(r.reassigned) delete(r.dependents);delete(r.dependent_files) - for mount in r.mounts {delete(mount.folder,r.allocator);delete(mount.prefix,r.allocator);delete(mount.imported,r.allocator)} + for mount in r.mounts {delete(mount.folder,r.allocator);delete(mount.prefix,r.allocator);registry_listed_free(mount.listed,r.allocator)} delete(r.mounts) + registry_listed_free(r.listed,r.allocator) + for path in r.listed_files {delete(path,r.allocator)} + delete(r.listed_files) for &held in r.memory {registry_memory_free(r,&held)} delete(r.memory) delete(r.removed) registry_refused_clear(r,"") delete(r.refused) - delete(r.root,r.allocator);delete(r.cache,r.allocator);delete(r.imported,r.allocator) + delete(r.root,r.allocator);delete(r.cache,r.allocator) r^={} } @@ -437,16 +513,7 @@ registry_probe :: proc(r:^Registry,full:string,file:^Registry_File,dependencies: if resize(dependencies,first+int(count))!=nil {return .Out_Of_Memory} target:=mem.slice_to_bytes(dependencies[first:]) copy(target,stored) - listed:=dependencies[first:] - slice.sort_by(listed,proc(a,b:base.Asset_Id) -> bool {return a[0]0 && id==listed[unique-1] {continue} - listed[unique]=id - unique+=1 - } - base.truncate(dependencies,first+unique) - file.dependency_count=u32(unique) + file.dependency_count=u32(dependencies_unique(dependencies,first)) } file.state=.Asset return @@ -475,13 +542,12 @@ registry_scan :: proc(r:^Registry) -> Error { registry_refused_clear(r,"") if !os.is_directory(r.root) {return .Not_Found} - sidecars:=make([dynamic]Sidecar_Found,context.temp_allocator) - if error:=registry_walk(r,{folder=r.root,imported=r.imported},known,previous[:],previous_dependencies[:],&sidecars);error!=.None {return error} + if error:=registry_walk(r,{folder=r.root},known,previous[:],previous_dependencies[:]);error!=.None {return error} for mount in r.mounts { if !os.is_directory(mount.folder) {continue} - if error:=registry_walk(r,mount,known,previous[:],previous_dependencies[:],&sidecars);error!=.None {return error} + if error:=registry_walk(r,mount,known,previous[:],previous_dependencies[:]);error!=.None {return error} } - if error:=registry_imports_list(r,sidecars[:],known,previous[:],previous_dependencies[:]);error!=.None {return error} + if error:=registry_listed_add(r,known,previous[:],previous_dependencies[:]);error!=.None {return error} indexed:=registry_index(r) slice.sort(r.folders[:]) if r.cache!="" && (r.stats.opened>0 || len(previous)!=len(r.files)) {registry_cache_write(r)} @@ -504,7 +570,7 @@ registry_scan :: proc(r:^Registry) -> Error { // known says previous had with the same size and time is not opened again. @(private) registry_walk_reuse :: proc(r:^Registry,file:^Registry_File,full,path:string,known:map[string]u32,previous:[]Registry_File,previous_dependencies:[]base.Asset_Id) -> Error { - if at,seen:=known[path];seen && previous[at].size==file.size && previous[at].time==file.time && previous[at].imported==file.imported { + if at,seen:=known[path];seen && previous[at].size==file.size && previous[at].time==file.time && previous[at].listed==file.listed { file^=previous[at] file.first_dependency=u32(len(r.dependencies)) append(&r.dependencies,..previous_dependencies[previous[at].first_dependency:][:previous[at].dependency_count]) @@ -516,13 +582,8 @@ registry_walk_reuse :: proc(r:^Registry,file:^Registry_File,full,path:string,kno return registry_probe(r,full,file,&r.dependencies) } -// A sidecar a walk found, with the imported folder of its mount. @(private="file") -Sidecar_Found :: struct {path,full,imported:string} - -// sidecars receives the sidecars the walk finds; nil for a guest, whose sources nobody imports. -@(private="file") -registry_walk :: proc(r:^Registry,folder:Registry_Mount,known:map[string]u32,previous:[]Registry_File,previous_dependencies:[]base.Asset_Id,sidecars:^[dynamic]Sidecar_Found) -> Error { +registry_walk :: proc(r:^Registry,folder:Registry_Mount,known:map[string]u32,previous:[]Registry_File,previous_dependencies:[]base.Asset_Id) -> Error { root,root_error:=os.get_absolute_path(folder.folder,context.temp_allocator) if root_error!=nil {return .IO} relative:=make([dynamic]u8,context.temp_allocator) @@ -560,9 +621,6 @@ registry_walk :: proc(r:^Registry,folder:Registry_Mount,known:map[string]u32,pre } continue } - if sidecars!=nil && strings.has_suffix(path,SIDECAR_EXTENSION) && folder.imported!="" { - append(sidecars,Sidecar_Found{strings.clone(path,context.temp_allocator),strings.clone(info.fullpath,context.temp_allocator),folder.imported}) - } file:=Registry_File{size=info.size,time=time.time_to_unix_nano(info.modification_time)} r.stats.files+=1 registry_walk_reuse(r,&file,info.fullpath,path,known,previous,previous_dependencies) or_return @@ -572,74 +630,104 @@ registry_walk :: proc(r:^Registry,folder:Registry_Mount,known:map[string]u32,pre return .None } -// Appends what the sidecars a scan found list of the cache whose file their -// imported folder has, at the paths sidecar_paths gives. A path a file of the -// registry or another output has is no identity: every output that would be -// there is listed with the file name of its source, " ()". +// Appends the outputs the packages of the root and of each mount that is no +// guest list, at their paths under its prefix, without looking for a source: +// a built one is its blob, opened as any file; one whose blob is not on this +// machine is Not_Built, never left out. A path a file or a folder of the +// registry or another entry has is no identity: the entry is listed with its +// id after its name, " ()". An entry without an id, or whose path is +// empty, leaves its folder or is one a scan leaves out, goes to refused. @(private="file") -registry_imports_list :: proc(r:^Registry,sidecars:[]Sidecar_Found,known:map[string]u32,previous:[]Registry_File,previous_dependencies:[]base.Asset_Id) -> Error { - Held :: struct {key:u64,size,time:i64} - Listed :: struct {path,file,source:string,held:Held} - folders:=make(map[string]map[base.Asset_Id]Held,context.temp_allocator) - listed:=make([dynamic]Listed,context.temp_allocator) - claims:=make(map[string]int,context.temp_allocator) - for file in r.files {claims[file.path]=2} - for folder in r.folders {claims[folder]=1} - for found in sidecars { - source:=found.full[:len(found.full)-len(SIDECAR_EXTENSION)] - if !os.is_file(source) {continue} - text,read_error:=serial.file_read(found.full,context.temp_allocator) - if read_error!=.None {continue} - sidecar,ok:=sidecar_parse(string(text),context.temp_allocator) - if !ok {continue} - held,seen:=folders[found.imported] - if !seen { - held=make(map[base.Asset_Id]Held,context.temp_allocator) - entries,_:=os.read_all_directory_by_path(found.imported,context.temp_allocator) - for entry in entries { - if id,key,named:=imported_name(entry.name);named && entry.type==.Regular {held[id]={key,entry.size,time.time_to_unix_nano(entry.modification_time)}} - } - folders[found.imported]=held - } - slash:=strings.last_index_byte(found.path,'/') - for path,at in sidecar_paths(sidecar,found.path[:slash+1]) { - output:=sidecar.outputs[at] - kept,built:=held[output.id] - if path=="" || !built {continue} - extension:=path[strings.last_index_byte(path,'.'):] - append(&listed,Listed{path,imported_file(found.imported,output.id,kept.key,extension,context.temp_allocator),source[strings.last_index_any(source,"/\\")+1:],kept}) - claims[path]+=1 - } +registry_listed_add :: proc(r:^Registry,known:map[string]u32,previous:[]Registry_File,previous_dependencies:[]base.Asset_Id) -> Error { + for path in r.listed_files {delete(path,r.allocator)} + clear(&r.listed_files) + claims:=make(map[string]bool,context.temp_allocator) + for file in r.files {claims[file.path]=true} + for folder in r.folders {claims[folder]=true} + registry_listed_under(r,"",r.listed,&claims,known,previous,previous_dependencies) or_return + for mount in r.mounts { + if !mount.guest {registry_listed_under(r,mount.prefix,mount.listed,&claims,known,previous,previous_dependencies) or_return} } - for &l in listed { - if claims[l.path]>1 { - dot:=strings.last_index_byte(l.path,'.') - l.path=fmt.tprintf("%s (%s)%s",l.path[:dot],l.source,l.path[dot:]) + return .None +} + +@(private="file") +registry_listed_under :: proc(r:^Registry,prefix:string,entries:[]Package_Listed,claims:^map[string]bool,known:map[string]u32,previous:[]Registry_File,previous_dependencies:[]base.Asset_Id) -> Error { + for entry in entries { + path:=prefix=="" ? entry.path : fmt.tprintf("%s/%s",prefix,entry.path) + valid:=entry.id!={} && entry.path!="" && folder_valid(entry.path) && !registry_skipped(entry.path) + if valid && (path in claims^ || fmt.tprintf("%s/",path) in claims^) { + dot:=strings.last_index_byte(path,'.') + if dot<=strings.last_index_byte(path,'/') {dot=len(path)} + buffer:[base.ASSET_ID_TEXT]u8 + path=fmt.tprintf("%s (%s)%s",path[:dot],base.asset_id_format(entry.id,buffer[:]),path[dot:]) + valid=!(path in claims^) + } + if !valid { + refused,no_room:=strings.clone(path,r.allocator) + if no_room!=nil {return .Out_Of_Memory} + if _,no_append:=append(&r.refused,refused);no_append!=nil { + delete(refused,r.allocator) + return .Out_Of_Memory + } + continue } + claims^[path]=true // The folders it is in are folders of the listing, also without a file of the assets. - for at:=strings.index_byte(l.path,'/');at>=0;at=strings.index_byte(l.path[at+1:],'/')<0?-1:at+1+strings.index_byte(l.path[at+1:],'/') { - if claims[l.path[:at+1]]>0 {continue} - folder,no_folder:=strings.clone(l.path[:at+1],r.allocator) + for at:=strings.index_byte(path,'/');at>=0;at=strings.index_byte(path[at+1:],'/')<0?-1:at+1+strings.index_byte(path[at+1:],'/') { + if path[:at+1] in claims^ {continue} + folder,no_folder:=strings.clone(path[:at+1],r.allocator) if no_folder!=nil {return .Out_Of_Memory} - append(&r.folders,folder) - claims[folder]=1 + if _,no_room:=append(&r.folders,folder);no_room!=nil { + delete(folder,r.allocator) + return .Out_Of_Memory + } + claims^[folder]=true + } + file:=Registry_File{listed=true,state=.Not_Built,id=entry.id,kind=entry.kind,first_dependency=u32(len(r.dependencies))} + if entry.state==.Built && entry.object!={} { + h:=entry.header + file=Registry_File{listed=true,size=i64(h.total_size),state=.Asset,id=h.id,owner=h.owner,kind=h.kind,kind_version=h.kind_version,flags=h.flags,content_hash=h.content_hash,first_dependency=u32(len(r.dependencies))} + registry_dependencies_add(r,&file,entry.dependencies) or_return + // An object that is not the asset its entry names contradicts it. + if h.magic!=MAGIC || file.id!=entry.id || file.kind!=entry.kind {file.state=.Broken} } - registered:=Registry_File{size=l.held.size,time=l.held.time,imported=l.held.key} r.stats.files+=1 - registry_walk_reuse(r,®istered,l.file,l.path,known,previous,previous_dependencies) or_return - if registered.state==.Asset {registry_file_add(r,registered,l.path) or_return} + registry_file_add(r,file,path) or_return + key,no_key:=strings.clone(path,r.allocator) + if no_key!=nil {return .Out_Of_Memory} + _,object,_,no_entry:=map_entry(&r.listed_files,key) + if no_entry!=nil { + delete(key,r.allocator) + return .Out_Of_Memory + } + object^=entry.object } return .None } -// The file of the imported asset the registry lists at path, built from key. +// The ids at the end of dependencies from first, sorted and each once: how many are left. @(private="file") -registry_imported_file :: proc(r:^Registry,path:string,id:base.Asset_Id,key:u64,allocator:=context.allocator) -> string { - folder:=r.imported - for mount in r.mounts { - if registry_under(path,mount.prefix) {folder=mount.imported;break} +dependencies_unique :: proc(dependencies:^[dynamic]base.Asset_Id,first:int) -> int { + listed:=dependencies[first:] + slice.sort_by(listed,proc(a,b:base.Asset_Id) -> bool {return a[0]0 && id==listed[unique-1] {continue} + listed[unique]=id + unique+=1 } - return imported_file(folder,id,key,path[strings.last_index_byte(path,'.'):],allocator) + base.truncate(dependencies,first+unique) + return unique +} + +@(private="file") +registry_dependencies_add :: proc(r:^Registry,file:^Registry_File,ids:[]base.Asset_Id) -> Error { + first:=len(r.dependencies) + if len(ids)>REGISTRY_MAX_DEPENDENCIES {return .None} + if _,no_room:=append(&r.dependencies,..ids);no_room!=nil {return .Out_Of_Memory} + file.dependency_count=u32(dependencies_unique(&r.dependencies,first)) + return .None } // Gives every file with the id of an older one an id of its own, in memory: @@ -763,13 +851,13 @@ registry_copy_id :: proc(r:^Registry,id:base.Asset_Id,path:string) -> base.Asset // the files an editor wrote or a watch saw change. A path that is no file // now leaves the registry, one that is new joins it, and one whose size and // time did not change is not opened. A path held in memory keeps what memory -// says, and names a scan leaves out are ignored. A sidecar, or a source that -// has one, scans the folder. +// says, and names a scan leaves out are ignored. A listed output stays as it +// is, unless a file of the folders is at its path now, which scans them. registry_refresh :: proc(r:^Registry,paths:[]string) -> Error { if !os.is_directory(r.root) {return .Not_Found} - // A sidecar or its source changes what the sidecars list, which a scan derives. + // Which of a file and a listed output takes a path is what a scan derives. for path in paths { - if strings.has_suffix(path,SIDECAR_EXTENSION) || strings.concatenate({path,SIDECAR_EXTENSION},context.temp_allocator) in r.by_path {return registry_scan(r)} + if path in r.listed_files && os.is_file(registry_disk_path(r,path,context.temp_allocator)) {return registry_scan(r)} } changed,exhausted:=false,false r.stats.opened,r.stats.reused=0,0 @@ -786,11 +874,15 @@ registry_refresh :: proc(r:^Registry,paths:[]string) -> Error { } forgot:=registry_reassigned_forget(r,path) if _,held:=registry_memory_find(r,path);held {continue} + // Blobs do not change: what is listed changes with the entries, at a scan. + if path in r.listed_files { + r.stats.reused+=1 + continue + } full:=registry_file_path(r,path,context.temp_allocator) info,stat_error:=os.stat(full,context.temp_allocator) present:=stat_error==nil && info.type==.Regular at,known:=r.by_path[path] - imported:=known?r.files[at].imported:0 if !present { if !known {continue} if !changed {r.revision+=1;changed=true} @@ -802,7 +894,7 @@ registry_refresh :: proc(r:^Registry,paths:[]string) -> Error { r.stats.files-=1 continue } - file:=Registry_File{size=info.size,time=time.time_to_unix_nano(info.modification_time),imported=imported} + file:=Registry_File{size=info.size,time=time.time_to_unix_nano(info.modification_time)} if known && !forgot && r.files[at].size==file.size && r.files[at].time==file.time { r.stats.reused+=1 continue @@ -962,8 +1054,8 @@ registry_save :: proc(r:^Registry,path:string) -> bool { serial.write_svarint(&w,file.size) serial.write_svarint(&w,file.time) serial.write_u8(&w,u8(file.state)) - if file.state!=.Asset {continue} - serial.write_u64(&w,file.imported) + if file.state!=.Asset && file.state!=.Not_Built {continue} + serial.write_u8(&w,u8(file.listed)) serial.write_u64(&w,file.id[0]);serial.write_u64(&w,file.id[1]) serial.write_u64(&w,file.owner[0]);serial.write_u64(&w,file.owner[1]) serial.write_u32(&w,u32(file.kind)) @@ -999,7 +1091,7 @@ registry_load :: proc(r:^Registry,path:string) -> Error { state:=registry_cache_read(path,&r.files,&r.dependencies,r.allocator) r.stats.files,r.stats.reused=len(r.files),len(r.files) for mount in r.mounts { - if mount.guest && os.is_directory(mount.folder) {registry_walk(r,mount,nil,nil,nil,nil)} + if mount.guest && os.is_directory(mount.folder) {registry_walk(r,mount,nil,nil,nil)} } indexed:=registry_index(r) slice.sort(r.folders[:]) @@ -1027,8 +1119,10 @@ registry_cache_read :: proc(path:string,files:^[dynamic]Registry_File,dependenci state:=serial.read_u8(&r) if r.failed || state>u8(max(Registry_State)) {ok=false;break} file.state=Registry_State(state) - if file.state==.Asset { - file.imported=serial.read_u64(&r) + if file.state==.Asset || file.state==.Not_Built { + listed:=serial.read_u8(&r) + if listed>1 {ok=false;break} + file.listed=listed==1 file.id={serial.read_u64(&r),serial.read_u64(&r)} file.owner={serial.read_u64(&r),serial.read_u64(&r)} file.kind=Fourcc(serial.read_u32(&r)) diff --git a/asset/registry_memory.odin b/asset/registry_memory.odin index 4169432dd..3d7affc70 100644 --- a/asset/registry_memory.odin +++ b/asset/registry_memory.odin @@ -16,7 +16,8 @@ import "mirai:std/serial" // file in memory takes the size of its content and the time it last changed. // A file outside the folders, named by its absolute path (the Slang source of // a program), is held the same way but is no file of the registry: no record, -// scan or lookup by id sees it. +// scan or lookup by id sees it. A listed output of a package is an object of +// the store, which nothing changes: memory holds none. Registry_Memory :: struct { path:string, // as Registry_File.path @@ -24,8 +25,10 @@ Registry_Memory :: struct { removed:bool, } -// path holds data in memory: a copy of it. Out_Of_Memory leaves it as it was. +// path holds data in memory: a copy of it. Out_Of_Memory leaves it as it was; +// Invalid for a listed output. registry_memory_set :: proc(r:^Registry,path:string,data:[]u8) -> Error { + if path in r.listed_files {return .Invalid} copied:=asset_bytes_clone(data,r.allocator) or_return held:=registry_memory_take(r,path) or_return registry_memory_free_data(r,held) @@ -34,8 +37,9 @@ registry_memory_set :: proc(r:^Registry,path:string,data:[]u8) -> Error { } // path is not there in memory: no file answers for it until it is written or -// dropped. +// dropped. Invalid for a listed output. registry_memory_remove :: proc(r:^Registry,path:string) -> Error { + if path in r.listed_files {return .Invalid} held:=registry_memory_take(r,path) or_return registry_memory_free_data(r,held) held.data,held.removed=nil,true @@ -133,9 +137,16 @@ registry_memory_write :: proc(r:^Registry,path:string) -> Error { } // What a file of the registry holds now: its content in memory, or the file -// on disk, aligned for the chunks of the asset it is. Not_Found for a removed -// file and for one that is not there. +// on disk, aligned for the chunks of the asset it is; of a listed output, its +// file as its object of the store gives it (object_file). Not_Found for a +// removed file, for one that is not there and for a listed output not built +// or without the store of the registry. registry_read :: proc(r:^Registry,path:string,allocator:=context.allocator,limit:=u64(DEFAULT_MEMORY_LIMIT)) -> (data:[]u8,error:Error) { + if object,listed:=r.listed_files[path];listed { + if object=={} || r.store==nil {return nil,.Not_Found} + if at,known:=r.by_path[path];known && u64(r.files[at].size)>limit {return nil,.Too_Large} + return object_file(r.store,object,allocator) + } if held,found:=registry_memory_find(r,path);found { if held.removed {return nil,.Not_Found} if u64(len(held.data))>limit {return nil,.Too_Large} diff --git a/asset/registry_test.odin b/asset/registry_test.odin index ec2812d7a..bafd8d005 100644 --- a/asset/registry_test.odin +++ b/asset/registry_test.odin @@ -2,10 +2,12 @@ package asset import "core:fmt" import "core:os" +import "core:strings" import "core:testing" import "core:time" import "mirai:std/base" import "mirai:std/serial" +import "mirai:store" @(private="file") write_material :: proc(t:^testing.T,types:^Types,path:string,id:base.Asset_Id,texture:base.Asset_Id) { @@ -557,70 +559,173 @@ registry_mounts_a_guest_folder_that_never_takes_an_id_the_registry_has :: proc(t } } -// What the imports of the sources of a folder build that nobody edits is in -// the imported folder that goes with it, each file named by its id and the -// key of what it was built from, and is listed where the sidecar that lists it -// is now, in the folder it names: never by the order of a scan. A sidecar -// that moves with its source moves the listing, also through a refresh; one -// whose source is gone lists nothing. +// What the packages a project and a plugin pin list of the cache is listed +// from the store through root_packages of each root, with no source +// anywhere: a built output is its object and loads, one whose +// build is not here is listed Not_Built and named as a problem, a path a file of the folder has takes the id of the output after +// its name, a refresh keeps every listed output and memory never holds one. @(test) -registry_lists_the_imported_folder_by_where_the_sources_are :: proc(t:^testing.T) { +registry_lists_what_a_project_pins_without_its_sources :: proc(t:^testing.T) { types:Types;types_init(&types);defer types_destroy(&types) - folder:=base.test_folder(t,"registry-imported") - if folder=="" {return} - root:=fmt.tprintf("%s/game/assets",folder) - imported:=fmt.tprintf("%s/game/.mirai/imported",folder) - plugin:=fmt.tprintf("%s/pack/assets",folder) - plugin_imported:=fmt.tprintf("%s/pack/.mirai/imported",folder) - for made in ([]string{fmt.tprintf("%s/props/crate",root),fmt.tprintf("%s/kit",root),imported,plugin,plugin_imported}) {os.make_directory_all(made)} - wood,body,lid,second,atlas,lost,unbuilt,other:=base.Asset_Id{1,1},base.Asset_Id{2,2},base.Asset_Id{3,3},base.Asset_Id{10,10},base.Asset_Id{5,5},base.Asset_Id{6,6},base.Asset_Id{7,7},base.Asset_Id{9,9} - pixels:[16]u8 - sidecar :: proc(t:^testing.T,source,folder:string,present:bool,outputs:[]Import_Output) { - listed:=make([dynamic]Import_Output,context.temp_allocator) - append(&listed,..outputs) - text:=sidecar_text(`{"lods": "None"}`,{folder=folder,outputs=listed},context.temp_allocator) - testing.expect_value(t,serial.file_write_atomic(fmt.tprintf("%s%s",source,SIDECAR_EXTENSION),transmute([]u8)text),serial.File_Error.None) - if present {testing.expect_value(t,serial.file_write_atomic(source,transmute([]u8)string("a source")),serial.File_Error.None)} + folder:=base.test_folder(t,"registry-listed");if folder=="" {return} + root,plugin,staging:=fmt.tprintf("%s/game/assets",folder),fmt.tprintf("%s/pack/assets",folder),fmt.tprintf("%s/staging",folder) + for made in ([]string{fmt.tprintf("%s/props/crate",root),plugin,staging}) {os.make_directory_all(made)} + s,open_error:=store.open(fmt.tprintf("%s/store",folder),context.temp_allocator) + if !testing.expect_value(t,open_error,store.Error.None) {return} + wood,lid,later,atlas,own:=base.Asset_Id{1,1},base.Asset_Id{3,3},base.Asset_Id{7,7},base.Asset_Id{5,5},base.Asset_Id{4,4} + blob :: proc(t:^testing.T,types:^Types,s:^store.Store,staging:string,id:base.Asset_Id) -> store.Hash { + buffer:[base.ASSET_ID_TEXT]u8 + path:=fmt.tprintf("%s/%s.tex",staging,base.asset_id_format(id,buffer[:])) + write_texture(t,types,path,id) + hash,error:=object_put_file(s,path) + testing.expect_value(t,error,Error.None) + testing.expect_value(t,store.commit(s),store.Error.None) + return hash } - sidecar(t,fmt.tprintf("%s/props/wood.png",root),"",true,{{kind=KIND_TEXTURE,name="wood",id=wood}}) - sidecar(t,fmt.tprintf("%s/props/crate.glb",root),"crate",true,{{kind=KIND_SCENE,name="crate",id={8,8}},{kind=KIND_TEXTURE,name="body",id=body,part=true},{kind=KIND_TEXTURE,name="lid",id=lid},{kind=KIND_TEXTURE,name="lid",index=1,id=second},{kind=KIND_TEXTURE,name="later",id=unbuilt}}) - sidecar(t,fmt.tprintf("%s/props/crate.obj",root),"crate",true,{{kind=KIND_TEXTURE,name="body",id=other}}) - sidecar(t,fmt.tprintf("%s/props/gone.png",root),"",false,{{kind=KIND_TEXTURE,name="gone",id=lost}}) - sidecar(t,fmt.tprintf("%s/atlas.png",plugin),"",true,{{kind=KIND_TEXTURE,name="atlas",id=atlas}}) - Built :: struct {imported:string,id:base.Asset_Id,key:u64,listed:string} - built:=[]Built{ - {imported,wood,0x11,"props/wood.tex"},{imported,body,0x22,"props/crate/body (crate.glb).tex"},{imported,lid,0x22,"props/crate/lid.tex"}, - {imported,second,0x22,"props/crate/lid (1).tex"},{imported,other,0x55,"props/crate/body (crate.obj).tex"},{plugin_imported,atlas,0x33,"plugins/pack/atlas.tex"},{imported,lost,0x44,""}, + pin :: proc(t:^testing.T,s:^store.Store,p:^Package,built:[]store.Hash,folder:string) -> Packages { + manifest,error:=package_put(s,p) + testing.expect_value(t,error,Error.None) + b:=Build{package_id=p.id,final=true} + b.outputs=make([dynamic]Build_Output,context.temp_allocator) + for hash,at in built { + info,_:=kind_info(p.assets[at].kind) + if hash!={} {append(&b.outputs,Build_Output{id=p.assets[at].id,kind=p.assets[at].kind,kind_version=info.version,object=hash})} + } + hash,_:=build_put(s,&b) + digits:[64]u8 + testing.expect_value(t,store.ref_set(s,REF_BUILDS,store.hex(build_select(manifest,p),&digits),hash),store.Error.None) + list:Packages + list.packages=make([dynamic]Project_Package,context.temp_allocator) + append(&list.packages,Project_Package{id=p.id,manifest=manifest,folder=folder}) + return list } - for b in built {testing.expect_value(t,texture_save(imported_file(b.imported,b.id,b.key,".tex",context.temp_allocator),&types,{id=b.id,width=2,height=2,format=.RGBA8,mips={pixels[:]}}),Error.None)} - write_material(t,&types,fmt.tprintf("%s/props/crate/Wood.mat",root),{4,4},wood) + crate:=Package{id={20,20},name="crate",identity=store.address_of_bytes(transmute([]u8)string("crate"))} + crate.assets=make([dynamic]Package_Asset,context.temp_allocator) + append(&crate.assets,Package_Asset{kind=KIND_SCENE,name="crate",id={8,8}},Package_Asset{kind=KIND_TEXTURE,name="wood",id=wood},Package_Asset{kind=KIND_TEXTURE,name="lid",id=lid},Package_Asset{kind=KIND_TEXTURE,name="later",id=later}) + wood_blob,lid_blob:=blob(t,&types,&s,staging,wood),blob(t,&types,&s,staging,lid) + crate_list:=pin(t,&s,&crate,{{},wood_blob,lid_blob,{}},"props/crate") + pack:=Package{id={30,30},name="pack",identity=store.address_of_bytes(transmute([]u8)string("pack"))} + pack.assets=make([dynamic]Package_Asset,context.temp_allocator) + append(&pack.assets,Package_Asset{kind=KIND_TEXTURE,name="atlas",id=atlas}) + pack_list:=pin(t,&s,&pack,{blob(t,&types,&s,staging,atlas)},"") + os.remove_all(staging) + // Each root lists from its own list and the store it is given; one without a list lists nothing. + game,pack_root:=fmt.tprintf("%s/game",folder),fmt.tprintf("%s/pack",folder) + testing.expect_value(t,packages_save(game,&crate_list),Error.None) + testing.expect_value(t,packages_save(pack_root,&pack_list),Error.None) + resolved,resolve_error:=root_packages(game,s.root,context.temp_allocator) + plugin_resolved,plugin_error:=root_packages(pack_root,s.root,context.temp_allocator) + if !testing.expect(t,resolve_error==.None && plugin_error==.None && len(resolved.listed)==3 && len(plugin_resolved.listed)==1) {return} + // The output whose build is not here is a problem that names its package; the others load. + testing.expectf(t,len(resolved.problems)==1 && strings.contains(resolved.problems[0],"package crate") && strings.contains(resolved.problems[0],"no build"),"%v",resolved.problems) + testing.expect_value(t,len(plugin_resolved.problems),0) + none,none_error:=root_packages(staging,fmt.tprintf("%s/unopened",folder),context.temp_allocator) + testing.expect(t,none_error==.None && len(none.listed)==0 && len(none.problems)==0 && !os.exists(fmt.tprintf("%s/unopened",folder)),"a root without a list opens no store") + // A file of the project at the path of a listed output keeps it. + write_texture(t,&types,fmt.tprintf("%s/props/crate/lid.tex",root),own) + write_material(t,&types,fmt.tprintf("%s/props/crate/Wood.mat",root),{9,9},wood) + buffer:[base.ASSET_ID_TEXT]u8 + renamed:=fmt.tprintf("props/crate/lid (%s).tex",base.asset_id_format(lid,buffer[:])) cache:=fmt.tprintf("%s/registry.cache",folder) for pass in 0..<2 { - r:Registry;registry_init(&r,root,cache,imported=imported);defer registry_destroy(&r) - testing.expect(t,registry_mount(&r,plugin,"plugins/pack",plugin_imported)) + r:Registry;registry_init(&r,root,cache,listed=resolved.listed[:]);defer registry_destroy(&r) + r.store=&s + testing.expect(t,registry_mount(&r,plugin,"plugins/pack",plugin_resolved.listed[:])) testing.expect_value(t,registry_scan(&r),Error.None) testing.expectf(t,pass==0 || r.stats.opened==0,"the second scan reads its cache: %v",r.stats) - for b in built { - file,found:=registry_find(&r,b.id) - if b.listed=="" { - testing.expect(t,!found,"what a source that is gone gave is not listed") - continue - } - if !testing.expectf(t,found && file.path==b.listed && file.imported==b.key,"%v is listed at %s, not %s",b.id,b.listed,file.path) {continue} - testing.expect_value(t,registry_file_path(&r,file.path,context.temp_allocator),imported_file(b.imported,b.id,b.key,".tex",context.temp_allocator)) + testing.expect_value(t,r.stats.files,6) + Want :: struct {id:base.Asset_Id,path,file:string} + for w in ([]Want{{wood,"props/crate/wood.tex",""},{lid,renamed,""},{own,"props/crate/lid.tex",fmt.tprintf("%s/props/crate/lid.tex",root)}}) { + file,found:=registry_find(&r,w.id) + if !testing.expectf(t,found && file.path==w.path && file.state==.Asset && file.listed==(w.id!=own),"%v is listed at %s, not %v",w.id,w.path,found?file^:Registry_File{}) {continue} + testing.expect_value(t,registry_file_path(&r,file.path,context.temp_allocator),w.file) + } + shipped,found_atlas:=registry_find(&r,atlas) + testing.expect(t,found_atlas && shipped.path=="plugins/pack/atlas.tex" && shipped.listed) + data,read_error:=registry_read(&r,"props/crate/wood.tex") + a:Asset + if testing.expect_value(t,read_error,Error.None) && testing.expect_value(t,asset_open(&a,data),Error.None) { + testing.expect(t,a.blob.header.id==wood,"a listed output loads from its object") + asset_close(&a) } - _,pending:=registry_find(&r,unbuilt) + delete(data) + pending,listed_pending:=registry_find_path(&r,"props/crate/later.tex") + testing.expect(t,listed_pending && pending.state==.Not_Built && pending.listed && pending.id==later && pending.kind==KIND_TEXTURE,"an output not built is listed as such") + _,by_id:=registry_find(&r,later) + _,not_built:=registry_read(&r,"props/crate/later.tex") + testing.expect(t,!by_id && not_built==.Not_Found) missing:=make([dynamic]Registry_Missing,context.temp_allocator) registry_missing(&r,&missing) - testing.expect(t,!pending && len(missing)==0,"an output that is not built is not listed, and one in the imported folder is not missing") + testing.expect_value(t,len(missing),0) + folders:=[]string{"plugins/","plugins/pack/","props/","props/crate/"} + testing.expectf(t,len(r.refused)==0 && len(r.folders)==len(folders),"%v",r.folders[:]) + for f,at in r.folders[:min(len(folders),len(r.folders))] {testing.expect_value(t,f,folders[at])} + testing.expect_value(t,registry_memory_set(&r,"props/crate/wood.tex",data),Error.Invalid) + testing.expect_value(t,registry_memory_remove(&r,"props/crate/wood.tex"),Error.Invalid) if pass==0 {continue} - // The source moves with its sidecar: its assets are listed there once the editor refreshes what moved, as they are. - for end in ([2]string{"",SIDECAR_EXTENSION}) {testing.expect(t,os.rename(fmt.tprintf("%s/props/crate.glb%s",root,end),fmt.tprintf("%s/kit/box.glb%s",root,end))==nil)} - testing.expect_value(t,registry_refresh(&r,{"props/crate.glb","kit/box.glb"+SIDECAR_EXTENSION}),Error.None) - moved,again:=registry_find(&r,body) - testing.expect(t,again && moved.path=="kit/crate/body.tex" && moved.imported==0x22,"the asset is listed where its source is, with its id and its file") - alone,listed:=registry_find(&r,other) - testing.expect(t,listed && alone.path=="props/crate/body.tex","a path no other output has takes no source name") + + // A refresh keeps what is listed, also what it names. + revision:=r.revision + testing.expect_value(t,registry_refresh(&r,{"props/crate/wood.tex","props/crate/later.tex",renamed,"plugins/pack/atlas.tex","props/crate/Wood.mat"}),Error.None) + testing.expect(t,r.revision==revision && r.stats.files==6 && r.stats.opened==0) + _,kept:=registry_find(&r,wood) + _,kept_pending:=registry_find_path(&r,"props/crate/later.tex") + testing.expect(t,kept && kept_pending) + // A file written where an output is listed takes the path over, through a scan. + write_texture(t,&types,fmt.tprintf("%s/props/crate/wood.tex",root),{11,11}) + testing.expect_value(t,registry_refresh(&r,{"props/crate/wood.tex"}),Error.None) + moved,again:=registry_find(&r,wood) + written,taken:=registry_find(&r,{11,11}) + testing.expect(t,again && moved.path==fmt.tprintf("props/crate/wood (%s).tex",base.asset_id_format(wood,buffer[:])) && taken && written.path=="props/crate/wood.tex" && !written.listed) + os.remove(fmt.tprintf("%s/props/crate/wood.tex",root)) + } +} + +// Entries no package list makes are refused or reported, never a crash nor +// an error of the scan: a path that is empty, leaves the folder or is one a +// scan leaves out, and a zero id are refused; an output not built is Not_Built; +// one that is not the asset its entry names is Broken; two entries with one +// path are told apart by id. +@(test) +registry_refuses_or_reports_hostile_listed_entries :: proc(t:^testing.T) { + types:Types;types_init(&types);defer types_destroy(&types) + folder:=base.test_folder(t,"registry-listed-hostile");if folder=="" {return} + root:=fmt.tprintf("%s/assets",folder) + os.make_directory_all(root) + s,open_error:=store.open(fmt.tprintf("%s/store",folder),context.temp_allocator) + if !testing.expect_value(t,open_error,store.Error.None) {return} + put :: proc(t:^testing.T,s:^store.Store,types:^Types,path:string,id:base.Asset_Id) -> store.Hash { + write_texture(t,types,path,id) + object,_:=object_put_file(s,path) + return object + } + texture,other:=put(t,&s,&types,fmt.tprintf("%s/texture.tex",folder),{1,1}),put(t,&s,&types,fmt.tprintf("%s/other.tex",folder),{2,2}) + text,_:=store.put(&s,transmute([]u8)string("not an asset")) + store.commit(&s) + entry :: proc(s:^store.Store,path:string,object:store.Hash,id:base.Asset_Id) -> Package_Listed { + listed:=Package_Listed{path=path,object=object,id=id,kind=KIND_TEXTURE,state=object!={}?.Built:.Not_Built} + dependencies:=make([dynamic]base.Asset_Id,context.temp_allocator) + listed.header,_=object_header(s,object,&dependencies) + listed.dependencies=dependencies[:] + return listed + } + entries:=[]Package_Listed{ + entry(&s,"",{},{3,3}),entry(&s,"../out.tex",texture,{1,1}),entry(&s,"/abs.tex",texture,{1,1}),entry(&s,"a\\b.tex",texture,{1,1}), + entry(&s,"a/./b.tex",texture,{1,1}),entry(&s,".hidden/c.tex",texture,{1,1}),entry(&s,"zero.tex",texture,{}),entry(&s,"c:x.tex",texture,{1,1}), + entry(&s,"gone.tex",{},{4,4}),entry(&s,"text.tex",text,{5,5}),entry(&s,"liar.tex",other,{6,6}), + entry(&s,"twice.tex",texture,{1,1}),entry(&s,"twice.tex",{},{7,7}),entry(&s,"twice.tex",{},{7,7}), + } + r:Registry;registry_init(&r,root,listed=entries);defer registry_destroy(&r) + testing.expect_value(t,registry_scan(&r),Error.None) + testing.expectf(t,len(r.refused)==9,"refused: %v",r.refused[:]) + buffer:[base.ASSET_ID_TEXT]u8 + State :: struct {path:string,state:Registry_State} + for want in ([]State{{"gone.tex",.Not_Built},{"text.tex",.Broken},{"liar.tex",.Broken},{"twice.tex",.Asset},{fmt.tprintf("twice (%s).tex",base.asset_id_format({7,7},buffer[:])),.Not_Built}}) { + file,found:=registry_find_path(&r,want.path) + testing.expectf(t,found && file.state==want.state && file.listed,"%s is %v, not %v",want.path,want.state,found?file^:Registry_File{}) } + testing.expect_value(t,len(r.files),5) + _,error:=registry_read(&r,"gone.tex") + testing.expect_value(t,error,Error.Not_Found) } diff --git a/asset/sidecar.odin b/asset/sidecar.odin deleted file mode 100644 index 341ca0b0a..000000000 --- a/asset/sidecar.odin +++ /dev/null @@ -1,279 +0,0 @@ -package asset - -import "base:runtime" -import "core:fmt" -import "core:strings" -import "mirai:std/base" -import "mirai:std/serial" - -// .import, the sidecar of a source of a project or a plugin, beside -// it in its assets folder: the settings its import applies (read by -// cook/import with the schema of its Import_Settings), the folder of the -// source's folder its outputs go to, the files the import reads besides the -// source and one output per asset the source gives, with its id. What nobody -// edits (import_cached) is no file of the assets but of the imported folder -// that goes with them, -, never versioned: the registry -// lists it at //, so a source -// that moves with its sidecar takes it along, and one that is gone takes it -// out of the listing. - -SIDECAR_EXTENSION :: ".import" -SIDECAR_VERSION :: 4 - -// One asset an import gives, by what it is in the source. -Import_Output :: struct { - kind:Fourcc, - name:string, // in the source: the mesh, the material, the image - index:int, // among those of its kind in the source - id:base.Asset_Id, - part:bool, // of the cache: it is a part of the root of the import, decided when it was first imported - // Of a document the import writes over: the content_hash of its header - // as the import last wrote it, so a file that holds anything else was - // changed since. 0: none, for what the import keeps changed itself (a - // material follows its source in the fields nobody changed). - written:u64, -} - -Sidecar :: struct { - settings:string, // the text of the settings object, "" without one - folder:string, // under the folder of the source, where its outputs go; "": beside it - inputs:[dynamic]string, // the files the import read besides the source, from the folder of the source - outputs:[dynamic]Import_Output, -} - -// Whether what an import builds of a kind is of the cache: nothing edits it -// after the import, so the source and its sidecar give it back whole. -import_cached :: proc "contextless" (kind:Fourcc) -> bool { - return kind==KIND_MESH || kind==KIND_TEXTURE || kind==KIND_SOUND || kind==KIND_FONT -} - -// The file of an imported asset: in folder, by its id and what it was built from. -imported_file :: proc(folder:string,id:base.Asset_Id,key:u64,extension:string,allocator:=context.allocator) -> string { - buffer:[base.ASSET_ID_TEXT]u8 - digits:[16]u8 - file,error:=strings.concatenate({folder,"/",base.asset_id_format(id,buffer[:]),"-",fmt.bprintf(digits[:],"%016x",key),extension},allocator) - if error!=nil {return ""} - return file -} - -// The id and the key a file name of an imported folder says. -imported_name :: proc(name:string) -> (id:base.Asset_Id,key:u64,ok:bool) { - if len(name)<49 || name[32]!='-' {return} - id=serial.text_parse_asset_id(name[:32]) or_return - key=serial.text_parse_hex(name[33:49]) or_return - return id,key,true -} - -// The text of a sidecar whose settings are already the text of their object. -sidecar_text :: proc(settings:string,sidecar:Sidecar,allocator:=context.allocator) -> string { - w:serial.Writer - serial.writer_init(&w,1024,allocator) - serial.text_write_raw(&w,"{\n \"version\": ") - serial.text_write_int(&w,SIDECAR_VERSION) - serial.text_write_raw(&w,",\n \"settings\": ") - serial.text_write_raw(&w,settings) - if sidecar.folder!="" { - serial.text_write_raw(&w,",\n \"folder\": ") - serial.text_write_string(&w,sidecar.folder) - } - if len(sidecar.inputs)>0 { - serial.text_write_raw(&w,",\n \"inputs\": [") - for input,at in sidecar.inputs { - if at>0 {serial.text_write_raw(&w,", ")} - serial.text_write_string(&w,input) - } - serial.text_write_raw(&w,"]") - } - serial.text_write_raw(&w,",\n \"outputs\": [") - for output,at in sidecar.outputs { - buffer:[4]u8 - serial.text_write_raw(&w,at>0?",\n {\"kind\": ":"\n {\"kind\": ") - serial.text_write_string(&w,strings.trim_right(fourcc_text(output.kind,&buffer)," ")) - serial.text_write_raw(&w,", \"name\": ") - serial.text_write_string(&w,output.name) - serial.text_write_raw(&w,", \"index\": ") - serial.text_write_int(&w,i64(output.index)) - serial.text_write_raw(&w,", \"id\": \"") - serial.text_write_asset_id(&w,output.id) - serial.text_write_raw(&w,"\"") - if output.part {serial.text_write_raw(&w,", \"part\": true")} - if output.written!=0 { - digits:[16]u8 - serial.text_write_raw(&w,", \"written\": ") - serial.text_write_string(&w,fmt.bprintf(digits[:],"%016x",output.written)) - } - serial.text_write_raw(&w,"}") - } - serial.text_write_raw(&w,len(sidecar.outputs)>0?"\n ]\n}\n":"]\n}\n") - return string(serial.writer_take(&w)) -} - -// False for a text that is not a sidecar of this version: a field or a value -// it does not know, a folder that leaves the folder of the source, an output -// without an id or kind, a part or a record of what was written that its -// kind does not take, twice the same id or the same output. What it holds -// lives in allocator, an arena: nothing frees it piece by piece; settings is -// a view of text. -sidecar_parse :: proc(text:string,allocator:=context.allocator) -> (sidecar:Sidecar,ok:bool) { - r:serial.Text_Reader - serial.text_reader_init(&r,text,context.temp_allocator) - defer serial.text_reader_destroy(&r) - sidecar.inputs=make([dynamic]string,allocator) - sidecar.outputs=make([dynamic]Import_Output,allocator) - version:=u64(0) - serial.text_expect(&r,'{') or_return - for first:=true;!serial.text_accept(&r,'}');first=false { - if !first {serial.text_expect(&r,',') or_return} - // The key is a view the next string replaces. - field:=strings.clone(serial.text_string(&r) or_return,context.temp_allocator) - serial.text_expect(&r,':') or_return - switch field { - case "version": - negative,magnitude:=serial.text_integer(&r) or_return - if negative {return} - version=magnitude - case "settings": - serial.text_space(&r) - start:=r.offset - if serial.text_peek(&r)!='{' {return} - serial.text_skip(&r) or_return - sidecar.settings=text[start:r.offset] - case "folder": - no_room:runtime.Allocator_Error - if sidecar.folder,no_room=strings.clone(serial.text_string(&r) or_return,allocator);no_room!=nil {return} - case "inputs": - serial.text_expect(&r,'[') or_return - for begun:=false;!serial.text_accept(&r,']');begun=true { - if begun {serial.text_expect(&r,',') or_return} - input,no_room:=strings.clone(serial.text_string(&r) or_return,allocator) - if no_room!=nil {return} - append(&sidecar.inputs,input) - } - case "outputs": - serial.text_expect(&r,'[') or_return - for begun:=false;!serial.text_accept(&r,']');begun=true { - if begun {serial.text_expect(&r,',') or_return} - append(&sidecar.outputs,output_read(&r,allocator) or_return) - } - case: - return - } - } - serial.text_end(&r) or_return - if version!=SIDECAR_VERSION || !folder_valid(sidecar.folder) {return} - for output,at in sidecar.outputs { - if output.id=={} || output.kind==0 || output.part && !import_cached(output.kind) || output.written!=0 && import_cached(output.kind) {return} - for other in sidecar.outputs[:at] { - if other.id==output.id || other.kind==output.kind && other.index==output.index && other.name==output.name {return} - } - } - return sidecar,true -} - -@(private="file") -output_read :: proc(r:^serial.Text_Reader,allocator:=context.allocator) -> (output:Import_Output,ok:bool) { - serial.text_expect(r,'{') or_return - for first:=true;!serial.text_accept(r,'}');first=false { - if !first {serial.text_expect(r,',') or_return} - field:=strings.clone(serial.text_string(r) or_return,context.temp_allocator) - serial.text_expect(r,':') or_return - switch field { - case "kind": - text:=serial.text_string(r) or_return - if len(text)==0 || len(text)>4 {return} - bytes:=[4]u8{' ',' ',' ',' '} - copy(bytes[:],text) - output.kind=Fourcc(u32(bytes[0])|u32(bytes[1])<<8|u32(bytes[2])<<16|u32(bytes[3])<<24) - case "name": - no_room:runtime.Allocator_Error - if output.name,no_room=strings.clone(serial.text_string(r) or_return,allocator);no_room!=nil {return} - case "index": - negative,magnitude:=serial.text_integer(r) or_return - if negative || magnitude>u64(max(i32)) {return} - output.index=int(magnitude) - case "id": - output.id=serial.text_parse_asset_id(serial.text_string(r) or_return) or_return - case "part": - output.part=serial.text_keyword(r,"true") - if !output.part && !serial.text_keyword(r,"false") {return} - case "written": - text:=serial.text_string(r) or_return - if len(text)!=16 {return} - output.written=serial.text_parse_hex(text) or_return - if output.written==0 {return} - case: - return - } - } - return output,true -} - -// A folder under the folder of a source: relative, with forward slashes, -// and never out of it. -@(private="file") -folder_valid :: proc(folder:string) -> bool { - if folder=="" {return true} - if folder[0]=='/' || strings.contains_any(folder,"\\:") {return false} - rest:=folder - for part in strings.split_iterator(&rest,"/") { - if part=="" || part=="." || part==".." {return false} - } - return true -} - -// The paths the registry lists the cache outputs of a sidecar at, under -// listed (the path of the sidecar's folder in the registry, "" or ending -// with '/'): / with the name as safe_path gives it, -// and " ()" before the extension for an output whose kind and name one -// of a lower index has. "" for an output that is not of the cache. In the -// temp allocator. -sidecar_paths :: proc(sidecar:Sidecar,listed:string) -> []string { - paths:=make([]string,len(sidecar.outputs),context.temp_allocator) - folder:=sidecar.folder!=""?fmt.tprintf("%s%s/",listed,sidecar.folder):listed - for output,at in sidecar.outputs { - info,known:=kind_info(output.kind) - if !known || !import_cached(output.kind) {continue} - name:=safe_path(output.name) - repeated:=false - for other in sidecar.outputs {repeated||=other.kind==output.kind && other.name==output.name && other.index string { - cleaned:=name_clean(name) - return cleaned if cleaned!="" else "unnamed" -} - -// Where under the folder of an import the output named name goes: each part -// between '/' as safe_name makes it, without the parts that are empty, "." -// or "..". "unnamed" when no part is left. In the temp allocator. -safe_path :: proc(name:string) -> string { - b:=strings.builder_make(context.temp_allocator) - rest:=name - for part in strings.split_iterator(&rest,"/") { - cleaned:=name_clean(part) - if cleaned=="" {continue} - if strings.builder_len(b)>0 {strings.write_byte(&b,'/')} - strings.write_string(&b,cleaned) - } - return strings.to_string(b) if strings.builder_len(b)>0 else "unnamed" -} - -@(private="file") -name_clean :: proc(name:string) -> string { - b:=strings.builder_make(context.temp_allocator) - for r in name { - switch { - case r<32 || r==127,r=='<',r=='>',r==':',r=='"',r=='/',r=='\\',r=='|',r=='?',r=='*',r=='#': - strings.write_byte(&b,'_') - case: - strings.write_rune(&b,r) - } - } - return strings.trim(strings.to_string(b)," .") -} diff --git a/asset/texture.odin b/asset/texture.odin index a897a3b7a..63dd49c89 100644 --- a/asset/texture.odin +++ b/asset/texture.odin @@ -196,7 +196,8 @@ texture_tables :: proc(doc:^Document,types:^Types,info:^Texture_Info,mips:[]Text #reverse for run in desc.layouts {at+=copy(bytes[at:],run.payload)} } else { for level,i in desc.mips {copy(bytes[mips[i].offset:],level)} - if doc.codecs!=.None && texture_fields_format(desc.format) && .Fast not_in desc.flags && (texture_fields_smaller(desc.format,mips,bytes,doc.codecs,doc.allocator) or_return) { + // Without a compressor to ask, planes are what compresses smaller where the bytes rest. + if texture_fields_format(desc.format) && .Fast not_in desc.flags && (doc.codecs==.None || (texture_fields_smaller(desc.format,mips,bytes,doc.codecs,doc.allocator) or_return)) { info.flags+={.Fields} for level,i in desc.mips {texture_fields_split(desc.format,level,bytes[mips[i].offset:][:mips[i].size])} } @@ -370,21 +371,31 @@ texture_decode :: proc(a:^Asset,types:^Types,view:^Texture_View,from:int) -> Err @(private="file") texels_read :: proc(a:^Asset,table:^Asset_Table,column:int,size:int) -> (data:[]u8,owned:bool,error:Error) { if size==0 {return nil,false,.None} - entry:=blob_chunk(&a.blob,table.columns[column].chunk) + chunk:=table.columns[column].chunk + entry:=blob_chunk(&a.blob,chunk) + total:=int(entry.uncompressed_size) + if total!=table.rows || size>total {return nil,false,.Corrupt} + // Of the store, only the chunks that hold the first size bytes are read. + if entry.compression==.None && a.object!=nil && !table.columns[column].ready && sizeu64(len(a.blob.data)) {return nil,false,.Truncated} - total:=int(entry.uncompressed_size) - if total!=table.rows || size>total {return nil,false,.Corrupt} decoded:=total // A stream restores whole pieces. if size [--out ] - mirai ui|scene write - mirai ui|scene check [...] - mirai texts list | missing | set | export | import ` +COMMAND_USAGE :: `Usage: mirai ui|scene read [--out ] [--store ] + mirai ui|scene write [--store ] + mirai ui|scene check [...] [--store ] + mirai texts list | missing | set | export | import [--store ]` // What a command reads and writes: the interfaces, or the scenes. Document_Kind :: enum u8 {Ui,Scene} @@ -91,24 +91,29 @@ Project :: struct { untyped:string, // why it has none, with what gives them recipes:[]Recipe, // of the engine and of its active plugins scratch:Scratch_World, // where a recipe is run; made the first time + package_problems:[]string, // what the packages of the project and its plugins could not list (host.project_assets) } // What builds the library the types of a game come from. BUILD_FIX :: "mirai build editor builds it" -// Opens the project at root with the assets of its plugins. library false -// leaves the types of its game out. problem: why it does not open. -project_open :: proc(p:^Project,root,engine:string,library:=true) -> (problem:string) { +// Opens the project at root with the assets of its plugins and of the +// packages of both, from the store at store_root ("": the machine's). library +// false leaves the types of its game out. problem: why it does not open. +project_open :: proc(p:^Project,root,engine:string,library:=true,store_root:="") -> (problem:string) { opened,error:=project.open(root,engine,context.temp_allocator) if error.code!=.None {return project.error_text(root,error,context.temp_allocator)} p.root=strings.clone(opened.root) p.name=strings.clone(base.text_get(opened.manifest.name[:])) p.assets_root=project.folder_path(p.root,.Assets) - asset.registry_init(&p.registry,p.assets_root,imported=project.imported_path(p.root,context.temp_allocator)) - for mount in host.project_asset_mounts(&opened,context.temp_allocator) {asset.registry_mount(&p.registry,mount.folder,mount.prefix,mount.imported)} + found,listed:=host.project_assets(&opened,store_root,context.temp_allocator) + if !listed {return "out of memory to list its assets"} + asset.registry_init(&p.registry,p.assets_root,listed=found.listed) + for mount in found.mounts {asset.registry_mount(&p.registry,mount.folder,mount.prefix,mount.listed)} + p.package_problems=make([]string,len(found.problems)) + for problem,i in found.problems {p.package_problems[i]=strings.clone(problem)} asset.types_init(&p.types) - p.types.codecs=asset.project_codecs(root,engine) p.assets={&p.registry,&p.types} if os.is_dir(p.assets_root) && asset.registry_scan(&p.registry)!=.None {return fmt.tprintf("the assets of %s could not be read",p.assets_root)} ui_settings:gui.UI_Settings @@ -130,6 +135,8 @@ project_close :: proc(p:^Project) { delete(p.assets_root) delete(p.source) delete(p.untyped) + for problem in p.package_problems {delete(problem)} + delete(p.package_problems) p^={} } @@ -341,14 +348,18 @@ run_fail :: proc(run:^Run,what,why:string) { } command :: proc(args:[]string,output,errors:io.Writer) -> int { - engine:string + engine,store_root:string roots:=make([dynamic]string,context.temp_allocator) rest:=make([dynamic]string,context.temp_allocator) for at:=0;at=len(args) {fmt.wprintln(errors,COMMAND_USAGE);return 2} - if args[at]=="--project" {append(&roots,args[at+1])} else {engine=args[at+1]} + switch args[at] { + case "--project": append(&roots,args[at+1]) + case "--engine": engine=args[at+1] + case "--store": store_root=args[at+1] + } at+=1 case: append(&rest,args[at]) } @@ -361,14 +372,15 @@ command :: proc(args:[]string,output,errors:io.Writer) -> int { return 2 } if len(roots)==0 {append(&roots,".")} - root:=project_choose(roots[:],engine,rest[2] if len(rest)>2 else "") + root:=project_choose(roots[:],engine,store_root,rest[2] if len(rest)>2 else "") p:Project reads:=kind==.Ui && verb=="read" || rest[0]=="texts" - if problem:=project_open(&p,root,engine,!reads);problem!="" { + if problem:=project_open(&p,root,engine,!reads,store_root);problem!="" { fmt.wprintfln(errors,"mirai %s %s: %s",rest[0],verb,problem) return 1 } defer project_close(&p) + for problem in p.package_problems {fmt.wprintfln(errors,"mirai %s %s: %s",rest[0],verb,problem)} if rest[0]=="texts" {return texts_command(&p,rest[1:],output,errors)} return project_run(&p,kind,verb,rest[2:],output,errors) } @@ -376,7 +388,7 @@ command :: proc(args:[]string,output,errors:io.Writer) -> int { // Of several projects, the one whose assets have the asset of a file: the // first when none has it, or named is no file. @(private="file") -project_choose :: proc(roots:[]string,engine,named:string) -> string { +project_choose :: proc(roots:[]string,engine,store_root,named:string) -> string { if len(roots)==1 || !os.is_file(named) {return roots[0]} file:asset.Asset if asset.asset_load(&file,named,context.temp_allocator)!=.None {return roots[0]} @@ -384,7 +396,7 @@ project_choose :: proc(roots:[]string,engine,named:string) -> string { asset.asset_close(&file) for root in roots { p:Project - problem:=project_open(&p,root,engine,false) + problem:=project_open(&p,root,engine,false,store_root) _,has:=asset.registry_find(&p.registry,id) project_close(&p) if problem=="" && has {return root} diff --git a/cook/build/binary_copy.odin b/cook/build/binary_copy.odin new file mode 100644 index 000000000..3e3a127f5 --- /dev/null +++ b/cook/build/binary_copy.odin @@ -0,0 +1,39 @@ +package build + +import "core:fmt" +import "core:io" +import "core:math/rand" +import "core:os" +import "core:strings" + +@(private) +BINARY_TEMP_PREFIX :: ".mirai-copy-" + +@(private) +copy_binary :: proc(to,from:string) -> (ok:bool) { + folder,name:=os.split_path(to) + if name=="" || strings.contains(to,"\x00") || strings.contains(from,"\x00") {return false} + source,open_error:=os.open(from) + if open_error!=nil {return false} + defer os.close(source) + info,stat_error:=os.fstat(source,context.temp_allocator) + if stat_error!=nil || info.type!=.Regular {return false} + staged,path_ok:=binary_temp_path(folder) + if !path_ok {return false} + output,create_error:=os.open(staged,{.Write,.Create,.Excl},info.mode&os.Permissions_All) + if create_error!=nil {return false} + defer if !ok {_=os.remove(staged)} + count,copy_error:=io.copy(os.to_writer(output),os.to_reader(source)) + close_error:=os.close(output) + if copy_error!=nil || close_error!=nil || count!=info.size {return false} + return publish(staged,to) +} + +@(private) +binary_temp_path :: proc(folder:string) -> (string,bool) { + digits:[16]u8 + suffix:=fmt.bprintf(digits[:],"%016x",rand.uint64()) + parent:=folder==""?".":folder + path,error:=strings.concatenate({parent,"/",BINARY_TEMP_PREFIX,suffix},context.temp_allocator) + return path,error==nil +} diff --git a/cook/build/binary_copy_darwin_test.odin b/cook/build/binary_copy_darwin_test.odin new file mode 100644 index 000000000..d2fd76028 --- /dev/null +++ b/cook/build/binary_copy_darwin_test.odin @@ -0,0 +1,51 @@ +#+build darwin +package build + +import "core:fmt" +import "core:os" +import "core:testing" +import "core:time" +import "mirai:project" +import "mirai:std/base" +import "mirai:std/subprocess" + +@(test) +a_replaced_binary_launches_while_the_previous_one_runs :: proc(t:^testing.T) { + folder:=base.test_folder(t,"binary-launch") + if folder=="" {return} + a:=context.temp_allocator + compiler,found:=compiler_find(".",a) + if !testing.expect(t,found) {return} + source:=project.path(folder,"fixture.odin",allocator=a) + code:=`package fixture +import "core:time" +main :: proc() {when #config(HOLD,false) {time.sleep(30*time.Second)}} +` + if !testing.expect(t,os.write_entire_file(source,code)==nil) {return} + old:=project.path(folder,"old",allocator=a) + new:=project.path(folder,"new",allocator=a) + for output,i in ([?]string{old,new}) { + command:=[]string{compiler,"build",source,"-file",fmt.tprintf("-out:%s",output), + i==0?"-define:HOLD=true":"-define:HOLD=false"} + state,_,stderr,error:=subprocess.exec({command=command},a) + if !testing.expectf(t,error==nil && state.exit_code==0,"%v: %s",error,stderr) {return} + } + published:=project.path(folder,"program",allocator=a) + if !testing.expect(t,copy_binary(published,old)) {return} + first,start_error:=subprocess.start({command={published}}) + if !testing.expectf(t,start_error==nil,"first launch: %v",start_error) {return} + defer subprocess.release(&first) + first_state,first_wait:=subprocess.wait(&first,0) + if !testing.expectf(t,first_wait==os.General_Error.Timeout, + "first process: %v, %v",first_state,first_wait) {return} + + if !testing.expect(t,copy_binary(published,new)) {return} + second,restart_error:=subprocess.start({command={published}}) + if !testing.expectf(t,restart_error==nil,"second launch: %v",restart_error) {return} + defer subprocess.release(&second) + state,wait_error:=subprocess.wait(&second,5*time.Second) + testing.expectf(t,wait_error==nil && state.exited && state.exit_code==0, + "second process: %v, %v",state,wait_error) + _,still_running:=subprocess.wait(&first,0) + testing.expect(t,still_running==os.General_Error.Timeout) +} diff --git a/cook/build/binary_copy_test.odin b/cook/build/binary_copy_test.odin new file mode 100644 index 000000000..bc9cafb46 --- /dev/null +++ b/cook/build/binary_copy_test.odin @@ -0,0 +1,49 @@ +package build + +import "core:mem/virtual" +import "core:os" +import "core:testing" +import "mirai:project" +import "mirai:std/base" + +@(test) +binary_copy_replaces_the_file_and_keeps_the_old_mapping :: proc(t:^testing.T) { + folder:=base.test_folder(t,"binary-copy") + if folder=="" {return} + a:=context.temp_allocator + published:=project.path(folder,"published.bin",allocator=a) + source:=project.path(folder,"source.bin",allocator=a) + if !testing.expect(t,os.write_entire_file(published,"old binary")==nil) {return} + if !testing.expect(t,os.write_entire_file(source,"new binary")==nil) {return} + old,stat_error:=binary_copy_info(published) + if !testing.expect(t,stat_error==nil && old.inode!=0) {return} + held,map_error:=virtual.map_file(published,{.Read}) + if !testing.expect(t,map_error==nil && raw_data(held)!=nil) {return} + defer virtual.unmap_file(held) + if !testing.expect(t,copy_binary(published,source)) {return} + current,current_error:=binary_copy_info(published) + if !testing.expect(t,current_error==nil) {return} + testing.expect(t,current.inode!=old.inode) + testing.expect_value(t,string(held),"old binary") + + blocked:=project.path(folder,"directory",allocator=a) + if !testing.expect(t,os.make_directory(blocked)==nil) {return} + before,before_error:=os.read_all_directory_by_path(folder,a) + if !testing.expect(t,before_error==nil) {return} + testing.expect(t,!copy_binary(published,project.path(folder,"missing",allocator=a))) + testing.expect(t,!copy_binary(blocked,source)) + after,after_error:=os.read_all_directory_by_path(folder,a) + testing.expect(t,after_error==nil && len(after)==len(before) && os.is_dir(blocked)) + contents,read_error:=os.read_entire_file(published,a) + testing.expect(t,read_error==nil) + testing.expect_value(t,string(contents),"new binary") + remaining,remaining_error:=binary_copy_info(published) + testing.expect(t,remaining_error==nil && remaining.inode==current.inode) +} + +binary_copy_info :: proc(path:string) -> (os.File_Info,os.Error) { + file,error:=os.open(path) + if error!=nil {return {},error} + defer os.close(file) + return os.fstat(file,context.temp_allocator) +} diff --git a/cook/build/binary_copy_windows_test.odin b/cook/build/binary_copy_windows_test.odin new file mode 100644 index 000000000..321f1f93e --- /dev/null +++ b/cook/build/binary_copy_windows_test.odin @@ -0,0 +1,36 @@ +#+build windows +package build + +import "core:os" +import win "core:sys/windows" +import "core:testing" +import "mirai:project" +import "mirai:std/base" + +@(test) +a_loaded_library_is_replaced_and_retired_until_it_is_free :: proc(t:^testing.T) { + folder:=base.test_folder(t,"binary-loaded") + if folder=="" {return} + a:=context.temp_allocator + published:=project.path(folder,"published.dll",allocator=a) + retired:=project.path(folder,ENGINE_RETIRED,allocator=a) + source:="C:/Windows/System32/version.dll" + if !testing.expect(t,copy_binary(published,source)) {return} + old,stat_error:=binary_copy_info(published) + if !testing.expect(t,stat_error==nil) {return} + { + handle:=win.LoadLibraryW(win.utf8_to_wstring(published,a)) + if !testing.expect(t,handle!=nil) {return} + defer win.FreeLibrary(handle) + if !testing.expect(t,copy_binary(published,source)) {return} + current,current_error:=binary_copy_info(published) + testing.expect(t,current_error==nil && current.inode!=old.inode) + engine_retired_sweep(retired) + moved,moved_error:=os.read_all_directory_by_path(retired,a) + if !testing.expect(t,moved_error==nil && len(moved)==1) {return} + held,held_error:=binary_copy_info(moved[0].fullpath) + testing.expect(t,held_error==nil && held.inode==old.inode) + } + engine_retired_sweep(retired) + testing.expect(t,!os.exists(retired)) +} diff --git a/cook/build/build.odin b/cook/build/build.odin index f45fded22..abf9dc326 100644 --- a/cook/build/build.odin +++ b/cook/build/build.odin @@ -12,6 +12,7 @@ import "mirai:asset" import "mirai:project" import "mirai:std/base" import "mirai:std/subprocess" +import "mirai:store" // Builds of a project. A job generates the world and the main package of // its target and runs the compiler without blocking; whoever started it polls @@ -77,6 +78,7 @@ Desc :: struct { action:Action, // Check and Test: of the Editor target test_package:string, // Test: the folder of the package, absolute test:Test_Options, // Test: how its executable runs + store_root:string, // the store the packages of the project and its plugins are refreshed in and listed from; "": the machine's (store.default_root) allocator:runtime.Allocator, // of the job. Unset: context.allocator } @@ -113,6 +115,7 @@ Job :: struct { engine:Engine_Mode, imports:[]string, // Build: the roots whose sources it imports once the target stands (imports_roots) import_program:string, // Build: that of the engine of the project's defines, which imports them + store_root:string, // Desc.store_root, or the machine's compiler:string, flags:[]string, // what identifies the build besides its sources staged:string, // what the compiler writes @@ -124,6 +127,7 @@ Job :: struct { copies:[]Copy, // Game: what goes next to the executable settings:string, // Game: its settings file: those of the engine and of the project merged, since it ships without the engine cook:Cook_Desc, // Game: the assets it ships; without assets, a project that has none + store:store.Store, // Game: the store of store_root, which cook.store names; unopened: none plugin_assets:[dynamic]project.Asset_Folder, // Game: the folders of assets of its active plugins, in cook.plugins development:[dynamic]string, // Game: the folders of the active development plugins, which it must not reach plugins:[]Plugin_Library, // the plugin libraries the target links (plugins.odin) @@ -150,7 +154,7 @@ Job :: struct { } @(private) -Copy :: struct {from,to:string,folder:bool} +Copy :: struct {from,to:string,folder,binary:bool} // How long job_wait waits on one compiler before it looks at the others. @(private) @@ -219,13 +223,14 @@ job_start :: proc(desc:Desc) -> ^Job { } } job.engine=desc.engine + job.store_root=strings.clone(desc.store_root!=""?desc.store_root:store.default_root(context.temp_allocator),a) if desc.action==.Build { roots:=imports_roots(p) job.imports=make([]string,len(roots),a) for root,i in roots {job.imports[i]=strings.clone(root,a)} job.import_program=project_engine(p,desc.engine,a).import_program } - generated,written:=generated_write(p,desc.target) + generated,written:=generated_write(p,desc.target,job.store_root) if !written { if len(generated.errors)>0 { job.result.text=strings.join(generated.errors,"\n",a) @@ -260,8 +265,14 @@ job_start :: proc(desc:Desc) -> ^Job { if p.plugins[index].manifest.development {append(&job.development,strings.clone(p.plugins[index].folder,a))} } if assets:=project.folder_path(p.root,.Assets,a);os.is_dir(assets) { - job.cook={assets=assets,output=project.folder_path(folder,.Assets,a),manifest=project.path(p.root,COOK_FILE,allocator=a),texture_cache=project.content_cache_path(p.root,project.TEXTURE_CACHE,allocator=a),mesh_cache=project.content_cache_path(p.root,project.MESH_CACHE,allocator=a),collision_cache=project.content_cache_path(p.root,project.COLLISION_CACHE,allocator=a),root=p.root,server=desc.server,quick=desc.dev,keys=project.path(job.work,fmt.tprintf("%s.cook",name),allocator=a),allocator=a} + job.cook={assets=assets,output=project.folder_path(folder,.Assets,a),manifest=project.path(p.root,COOK_FILE,allocator=a),collision_cache=project.content_cache_path(p.root,project.COLLISION_CACHE,allocator=a),root=p.root,server=desc.server,quick=desc.dev,keys=project.path(job.work,fmt.tprintf("%s.cook",name),allocator=a),allocator=a} job.cook.codecs=asset.project_codecs(p.root,p.engine_root) + if job.store_root!="" { + if opened,error:=store.open(job.store_root,a);error==.None { + job.store=opened + job.cook.store=&job.store + } + } job.plugin_assets=make([dynamic]project.Asset_Folder,a) for found in project.asset_folders(p,a) { if found.prefix!="" {append(&job.plugin_assets,found)} @@ -376,9 +387,7 @@ job_plugins_plan :: proc(job:^Job,p:^project.Project,desc:Desc) -> Failure { @(private) job_plugin_restore :: proc(job:^Job,index:int) -> bool { lib:=&job.plugins[index] - if os.make_directory_all(job.work)!=nil && !os.is_dir(job.work) {return false} - staged:=project.path(job.work,strings.concatenate({job_plugin_name(job,index),project.LIBRARY_SUFFIX},context.temp_allocator),allocator=context.temp_allocator) - if !plugin_cache_take(job.engine_root,lib,job.plugin_hashes[index],staged) || !publish(staged,lib.library,job_plugin_retired(job,lib)) {return false} + if !plugin_cache_take(job.engine_root,lib,job.plugin_hashes[index]) {return false} job_plugin_stamp(job,index) job.plugin_published[index]=true job.result.plugins_built+=1 @@ -535,7 +544,8 @@ job_project_start :: proc(job:^Job) -> ^Job { // The game built for another folder is the same executable. if current==saved && !present && job.target==.Game && built!="" && !project.path_same(built,job.published) && os.is_file(built) { folder:=job.published[:strings.last_index_byte(job.published,'/')] - present=(os.make_directory_all(folder)==nil || os.is_dir(folder)) && os.copy_file(job.published,built)==nil + present=(os.make_directory_all(folder)==nil || os.is_dir(folder)) && + copy_binary(job.published,built) } if present && len(job.compiles)>0 { job.project_stands=true @@ -878,14 +888,14 @@ job_run :: proc(desc:Desc) -> ^Job { return job } -// What the imports of the project keep outside version control, from its -// sources, once the target stands: the library of its editor, built, has the -// formats its plugins add. False, with the job failed, when a source did not -// import. +// The packages of the project and its plugins brought to builds this engine +// reads, once the target stands: the library of its editor, built, has the +// formats its plugins add. A build that cooks requires Final textures. False, +// with the job failed, when a package has no build and could not get one. @(private) job_imports :: proc(job:^Job) -> bool { if len(job.imports)==0 {return true} - ok,message:=imports_sync(job.import_program,job.imports,final=job.cook.assets!="") + ok,message:=imports_refresh(job.import_program,job.imports,job.cook.assets!="",job.store_root) if !ok { job.result.text=strings.clone(message,virtual.arena_allocator(&job.arena)) job_fail(job,.Import_Failed) diff --git a/cook/build/compiler.odin b/cook/build/compiler.odin index e1f7d9bf2..7273d7dda 100644 --- a/cook/build/compiler.odin +++ b/cook/build/compiler.odin @@ -438,15 +438,6 @@ compile_end :: proc(c:^Compile,exit_code:int,allocator:=context.allocator,links: return diagnostics,text,failure,said[0] } -// Puts what a compiler wrote at staged in place of published. Windows does -// not replace a file a program has loaded, and lets it be renamed: the -// previous one goes to retired first. False when it cannot be replaced. -@(private) -publish :: proc(staged,published,retired:string) -> bool { - if os.rename(staged,published)==nil {return true} - return os.rename(published,retired)==nil && os.rename(staged,published)==nil -} - // What a published build is stamped with: the fingerprint of the sources its // compiler read (fingerprint), with root, compiler, flags and tests. @(private) diff --git a/cook/build/cook.odin b/cook/build/cook.odin index 2e8caac9b..a273c967f 100644 --- a/cook/build/cook.odin +++ b/cook/build/cook.odin @@ -13,6 +13,7 @@ import "core:time" import "compressor:data" import "compressor:blocks" import "mirai:asset" +import "mirai:cook/cache" import "mirai:cook/far" import anim "mirai:std/anim" import "mirai:cook/textures" @@ -21,6 +22,7 @@ import "mirai:std/base" import "mirai:std/jobs" import "mirai:std/serial" import "mirai:std/subprocess" +import "mirai:store" // The assets a game ships: of the assets folder of a project, the ones the // game can reach, as the runtime loads them fastest. @@ -31,10 +33,14 @@ import "mirai:std/subprocess" // interfaces, rows and code show) of the project and what cook.mirai // names, the other assets only code refers to. From them, everything an // asset depends on, and so on. The rest stays behind and is listed in the -// report. A texture that stores RGBA8 is written in the blocks of its use, -// unless only interfaces show it, and the blocks are kept in the texture -// cache for the next cook; a texture the editor imported Fast is encoded -// again as Final from its source, which the cook imports again; a scene or prefab with a stale prefab +// report. The assets of the project and of each plugin are their files and +// the outputs of the packages each root pins, listed from the store of the +// build (asset.root_packages_in), which its refresh brought to Final builds: +// a package not listed whole fails the cook by its name, and a texture still +// Fast is not shipped. What a package was imported from, inside a root, is a +// source and stays behind. A texture that stores RGBA8 is written in the +// blocks of its use, unless only interfaces show it, and the blocks are kept +// in the texture cache for the next cook; a scene or prefab with a stale prefab // instance is written with its instances brought up to date, so a shipped // game loads them as they are; every other file is copied byte for byte. // With far, every scene a section references gets its far mesh beside it @@ -44,7 +50,8 @@ import "mirai:std/subprocess" // (asset.mesh_collision_pieces, kept in the collision cache), which a // shipped game reads and never makes. The // folder gets a packed index, so the game does not look at its files to -// know what they are, and everything written is read back and checked. +// know what they are, and never opens a store; everything written is read +// back and checked. COOK_FILE :: "cook.mirai" @@ -57,12 +64,12 @@ Cook_Desc :: struct { assets:string, // the assets folder of the project output:string, // the assets folder of the shipped game: it ends up holding what was cooked and nothing else manifest:string, // the cook.mirai of the project. "", or a file that does not exist: the scenes are the only roots - plugins:[]project.Asset_Folder, // the assets of active plugins: every file a root, cooked under its prefix - texture_cache:string, // folder of compressed textures kept by content (textures.cache_read); "": none - mesh_cache:string, // folder of the meshes an import keeps (cook/import Import_Options); "": none + plugins:[]project.Asset_Folder, // the assets of active plugins: every file but the sources of its packages a root, cooked under its prefix + store:^store.Store, // that lists the packages of the roots and whose texture cache keeps compressed textures by content (textures.cache_read); nil: none, for roots without packages + cache_source:cache.Source, // asked on a miss of that cache before compressing collision_cache:string, // folder of the convex pieces of meshes kept by content (asset.mesh_collision_pieces); "": none codecs:asset.Codecs, // what the project ships with (build.compression) - root:string, // of the project: the sources of imported assets are named relative to it. "": the folder of assets + root:string, // of the project, where its packages.mirai is. "": a project without packages worker:jobs.Worker_Ref, // where the files are cooked, in parallel on its system. No system: a system of the cook, a thread per core server:bool, // for a dedicated server: what is only shown or heard (asset.Kind_Info.presentation) stays behind, and is not missing quick:bool, // a cook that checks a change (a dev game): the profile Balanced in place of Ultra @@ -84,7 +91,8 @@ Cook_Error :: enum u8 { Write_Failed, // detail: the file Verify_Failed, // detail: the file that does not read back as what was written Resolve_Failed, // detail: the scene or prefab whose prefab instances could not be brought up to date - Source_Failed, // detail: the texture that is still Fast, which the import of its source by a build makes Final + Source_Failed, // detail: the texture that is still Fast and the package it is of, which a refresh that requires Final brings + Package_Missing, // detail: one line per package the store does not list whole, with how to fix it Duplicate_Id, // detail: the two files with one id and how to tell them apart } @@ -169,6 +177,57 @@ manifest_read :: proc(path:string,allocator:runtime.Allocator) -> (manifest:Cook return } +// A package a root pins, with its outputs as the registry lists them. +@(private="file") +Cook_Package :: struct { + name:string, + source:string, // absolute, when it is inside the root: what it was imported from + folder:bool, // the source is a folder + inputs:[]string, // absolute, with the source: the other files the import read + listed:[]asset.Package_Listed, +} + +// The outputs of the packages root pins, from the store of the cook, with +// each package added to packages; a package not listed whole adds a line to +// missing. In the temp allocator. +@(private="file") +cook_listed :: proc(desc:Cook_Desc,root:string,packages:^[dynamic]Cook_Package,missing:^[dynamic]string) -> []asset.Package_Listed { + if root=="" {return nil} + if desc.store==nil { + if list,found,_:=asset.packages_load(root,context.temp_allocator);found && len(list.packages)>0 { + append(missing,fmt.tprintf("the packages of %s are listed from a store, and the cook has none",root)) + } + return nil + } + resolved,error:=asset.root_packages_in(desc.store,root,context.temp_allocator) + if error!=.None { + append(missing,fmt.tprintf("the packages of %s could not be listed (%v)",root,error)) + return nil + } + append(missing,..resolved.problems) + for &found in resolved.packages { + entry:=Cook_Package{name=found.name,folder=found.origin.folder} + if found.origin.path!="" && !filepath.is_abs(found.origin.path) && !strings.has_prefix(found.origin.path,"/") { + entry.source=fmt.tprintf("%s/%s",root,found.origin.path) + beside:=entry.source if found.origin.folder else entry.source[:strings.last_index_byte(entry.source,'/')] + inputs:=make([]string,len(found.inputs),context.temp_allocator) + for input,at in found.inputs {inputs[at]=fmt.tprintf("%s/%s",beside,input)} + entry.inputs=inputs + } + first:=-1 + count:=0 + for &l,at in resolved.listed { + if l.package_id!=found.id {continue} + if first<0 {first=at} + count+=1 + } + // packages_resolve lists the outputs of each package together. + if first>=0 {entry.listed=resolved.listed[first:first+count]} + append(packages,entry) + } + return resolved.listed +} + @(private="file") cook_fail :: proc(report:^Cook_Report,error:Cook_Error,detail:string,allocator:runtime.Allocator) { report.error,report.detail=error,strings.clone(detail,allocator) @@ -192,11 +251,34 @@ cook_run :: proc(desc:Cook_Desc) -> (report:Cook_Report) { if problem!="" {cook_fail(&report,.Manifest_Invalid,problem,allocator);return} registry:asset.Registry - asset.registry_init(®istry,desc.assets,"",temp,desc.root!=""?project.imported_path(desc.root,temp):"") - for folder in desc.plugins {asset.registry_mount(®istry,folder.path,folder.prefix,folder.imported)} + packages:=make([dynamic]Cook_Package,temp) + missing_packages:=make([dynamic]string,temp) + asset.registry_init(®istry,desc.assets,"",temp,cook_listed(desc,desc.root,&packages,&missing_packages)) + for folder in desc.plugins {asset.registry_mount(®istry,folder.path,folder.prefix,cook_listed(desc,folder.root,&packages,&missing_packages))} + registry.store=desc.store defer asset.registry_destroy(®istry) + if len(missing_packages)>0 {cook_fail(&report,.Package_Missing,strings.join(missing_packages[:],"\n",temp),allocator);return} if asset.registry_scan(®istry)!=.None {cook_fail(&report,.Assets_Missing,desc.assets,allocator);return} files:=registry.files[:] + // What a package was imported from, inside a root, is not for the game. + source:=make([]bool,len(files),temp) + owners:=make(map[base.Asset_Id]string,temp) + for &found in packages { + for &l in found.listed {owners[l.id]=found.name} + if found.source=="" {continue} + path:=asset.registry_relative(®istry,found.source) or_continue + for input in found.inputs { + if read,inside:=asset.registry_relative(®istry,input);inside { + if at,listed:=registry.by_path[read];listed {source[at]=true} + } + } + if !found.folder { + if at,listed:=registry.by_path[path];listed {source[at]=true} + continue + } + under:=strings.concatenate({path,"/"},temp) + for file,at in files {source[at]||=strings.has_prefix(file.path,under)} + } // Which of two files with one id is the asset depends on the machine: the cook takes neither. if len(registry.duplicates)>0 { twins:=registry.duplicates[0] @@ -220,11 +302,9 @@ cook_run :: proc(desc:Cook_Desc) -> (report:Cook_Report) { } for file,at in files { if file.state==.Asset && (file.kind==asset.KIND_SCENE || file.kind==asset.KIND_RECORDS || file.kind==asset.KIND_STRINGS) {root_add(reached,&pending,&report,u32(at))} - // What a plugin brings is for its code, which no scene names; its sources - // (a file with a sidecar beside it) and their sidecars stay behind. - source:=strings.has_suffix(file.path,asset.SIDECAR_EXTENSION) || strings.concatenate({file.path,asset.SIDECAR_EXTENSION},temp) in registry.by_path + // What a plugin brings is for its code, which no scene names; its sources stay behind. for folder in desc.plugins { - if !source && strings.has_prefix(file.path,folder.prefix) && len(file.path)>len(folder.prefix) && file.path[len(folder.prefix)]=='/' {root_add(reached,&pending,&report,u32(at))} + if !source[at] && strings.has_prefix(file.path,folder.prefix) && len(file.path)>len(folder.prefix) && file.path[len(folder.prefix)]=='/' {root_add(reached,&pending,&report,u32(at))} } } for root in manifest.roots { @@ -283,7 +363,7 @@ cook_run :: proc(desc:Cook_Desc) -> (report:Cook_Report) { } defer if on.system==&own {jobs.system_destroy(&own)} work:=Cook_Work{ - desc=desc,registry=®istry,files=files,textures=manifest.textures,drawn=drawn,shown=shown,pieces=pieces_wanted(®istry,reached,left,temp), + desc=desc,registry=®istry,files=files,owners=owners,textures=manifest.textures,drawn=drawn,shown=shown,pieces=pieces_wanted(®istry,reached,left,temp), on=on,allocator=on.system.allocator,first_error=len(files), results=make([]Cook_Result,len(files),temp), types=make([]asset.Types,jobs.worker_count(on.system),temp), @@ -304,6 +384,7 @@ cook_run :: proc(desc:Cook_Desc) -> (report:Cook_Report) { slice.sort_by(order[:],proc(a,b:Cook_Order) -> bool {return a.bytes>b.bytes || a.bytes==b.bytes && a.at ^asset.Types { } // A texture still built Fast is not shipped: the imports of the editor build one so until its -// Final is there, and the sync before the cook builds every one Final. +// Final is there, and the refresh before the cook requires Final. @(private="file") texture_probe :: proc(work:^Cook_Work,first,last:int,worker:int) { context.allocator=work.allocator @@ -437,15 +519,18 @@ texture_probe :: proc(work:^Cook_Work,first,last:int,worker:int) { result:=&work.results[at] if result.entry.reused {continue} file:asset.Asset - if asset.asset_load(&file,asset.registry_file_path(work.registry,work.files[at].path,context.temp_allocator),context.temp_allocator)!=.None { + if cook_open(work,work.files[at].path,&file)!=.None { result.problem=work_text(work,"it cannot be read") continue } defer asset.asset_close(&file) view,error:=asset.texture_view(&file,types) if error!=.None || .Fast not_in view.info.flags {continue} - named:=view.source!=nil?base.text_get(view.source.source[:]):"a source it does not name" - result.problem=work_text(work,"it was imported Fast and %s was not imported again as Final: `mirai build game` does it before it cooks",named) + if name,listed:=work.owners[work.files[at].id];listed { + result.problem=work_text(work,"the package %s has only a build with Fast textures here, and a game ships Final: `mirai build game` refreshes it requiring Final, which a machine with its source must ingest with this engine, or a peer that has that build must give",name) + } else { + result.problem=work_text(work,"it was written Fast, and a game ships Final: write it again Final") + } } } @@ -475,7 +560,7 @@ files_cook :: proc(work:^Cook_Work,first,last:int,worker:int) { file_cook :: proc(work:^Cook_Work,at:int,worker:int,entry:^Cook_File) -> (error:Cook_Error,detail:string) { file:=&work.files[at] types:=work_types(work,worker) - source:=asset.registry_file_path(work.registry,file.path,context.temp_allocator) + source:=file.path target:=fmt.tprintf("%s/%s",work.desc.output,file.path) if slash:=strings.last_index_byte(target,'/');slash>=0 { if os.make_directory_all(target[:slash])!=nil && !os.is_dir(target[:slash]) {return .Write_Failed,work_text(work,"%s",file.path)} @@ -500,8 +585,17 @@ file_cook :: proc(work:^Cook_Work,at:int,worker:int,entry:^Cook_File) -> (error: laid:=!entry.compressed && file.state==.Asset && file.kind==asset.KIND_TEXTURE if laid && !texture_lay(work,source,target,types) {return .Write_Failed,work_text(work,"%s",file.path)} if !entry.compressed && !laid { - data,read_error:=serial.file_read(source,context.temp_allocator) - if read_error!=.None {return .Read_Failed,work_text(work,"%s",file.path)} + // A listed output of a package is its object of the store, in the container format. + data:[]u8 + if object,listed:=asset.registry_listed_object(work.registry,file.path);listed { + read_error:asset.Error + data,read_error=asset.object_file(work.registry.store,object,context.temp_allocator) + if read_error!=.None {return .Read_Failed,work_text(work,"%s",file.path)} + } else { + read_error:serial.File_Error + data,read_error=serial.file_read(asset.registry_file_path(work.registry,file.path,context.temp_allocator),context.temp_allocator) + if read_error!=.None {return .Read_Failed,work_text(work,"%s",file.path)} + } if file.state==.Asset && (file.kind==asset.KIND_SCENE) { sync.mutex_lock(&work.resolve) resolved,resolve_ok:=scene_cook(&work.resolver,&work.resolver_types,data) @@ -534,9 +628,36 @@ file_cook :: proc(work:^Cook_Work,at:int,worker:int,entry:^Cook_File) -> (error: return .None,"" } +// Opens the asset at path of the registry of the cook, in the temp allocator: a listed output of +// a package from its object of the store, any other from its file. +@(private="file") +cook_open :: proc(work:^Cook_Work,path:string,file:^asset.Asset) -> asset.Error { + if object,listed:=asset.registry_listed_object(work.registry,path);listed { + if work.registry.store==nil {return .Not_Found} + return asset.asset_open_object(file,work.registry.store,object,context.temp_allocator) + } + return asset.asset_load(file,asset.registry_file_path(work.registry,path,context.temp_allocator),context.temp_allocator) +} + +// Looks the texture cache up once for every texture the pass may compress, asking its source in +// one fetch for what the store lacks: the jobs then read the cache without asking. +@(private="file") +textures_lookup :: proc(work:^Cook_Work) { + if work.desc.store==nil || work.textures!=.Blocks {return} + keys:=make([dynamic]u64,context.temp_allocator) + rdo:=work.desc.codecs!=.None + for at in work.probed { + if work.results[at].entry.reused || !(work.drawn[at] || !work.shown[at]) {continue} + append(&keys,textures.cache_key_blocks(work.files[at].content_hash,rdo)) + } + if len(keys)==0 {return} + here:=make([]bool,len(keys),context.temp_allocator) + textures.cache_lookup({store=work.desc.store,source=work.desc.cache_source},keys[:],here) +} + // Raise it when the cook writes a file otherwise from the same inputs: every key changes. @(private="file") -COOK_KEYS_VERSION :: 1 +COOK_KEYS_VERSION :: 3 // What an earlier cook wrote of a file: the key of what it was made of, and the output as it left it. @(private="file") @@ -633,7 +754,7 @@ far_meshes_cook :: proc(work:^Cook_Work,reached,left:[]bool,allocator:runtime.Al for file,at in work.files { if !reached[at] || left[at] || file.state!=.Asset || file.kind!=asset.KIND_SCENE {continue} source:asset.Asset - if asset.asset_load(&source,asset.registry_file_path(work.registry,file.path,context.temp_allocator),context.temp_allocator)!=.None {return nil,.Read_Failed,strings.clone(file.path,allocator)} + if cook_open(work,file.path,&source)!=.None {return nil,.Read_Failed,strings.clone(file.path,allocator)} far.far_scenes(&source,types,&scenes) asset.asset_close(&source) } @@ -693,25 +814,26 @@ far_meshes_cook :: proc(work:^Cook_Work,reached,left:[]bool,allocator:runtime.Al @(private="file") texture_cook :: proc(work:^Cook_Work,source,target:string,types:^asset.Types,worker:jobs.Worker_Ref) -> (compressed,ok:bool) { file:asset.Asset - if asset.asset_load(&file,source,context.temp_allocator)!=.None {return} + if cook_open(work,source,&file)!=.None {return} defer asset.asset_close(&file) view,error:=asset.texture_view(&file,types) if error!=.None {return} if view.info.format!=.RGBA8 {return false,true} - cache:=work.desc.texture_cache + // The pass looked up the cache for every texture at once (textures_lookup): no texture asks the source. + kept:=cache.Cache{store=work.desc.store} // Blocks that nothing compresses keep the least error. rdo:=work.desc.codecs!=.None key:=textures.cache_key_blocks(file.blob.header.content_hash,rdo) - built,cached:=textures.cache_read(cache,key,types,context.temp_allocator) + built,cached:=textures.cache_read(kept,key,types,context.temp_allocator) if !cached { built_ok:bool if built,built_ok=textures.blocks_of(view,context.temp_allocator,worker,rdo);!built_ok {return} // Textures of one content build the same blocks: the first to get here keeps them. - if cache!="" { + if kept.store!=nil { sync.mutex_lock(&work.cache) if key not_in work.cached { work.cached[key]={} - textures.cache_write(cache,key,types,&built) + textures.cache_write(kept,key,types,&built) } sync.mutex_unlock(&work.cache) } @@ -724,7 +846,7 @@ texture_cook :: proc(work:^Cook_Work,source,target:string,types:^asset.Types,wor @(private="file") texture_lay :: proc(work:^Cook_Work,source,target:string,types:^asset.Types) -> bool { file:asset.Asset - if asset.asset_load(&file,source,context.temp_allocator)!=.None {return false} + if cook_open(work,source,&file)!=.None {return false} defer asset.asset_close(&file) view,error:=asset.texture_view(&file,types) if error!=.None {return false} @@ -802,7 +924,7 @@ cook_summary :: proc(report:^Cook_Report,allocator:=context.allocator) -> string @(private="file") animation_cook :: proc(work:^Cook_Work,source,target:string,types:^asset.Types) -> (compressed,ok:bool) { file:asset.Asset - if asset.asset_load(&file,source,context.temp_allocator)!=.None {return} + if cook_open(work,source,&file)!=.None {return} defer asset.asset_close(&file) view,error:=asset.animation_view(&file,types) if error!=.None {return} @@ -887,7 +1009,7 @@ pieces_wanted :: proc(registry:^asset.Registry,reached,left:[]bool,allocator:run @(private="file") mesh_pieces_cook :: proc(work:^Cook_Work,source:string,data:[]u8,types:^asset.Types) -> (out:[]u8,ok:bool) { file:asset.Asset - if asset.asset_load(&file,source,context.temp_allocator)!=.None {return} + if cook_open(work,source,&file)!=.None {return} defer asset.asset_close(&file) points:=make([dynamic][3]f32,context.temp_allocator) sizes:=make([dynamic]u32,context.temp_allocator) diff --git a/cook/build/cook_bench_test.odin b/cook/build/cook_bench_test.odin index fc66bf781..cb9f654de 100644 --- a/cook/build/cook_bench_test.odin +++ b/cook/build/cook_bench_test.odin @@ -5,6 +5,7 @@ import "core:os" import "core:testing" import "mirai:project" import "mirai:std/base" +import "mirai:store" // The cook of a project as the job Game does it: filling an empty texture // cache and from it. Run with --release -define:COOK_BENCH=true; @@ -32,7 +33,9 @@ bench_cook_of_a_project_filling_its_texture_cache_and_from_it :: proc(t:^testing if found.prefix!="" {append(&plugins,found)} } desc.plugins=plugins[:] - desc.texture_cache=fmt.tprintf("%s/cache",folder) + kept,opened:=store.open(fmt.tprintf("%s/store",folder),context.temp_allocator) + if !testing.expect_value(t,opened,store.Error.None) {return} + desc.store=&kept filling:=cook_assets(desc) testing.expect_value(t,filling.error,Cook_Error.None) cached:=cook_assets(desc) diff --git a/cook/build/cook_cache_test.odin b/cook/build/cook_cache_test.odin index 7a286c90f..7cc64854b 100644 --- a/cook/build/cook_cache_test.odin +++ b/cook/build/cook_cache_test.odin @@ -5,15 +5,20 @@ import "core:os" import "core:slice" import "core:testing" import "mirai:asset" +import "mirai:cook/cache" import "mirai:cook/textures" import "mirai:std/base" +import "mirai:store" @(test) a_cook_takes_the_blocks_of_a_texture_from_the_cache_of_its_content :: proc(t:^testing.T) { types:asset.Types;asset.types_init(&types);defer asset.types_destroy(&types) folder:=base.test_folder(t,"cook-texture-cache") if folder=="" {return} - assets,cache:=fmt.tprintf("%s/assets",folder),fmt.tprintf("%s/cache",folder) + assets:=fmt.tprintf("%s/assets",folder) + kept_store,opened:=store.open(fmt.tprintf("%s/store",folder),context.temp_allocator) + if !testing.expect_value(t,opened,store.Error.None) {return} + texture_cache:=cache.Cache{store=&kept_store} os.make_directory_all(fmt.tprintf("%s/textures",assets)) source:=fmt.tprintf("%s/textures/wood.tex",assets) pixels:[8*8*4]u8 @@ -30,7 +35,7 @@ a_cook_takes_the_blocks_of_a_texture_from_the_cache_of_its_content :: proc(t:^te } // The first cook compresses the texture and keeps its blocks. - report:=cook_assets({assets=assets,output=fmt.tprintf("%s/first",folder),manifest=manifest,texture_cache=cache,allocator=context.temp_allocator}) + report:=cook_assets({assets=assets,output=fmt.tprintf("%s/first",folder),manifest=manifest,store=&kept_store,codecs=.Open,allocator=context.temp_allocator}) testing.expect(t,report.error==.None && report.textures_compressed==1) testing.expect(t,report.seconds>0,"the report says how long the cook took") format,blocks:=level(t,&types,fmt.tprintf("%s/first/textures/wood.tex",folder)) @@ -38,19 +43,19 @@ a_cook_takes_the_blocks_of_a_texture_from_the_cache_of_its_content :: proc(t:^te original:asset.Asset testing.expect_value(t,asset.asset_load(&original,source,context.temp_allocator),asset.Error.None) key:=textures.cache_key_blocks(original.blob.header.content_hash) - kept,found:=textures.cache_read(cache,key,&types,context.temp_allocator) + kept,found:=textures.cache_read(texture_cache,key,&types,context.temp_allocator) testing.expect(t,found && kept.format==.BC1 && slice.equal(kept.mips[0],blocks)) // The next cook reads them: blocks the cache was given are the ones it ships. kept.mips[0][0]~=0xff - testing.expect(t,textures.cache_write(cache,key,&types,&kept)) - report=cook_assets({assets=assets,output=fmt.tprintf("%s/second",folder),manifest=manifest,texture_cache=cache,allocator=context.temp_allocator}) + testing.expect(t,textures.cache_write(texture_cache,key,&types,&kept)) + report=cook_assets({assets=assets,output=fmt.tprintf("%s/second",folder),manifest=manifest,store=&kept_store,codecs=.Open,allocator=context.temp_allocator}) testing.expect(t,report.error==.None && report.textures_compressed==1) _,cached:=level(t,&types,fmt.tprintf("%s/second/textures/wood.tex",folder)) testing.expect(t,len(cached)==32 && cached[0]==blocks[0]~0xff && slice.equal(cached[1:],blocks[1:])) // A cook without a cache compresses, and another content is another key. - report=cook_assets({assets=assets,output=fmt.tprintf("%s/third",folder),manifest=manifest,allocator=context.temp_allocator}) + report=cook_assets({assets=assets,output=fmt.tprintf("%s/third",folder),manifest=manifest,codecs=.Open,allocator=context.temp_allocator}) _,fresh:=level(t,&types,fmt.tprintf("%s/third/textures/wood.tex",folder)) testing.expect(t,report.error==.None && slice.equal(fresh,blocks)) pixels[0]+=1 @@ -60,3 +65,91 @@ a_cook_takes_the_blocks_of_a_texture_from_the_cache_of_its_content :: proc(t:^te testing.expect(t,textures.cache_key_blocks(changed.blob.header.content_hash)!=key) free_all(context.temp_allocator) } + +// Every table and column of two opened assets, decoded. +@(private="file") +same_tables :: proc(a,b:^asset.Asset) -> bool { + if a.blob.header.id!=b.blob.header.id || a.blob.header.kind!=b.blob.header.kind || len(a.tables)!=len(b.tables) {return false} + for &table,at in a.tables { + other:=&b.tables[at] + if table.name!=other.name || table.rows!=other.rows || len(table.columns)!=len(other.columns) {return false} + for _,column in table.columns { + x,x_error:=asset.asset_column_data(a,&table,column) + y,y_error:=asset.asset_column_data(b,other,column) + if x_error!=.None || y_error!=.None || !slice.equal(x,y) {return false} + } + } + return true +} + +// The assets a project lists from its packages are objects of the store: a +// cook ships each as a file that loads as the object does, and looks the +// texture cache up once for the textures of its pass, in one fetch. +@(test) +a_cook_ships_package_assets_from_their_objects_and_looks_the_cache_up_once :: proc(t:^testing.T) { + types:asset.Types;asset.types_init(&types);defer asset.types_destroy(&types) + folder:=base.test_folder(t,"cook-package-objects") + if folder=="" {return} + root:=fmt.tprintf("%s/game",folder) + assets:=fmt.tprintf("%s/assets",root) + os.make_directory_all(assets) + s,opened:=store.open(fmt.tprintf("%s/store",folder),context.temp_allocator) + if !testing.expect_value(t,opened,store.Error.None) {return} + defer store.close(&s) + file_of :: proc(t:^testing.T,types:^asset.Types,id:base.Asset_Id,fill:u8) -> []u8 { + pixels:=make([]u8,8*8*4,context.temp_allocator) + for &value,at in pixels {value=at%4==3?255:u8(at)*fill} + doc:asset.Document + testing.expect_value(t,asset.texture_document(&doc,types,{id=id,width=8,height=8,format=.RGBA8,mips={pixels}},context.temp_allocator),asset.Error.None) + defer asset.document_destroy(&doc) + bytes,_:=asset.document_bytes(&doc,context.temp_allocator) + return bytes + } + p:=asset.Package{id={40,40},name="kit"} + p.assets=make([dynamic]asset.Package_Asset,context.temp_allocator) + b:=asset.Build{package_id=p.id,final=true} + b.outputs=make([dynamic]asset.Build_Output,context.temp_allocator) + info,_:=asset.kind_info(asset.KIND_TEXTURE) + objects:[2]store.Hash + for name,at in ([2]string{"wood","stone"}) { + id:=base.Asset_Id{41+u64(at),1} + append(&p.assets,asset.Package_Asset{kind=asset.KIND_TEXTURE,name=name,id=id}) + objects[at],_=asset.object_put(&s,file_of(t,&types,id,u8(3+at))) + append(&b.outputs,asset.Build_Output{id=id,kind=asset.KIND_TEXTURE,kind_version=info.version,object=objects[at]}) + } + manifest,_:=asset.package_put(&s,&p) + built,_:=asset.build_put(&s,&b) + digits:[64]u8 + testing.expect_value(t,store.ref_set(&s,asset.REF_BUILDS,store.hex(asset.build_select(manifest,&p),&digits),built),store.Error.None) + list:asset.Packages + list.packages=make([dynamic]asset.Project_Package,context.temp_allocator) + append(&list.packages,asset.Project_Package{id=p.id,manifest=manifest,folder="kit"}) + testing.expect_value(t,asset.packages_save(root,&list),asset.Error.None) + Counted :: struct {calls,names:int} + counted:Counted + fetch :: proc(user:rawptr,into:^store.Store,namespace:string,names:[]string,given:[]bool) { + c:=(^Counted)(user) + c.calls+=1 + c.names+=len(names) + } + for pass in ([2]string{"blocks","keep"}) { + manifest_file:=fmt.tprintf("%s/%s-%s",folder,pass,COOK_FILE) + testing.expect(t,os.write_entire_file(manifest_file,fmt.tprintf(`{{"roots":["kit/"],"textures":"%s"}}`,pass))==nil) + output:=fmt.tprintf("%s/%s",folder,pass) + report:=cook_assets({assets=assets,root=root,output=output,manifest=manifest_file,store=&s,cache_source={fetch=fetch,user=&counted},allocator=context.temp_allocator}) + if !testing.expectf(t,report.error==.None,"%s: %v %s",pass,report.error,report.detail) {continue} + testing.expect_value(t,len(report.files),2) + if pass=="blocks" { + testing.expectf(t,counted.calls==1 && counted.names==2 && report.textures_compressed==2,"one lookup for the pass: %v",counted) + continue + } + for name,at in ([2]string{"wood","stone"}) { + shipped,stored:asset.Asset + testing.expect_value(t,asset.asset_load(&shipped,fmt.tprintf("%s/kit/%s.tex",output,name),context.temp_allocator),asset.Error.None) + testing.expect_value(t,asset.asset_open_object(&stored,&s,objects[at],context.temp_allocator),asset.Error.None) + testing.expectf(t,same_tables(&shipped,&stored),"%s ships as its object loads",name) + asset.asset_close(&shipped) + asset.asset_close(&stored) + } + } +} diff --git a/cook/build/cook_test.odin b/cook/build/cook_test.odin index 9bd63da63..b3450ff50 100644 --- a/cook/build/cook_test.odin +++ b/cook/build/cook_test.odin @@ -12,6 +12,7 @@ import anim "mirai:std/anim" import mm "mirai:std/math" import "mirai:std/base" import "mirai:std/jobs" +import "mirai:store" @(private="file") WOOD_COLOR :: base.Asset_Id{1,1} @(private="file") WOOD_NORMAL :: base.Asset_Id{1,2} @@ -133,7 +134,7 @@ a_cook_ships_what_the_scenes_and_the_roots_reach_as_the_runtime_loads_it :: proc manifest:=fmt.tprintf("%s/project/%s",folder,COOK_FILE) // Without cook.mirai the scenes and the prefabs are the roots. - report:=cook_assets({assets=assets,output=output,manifest=manifest,allocator=context.temp_allocator}) + report:=cook_assets({assets=assets,output=output,manifest=manifest,codecs=.Open,allocator=context.temp_allocator}) testing.expect_value(t,report.error,Cook_Error.None) kept,dropped:=names(&report) testing.expect_value(t,kept,"materials/wood.mat models/crate.mesh prefabs/prop.scene scenes/main.scene textures/wood_color.tex textures/wood_normal.tex ") @@ -267,8 +268,6 @@ a_cook_ships_every_asset_of_an_active_plugin_under_its_prefix :: proc(t:^testing BRUSH :: base.Asset_Id{7,1} material_write(t,&types,fmt.tprintf("%s/brushes/soft.mat",plugin),BRUSH,WOOD_COLOR,{}) testing.expect(t,os.write_entire_file(fmt.tprintf("%s/readme.txt",plugin),"for the code of the plugin")==nil) - // A source and its sidecar stay behind: what its import made ships, not what it was made from. - testing.expect(t,os.write_entire_file(fmt.tprintf("%s/brushes/soft.png",plugin),"a source")==nil && os.write_entire_file(fmt.tprintf("%s/brushes/soft.png.import",plugin),"{}")==nil) report:=cook_assets({assets=assets,output=output,plugins={{path=plugin,prefix="plugins/probe"}},allocator=context.temp_allocator}) testing.expect_value(t,report.error,Cook_Error.None) @@ -393,7 +392,9 @@ a_cook_on_many_workers_writes_what_a_cook_on_one_does :: proc(t:^testing.T) { js:jobs.Job_System if !jobs.system_init(&js,{threads=threads,allocator=runtime.heap_allocator()}) {return {error=.Write_Failed}} defer jobs.system_destroy(&js) - return cook_assets({assets=assets,output=fmt.tprintf("%s/%s",folder,name),manifest=manifest,texture_cache=fmt.tprintf("%s/%s_cache",folder,name),worker={system=&js},allocator=context.temp_allocator}) + kept,opened:=store.open(fmt.tprintf("%s/%s_store",folder,name),context.temp_allocator) + if opened!=.None {return {error=.Write_Failed}} + return cook_assets({assets=assets,output=fmt.tprintf("%s/%s",folder,name),manifest=manifest,store=&kept,worker={system=&js},allocator=context.temp_allocator}) } one:=cook(folder,assets,manifest,"one",0) many:=cook(folder,assets,manifest,"many",7) @@ -567,8 +568,10 @@ a_cook_writes_again_only_what_is_made_of_what_changed :: proc(t:^testing.T) { } } -// What a cook writes is compressed as its project allows and reads back: nothing, or streams of -// the formats of the codecs it allows, with the layouts of blocks only of the best. +// What a cook writes is compressed as its project allows and reads back as what the project +// holds: nothing, or streams of the formats of the codecs it allows, with the layouts of blocks +// only of the best. The cook is where the container's own compression is made: the project keeps +// its assets written without a compressor. @(test) a_cook_compresses_as_its_project_says_and_what_it_writes_decodes :: proc(t:^testing.T) { types:asset.Types;asset.types_init(&types);defer asset.types_destroy(&types) @@ -601,6 +604,18 @@ a_cook_compresses_as_its_project_says_and_what_it_writes_decodes :: proc(t:^test _,laid:=asset.asset_table(&a,asset.TEXTURE_TABLE_LAYOUTS) testing.expect(t,!laid || codecs==.Best,"layouts only of the best codecs") } + if file.compressed || file.resolved {continue} + // Written again without a compressor, the file shipped is the one the project holds. + source,_:=os.read_entire_file(fmt.tprintf("%s/%s",assets,file.path),context.temp_allocator) + shipped,_:=os.read_entire_file(path,context.temp_allocator) + header,_:=asset.header_read(source) + raw :: proc(bytes:[]u8,types:^asset.Types,texture:bool) -> []u8 { + out:[]u8 + if texture {out,_=asset.texture_repack(bytes,types,.Fast,context.temp_allocator)} else {out,_=asset.document_repack(bytes,.Fast,.None,context.temp_allocator)} + return out + } + want,got:=raw(source,&types,header.kind==asset.KIND_TEXTURE),raw(shipped,&types,header.kind==asset.KIND_TEXTURE) + testing.expectf(t,len(want)>0 && string(want)==string(got),"%s ships what the project holds (%v)",file.path,codecs) } testing.expect(t,(streams==0)==(codecs==.None),"something compresses unless nothing may") } diff --git a/cook/build/engine.odin b/cook/build/engine.odin index 2f86a3783..29ad25fa3 100644 --- a/cook/build/engine.odin +++ b/cook/build/engine.odin @@ -77,11 +77,11 @@ Engine_Package :: struct { ENGINE_PACKAGES :: [?]Engine_Package{ {.Base,"std/base","std"},{.Base,"std/subprocess","std"},{.Base,"std/math","std"},{.Base,"std/geom","std"},{.Base,"std/serial","std"}, {.Base,"std/spatial","std"},{.Base,"std/jobs","std"},{.Base,"std/anim","std"},{.Base,"std/ai","std"},{.Base,"std/input","std"}, - {.Base,"std/ui_format","std"}, + {.Base,"std/ui_format","std"},{.Base,"std/blake3","std"}, {.Base,"audio","system"},{.Base,"platform","system"},{.Base,"rhi","system"}, - {.Base,"project","system"},{.Base,"core",""}, + {.Base,"project","system"},{.Base,"store","system"},{.Base,"core",""}, - {.Engine,"asset",""},{.Engine,"net",""},{.Engine,"render/abi","render"},{.Engine,"render","render"},{.Engine,"third_party/freetype","freetype"}, + {.Engine,"asset",""},{.Engine,"net",""},{.Engine,"exchange",""},{.Engine,"render/abi","render"},{.Engine,"render","render"},{.Engine,"third_party/freetype","freetype"}, {.Engine,"ui","ui"},{.Engine,"host",""},{.Engine,"engine/input","engine"},{.Engine,"engine/scene","engine"},{.Engine,"engine/volume","engine"}, {.Engine,"engine/render","engine"},{.Engine,"engine/audio","engine"},{.Engine,"engine/gui","engine"},{.Engine,"engine/camera","engine"},{.Engine,"engine/save","engine"}, @@ -89,7 +89,7 @@ ENGINE_PACKAGES :: [?]Engine_Package{ {.Cook,"cook/textures","cook"},{.Cook,"cook/image","cook"},{.Cook,"cook/movie","cook"},{.Cook,"cook/rig","cook"}, {.Cook,"cook/import","cook"},{.Cook,"cook/import/obj","cook"},{.Cook,"cook/import/gltf","cook"},{.Cook,"cook/import/usd","cook"}, {.Cook,"cook/import/fbx","cook"},{.Cook,"cook/shaders","cook"},{.Cook,"cook/author","cook"},{.Cook,"cook/prefab","cook"},{.Cook,"cook/import/sources","cook"}, - {.Cook,"cook/install","cook"},{.Cook,"cook/build",""}, + {.Cook,"cook/share","cook"},{.Cook,"cook/install","cook"},{.Cook,"cook/build",""}, {.Editor,"third_party/sqlite","sqlite"},{.Editor,"editor/api","editor-parts"},{.Editor,"editor/acp","editor-parts"},{.Editor,"editor/diff","editor-parts"},{.Editor,"editor/ai_store","editor-parts"}, {.Editor,"editor/git","editor-parts"},{.Editor,"editor/history","editor-parts"},{.Editor,"editor/records","editor-parts"},{.Editor,"editor/kit","editor-parts"},{.Editor,"editor",""}, @@ -173,7 +173,7 @@ Engine_Paths :: struct { stage:string, work:string, // what the compilers say while a build runs editor_program,player_program:string, - import_program:string, // runs cook/import/sources.command: what mirai import does + import_program:string, // runs cook/import/sources.command, what mirai import does, and cook/share.command, mirai share and join author_program:string, // runs cook/author.command: what mirai ui and mirai scene do stamp:string, defines:[]string, // what the binaries are built with (engine_defines) @@ -479,9 +479,14 @@ engine_program_sources :: proc(paths:Engine_Paths) -> [4]Engine_Program { import "core:os" import "mirai:cook/import/sources" +import "mirai:cook/share" import "mirai:editor" -main :: proc() {os.exit(sources.command(os.args[1:],os.to_writer(os.stdout),os.to_writer(os.stderr),editor.import_library))} +main :: proc() { + args:=os.args[1:] + if len(args)>0 && (args[0]=="share" || args[0]=="join") {os.exit(share.command(args,os.to_writer(os.stdout),os.to_writer(os.stderr),editor.import_library))} + os.exit(sources.command(args,os.to_writer(os.stdout),os.to_writer(os.stderr),editor.import_library)) +} `,paths.import_program}, {"author program","author",`package main @@ -639,9 +644,9 @@ engine_build_parts :: proc(desc:Engine_Desc,done:^Engine_Result,compiled:^[dynam result.status=.Up_To_Date return } - // What runs these binaries goes on with them; the build writes in their place (engine_unlock.odin). - if file,unlocked:=engine_unlock(paths.folder);!unlocked { - return engine_fail(.Write_Failed,"binaries of the engine",fmt.tprintf("%s runs and could not be moved aside: close what runs it",file)) + when ODIN_OS==.Windows { + retired:=project.path(paths.folder,ENGINE_RETIRED,allocator=context.temp_allocator) + engine_retired_sweep(retired) } // No stamp says what the stage, the declarations and the programs there were written from. if desc.force || !plan.stamped { @@ -748,13 +753,23 @@ engine_build_parts :: proc(desc:Engine_Desc,done:^Engine_Result,compiled:^[dynam if build_programs {stamp.programs,stamp.programs_against=0,0} _=engine_stamp_write(paths.stamp,stamp,libraries) + linking,link_error:=os.make_directory_temp(paths.folder,".link-*",context.temp_allocator) + if link_error!=nil {return engine_fail(.Write_Failed,paths.folder,"")} + defer base.remove_all(linking) tasks:=make([dynamic]Engine_Task,context.temp_allocator) for library,which in libraries { output,_:=engine_library_files(paths,library,context.temp_allocator) - task:=Engine_Task{name=fmt.tprintf("library %s",library.file),compile={streams=engine_streams(paths,library.file),staged=output,library=which},state=.Done} + task:=Engine_Task{ + name=fmt.tprintf("library %s",library.file),output=output,state=.Done, + compile={streams=engine_streams(paths,library.file),library=which}, + } if compile[which] { task.state=.Pending - task.command=engine_library_command(compiler,paths,library,slice.contains(reach.needs[which],true),mode,plan.library_flags[which]) + _,name:=os.split_path(output) + task.compile.staged=project.path(linking,name,allocator=context.temp_allocator) + links:=slice.contains(reach.needs[which],true) + task.command=engine_library_command(compiler,paths,library,task.compile.staged, + links,mode,plan.library_flags[which]) } append(&tasks,task) } @@ -762,10 +777,17 @@ engine_build_parts :: proc(desc:Engine_Desc,done:^Engine_Result,compiled:^[dynam every:=make([]bool,len(libraries),context.temp_allocator) slice.fill(every,true) for program in programs { - task:=Engine_Task{name=program.name,compile={streams=engine_streams(paths,fmt.tprintf("program-%s",program.package_name)),staged=program.output,library=-1},state=.Done,needs=every} + streams:=engine_streams(paths,fmt.tprintf("program-%s",program.package_name)) + task:=Engine_Task{ + name=program.name,output=program.output,state=.Done,needs=every, + compile={streams=streams,library=-1}, + } if build_programs { task.state=.Pending - task.command=engine_program_command(compiler,paths,program,plan.flags[:]) + _,name:=os.split_path(program.output) + task.compile.staged=project.path(linking,name,allocator=context.temp_allocator) + task.command=engine_program_command(compiler,paths,program,task.compile.staged, + plan.flags[:]) } append(&tasks,task) } @@ -780,9 +802,10 @@ engine_build_parts :: proc(desc:Engine_Desc,done:^Engine_Result,compiled:^[dynam if len(compiled)>0 {cache_trim(workspace.cache_folder(root,ENGINE_CACHE,context.temp_allocator),ENGINE_CACHE_ENTRIES)} if !ok {return failed} done.programs=build_programs - for copy in engine_copies(root,paths.folder,compiler) { - if !build_programs && os.is_file(copy.to) {continue} - if os.copy_file(copy.to,copy.from)!=nil {return engine_fail(.Copy_Failed,copy.to,"")} + if copy:=sdl_copy(root,paths.folder,compiler,context.temp_allocator);copy.from!="" { + if (build_programs || !os.is_file(copy.to)) && !copy_binary(copy.to,copy.from) { + return engine_fail(.Copy_Failed,copy.to,"") + } } stamp.programs,stamp.programs_against=plan.programs,programs_against _=engine_stamp_write(paths.stamp,stamp,libraries) @@ -795,6 +818,7 @@ engine_build_parts :: proc(desc:Engine_Desc,done:^Engine_Result,compiled:^[dynam Engine_Task :: struct { name:string, // the step it is command:[]string, + output:string, needs:[]bool, // the libraries it links against: it starts once none of them is being built state:enum u8 {Done,Pending,Running,Failed}, compile:Compile, // library: of the libraries, -1 for a program @@ -851,6 +875,7 @@ engine_tasks_run :: proc(root:string,paths:Engine_Paths,libraries:[]Engine_Libra continue } diagnostics,text,failure,_:=compile_end(&task.compile,state.exit_code) + if failure==.None && !engine_publish(&task) {failure=.Output_Locked} if failure==.None { task.state=.Done if which:=task.compile.library;ok && which>=0 { @@ -878,8 +903,8 @@ engine_tasks_run :: proc(root:string,paths:Engine_Paths,libraries:[]Engine_Libra // The compiler's command for a library of the engine: its own packages from // the stage, every other one as its declarations. @(private) -engine_library_command :: proc(compiler:string,paths:Engine_Paths,library:Engine_Library,links_libraries:bool,mode:Engine_Mode,flags:[]string) -> []string { - output,_:=engine_library_files(paths,library,context.temp_allocator) +engine_library_command :: proc(compiler:string,paths:Engine_Paths,library:Engine_Library, + output:string,links_libraries:bool,mode:Engine_Mode,flags:[]string) -> []string { command:=make([dynamic]string,context.temp_allocator) append(&command,compiler,"build",project.path(paths.stage,engine_library_entry(library,context.temp_allocator),allocator=context.temp_allocator),"-build-mode:dll") append(&command,fmt.tprintf("-collection:mirai=%s",paths.decl),fmt.tprintf("-collection:%s=%s",STAGE_COLLECTION,paths.stage),fmt.tprintf("-out:%s",output)) @@ -898,19 +923,42 @@ engine_library_command :: proc(compiler:string,paths:Engine_Paths,library:Engine } @(private) -engine_program_command :: proc(compiler:string,paths:Engine_Paths,program:Engine_Program,flags:[]string) -> []string { +engine_program_command :: proc(compiler:string,paths:Engine_Paths,program:Engine_Program, + output:string,flags:[]string) -> []string { folder:=project.path(paths.folder,"programs",program.package_name,allocator=context.temp_allocator) command:=make([dynamic]string,context.temp_allocator) - append(&command,compiler,"build",folder,fmt.tprintf("-collection:mirai=%s",paths.decl),fmt.tprintf("-out:%s",program.output)) + append(&command,compiler,"build",folder,fmt.tprintf("-collection:mirai=%s",paths.decl), + fmt.tprintf("-out:%s",output)) append(&command,..flags) append(&command,"-json-errors") // The program of the editor has the name of its library, and so would their PDBs. - when ODIN_OS==.Windows {append(&command,fmt.tprintf("-pdb-name:%s",strings.concatenate({program.output[:len(program.output)-len(project.EXECUTABLE_SUFFIX)],"-program.pdb"},context.temp_allocator)))} + when ODIN_OS==.Windows {append(&command,fmt.tprintf("-pdb-name:%s",engine_pdb(output,true)))} when ODIN_OS==.Linux {linker_flags_append(&command,"-Wl,-rpath,$ORIGIN",context.temp_allocator)} else when ODIN_OS==.Darwin {linker_flags_append(&command,"-Wl,-rpath,@executable_path",context.temp_allocator)} return command[:] } +@(private="file") +engine_publish :: proc(task:^Engine_Task) -> bool { + staged:=task.compile.staged + when ODIN_OS==.Windows { + program:=task.compile.library<0 + pdb:=engine_pdb(staged,program) + if os.is_file(pdb) && !publish(pdb,engine_pdb(task.output,program)) {return false} + } else when ODIN_OS==.Darwin { + from,to:=fmt.tprintf("%s.dSYM",staged),fmt.tprintf("%s.dSYM",task.output) + if os.is_dir(from) && (base.remove_all(to)!=nil || os.rename(from,to)!=nil) {return false} + } + return publish(staged,task.output) +} + +when ODIN_OS==.Windows { + @(private="file") + engine_pdb :: proc(output:string,program:bool) -> string { + stem:=strings.trim_suffix(output,program?project.EXECUTABLE_SUFFIX:project.LIBRARY_SUFFIX) + return fmt.tprintf("%s%s",stem,program?"-program.pdb":".pdb") + } +} // Whether an installed engine has every binary its package carries. @(private) @@ -964,7 +1012,8 @@ engine_cache_take :: proc(entry:string,paths:Engine_Paths,library:Engine_Library from:=project.path(entry,name,allocator=context.temp_allocator) // The debug information is there only when the library was built with it. if at==1 && !os.is_file(from) {continue} - if os.copy_file(project.path(paths.folder,name,allocator=context.temp_allocator),from)!=nil {return false} + to:=project.path(paths.folder,name,allocator=context.temp_allocator) + if !copy_binary(to,from) {return false} } cache_entry_used(entry) return true @@ -1230,26 +1279,26 @@ engine_fail :: proc(failure:Failure,step,text:string) -> Engine_Result { } // What the programs load besides the libraries of the engine: SDL3. -@(private="file") -engine_copies :: proc(root,folder,compiler:string) -> []Copy { - copies:=make([dynamic]Copy,context.temp_allocator) +@(private) +sdl_copy :: proc(root,folder,compiler:string,allocator:=context.allocator) -> Copy { + from,name:string when ODIN_OS==.Windows { odin_root,configured:=os.lookup_env("ODIN_ROOT",context.temp_allocator) if !configured || odin_root=="" { cut:=strings.last_index_any(compiler,"/\\") odin_root=cut>=0?compiler[:cut]:"." } - if sdl:=project.path(odin_root,"vendor/sdl3/SDL3.dll",allocator=context.temp_allocator);os.is_file(sdl) { - append(&copies,Copy{sdl,project.path(folder,"SDL3.dll",allocator=context.temp_allocator),false}) - } + name="SDL3.dll" + from=project.path(odin_root,"vendor/sdl3",name,allocator=allocator) } else when ODIN_OS==.Linux { - if sdl:=project.path(linux_deps(root),"sdl3/lib/libSDL3.so.0",allocator=context.temp_allocator);os.is_file(sdl) { - append(&copies,Copy{sdl,project.path(folder,"libSDL3.so.0",allocator=context.temp_allocator),false}) - } + name="libSDL3.so.0" + from=project.path(linux_deps(root),"sdl3/lib",name,allocator=allocator) } else when ODIN_OS==.Darwin { if sdl,relocatable:=macos_sdl(root);relocatable { - append(&copies,Copy{project.path(sdl,MACOS_SDL,allocator=context.temp_allocator),project.path(folder,MACOS_SDL,allocator=context.temp_allocator),false}) + name=MACOS_SDL + from=project.path(sdl,name,allocator=allocator) } } - return copies[:] + if from=="" || !os.is_file(from) {return {}} + return {from=from,to=project.path(folder,name,allocator=allocator),binary=true} } diff --git a/cook/build/engine_unlock.odin b/cook/build/engine_unlock.odin deleted file mode 100644 index 3ada5ba0a..000000000 --- a/cook/build/engine_unlock.odin +++ /dev/null @@ -1,70 +0,0 @@ -package build - -import "core:fmt" -import "core:os" -import "core:strings" -import "core:time" -import "mirai:project" - -// A program that runs holds the binaries it loaded: Windows neither replaces -// nor deletes them, but it does move them. Before a build writes, each binary -// of the folder that something runs moves to ENGINE_RETIRED and a copy takes -// its place, which the build replaces or keeps: what runs goes on with the -// binaries it started with, and what starts afterwards takes the new ones. A -// moved binary is deleted by a later build once nothing runs it. - -ENGINE_RETIRED :: "retired" - -// Moves aside each binary of folder that cannot be written, leaving a copy -// in its place, after deleting the ones moved before that nothing runs any -// more. ok is false, with the file, when one could not be moved or copied. -engine_unlock :: proc(folder:string) -> (file:string,ok:bool) { - retired:=project.path(folder,ENGINE_RETIRED,allocator=context.temp_allocator) - engine_retired_sweep(retired) - entries,error:=os.read_all_directory_by_path(folder,context.temp_allocator) - if error!=nil {return "",true} - stamp:=time.time_to_unix_nano(time.now()) - for entry in entries { - if entry.type!=.Regular || !engine_binary(entry.name) || engine_writable(entry.fullpath) {continue} - if os.make_directory_all(retired)!=nil && !os.is_dir(retired) {return retired,false} - moved:=project.path(retired,fmt.tprintf("%x-%s",stamp,entry.name),allocator=context.temp_allocator) - if os.rename(entry.fullpath,moved)!=nil {return entry.fullpath,false} - if os.copy_file(entry.fullpath,moved)!=nil { - // Back where it was rather than missing: the build stops at it as before. - _=os.rename(moved,entry.fullpath) - return entry.fullpath,false - } - } - return "",true -} - -// Deletes the binaries moved aside that nothing runs; the others stay for a later build. -engine_retired_sweep :: proc(retired:string) { - entries,error:=os.read_all_directory_by_path(retired,context.temp_allocator) - if error!=nil {return} - left:=false - for entry in entries { - if os.remove(entry.fullpath)!=nil {left=true} - } - if !left {_=os.remove(retired)} -} - -// What a program loads or a debugger holds open: what a build writes and a -// running program keeps from being written. -@(private) -engine_binary :: proc(name:string) -> bool { - // The copy a program loaded to take a library in place (host.engine_library_swap) is its own. - if strings.contains(name,"-live-") {return false} - for extension in ([?]string{".dll",".exe",".pdb",".so",".dylib"}) { - if strings.has_suffix(name,extension) {return true} - } - return strings.index_byte(name,'.')<0 // a program of Linux or macOS -} - -@(private) -engine_writable :: proc(path:string) -> bool { - f,error:=os.open(path,{.Write}) - if error!=nil {return false} - os.close(f) - return true -} diff --git a/cook/build/engine_unlock_test.odin b/cook/build/engine_unlock_test.odin deleted file mode 100644 index 4b2a5f25a..000000000 --- a/cook/build/engine_unlock_test.odin +++ /dev/null @@ -1,45 +0,0 @@ -#+build windows -package build - -import "core:os" -import win "core:sys/windows" -import "core:testing" -import "mirai:project" -import "mirai:std/base" - -// A binary something holds is moved aside and a copy takes its place, which the build can write; the -// others stay as they are, and a later build deletes what was moved once nothing holds it. -@(test) -a_binary_in_use_moves_aside_and_leaves_a_copy_to_write :: proc(t:^testing.T) { - folder:=base.test_folder(t,"engine-unlock") - if folder=="" {return} - held:=project.path(folder,"mirai-engine.dll",allocator=context.temp_allocator) - free:=project.path(folder,"mirai-editor.dll",allocator=context.temp_allocator) - stamp:=project.path(folder,ENGINE_STAMP,allocator=context.temp_allocator) - for path in ([?]string{free,stamp}) {testing.expect(t,os.write_entire_file(path,transmute([]u8)path)==nil)} - // A library this process loads: Windows holds it as it holds the binaries of a running engine. - testing.expect(t,os.copy_file(held,"C:/Windows/System32/version.dll")==nil) - original,_:=os.read_entire_file(held,context.temp_allocator) - retired:=project.path(folder,ENGINE_RETIRED,allocator=context.temp_allocator) - { - handle:=win.LoadLibraryW(win.utf8_to_wstring(held)) - testing.expect(t,handle!=nil) - testing.expect(t,!engine_writable(held),"loaded, it cannot be written") - file,ok:=engine_unlock(folder) - testing.expectf(t,ok,"it moved aside: %s",file) - testing.expect(t,engine_writable(held),"the copy in its place can be written") - data,_:=os.read_entire_file(held,context.temp_allocator) - testing.expect(t,string(data)==string(original),"the copy is the same library") - moved,_:=os.read_all_directory_by_path(retired,context.temp_allocator) - testing.expect_value(t,len(moved),1) - engine_unlock(folder) - testing.expect(t,os.is_dir(retired),"what is still held stays") - win.FreeLibrary(handle) - } - engine_unlock(folder) - testing.expect(t,!os.exists(retired),"once nothing holds it, it is deleted") - for path in ([?]string{free,stamp}) { - data,_:=os.read_entire_file(path,context.temp_allocator) - testing.expect_value(t,string(data),path) - } -} diff --git a/cook/build/game.odin b/cook/build/game.odin index 1cbe86108..e9950e511 100644 --- a/cook/build/game.odin +++ b/cook/build/game.odin @@ -18,37 +18,24 @@ game_copies :: proc(p:^project.Project,folder,compiler:string,a:runtime.Allocato copies:=make([dynamic]Copy,a) names:=project.FOLDER_NAMES parts:=project.PART_NAMES - append(&copies,Copy{project.path(p.root,project.PROJECT_FILE,allocator=a),project.path(folder,project.PROJECT_FILE,allocator=a),false}) + append(&copies,Copy{ + from=project.path(p.root,project.PROJECT_FILE,allocator=a), + to=project.path(folder,project.PROJECT_FILE,allocator=a), + }) if icon:=base.text_get(p.manifest.icon[:]);icon!="" && !server { - if from:=project.path(p.root,icon,allocator=a);os.is_file(from) {append(&copies,Copy{from,project.path(folder,icon,allocator=a),false})} + if from:=project.path(p.root,icon,allocator=a);os.is_file(from) { + append(&copies,Copy{from=from,to=project.path(folder,icon,allocator=a)}) + } } if !server {shaders_copies(&copies,p.root,folder,a)} // What the lists of the renderer asked for in each scene the runs of the // project opened: the game starts its lists there and goes on filling it. if capacities:=project.cache_path(p.root,project.CAPACITY_CACHE,allocator=a);!server && os.is_file(capacities) { - append(&copies,Copy{capacities,project.cache_path(folder,project.CAPACITY_CACHE,allocator=a),false}) - } - when ODIN_OS==.Windows { - // SDL3 is linked dynamically and ships with the compiler. - odin_root,configured:=os.lookup_env("ODIN_ROOT",context.temp_allocator) - if !configured || odin_root=="" { - cut:=strings.last_index_any(compiler,"/\\") - odin_root=cut>=0?compiler[:cut]:"." - } - if sdl:=project.path(odin_root,"vendor/sdl3/SDL3.dll",allocator=a);os.is_file(sdl) { - append(&copies,Copy{sdl,project.path(folder,"SDL3.dll",allocator=a),false}) - } - } else when ODIN_OS==.Linux { - // The SDL3 of scripts/setup_linux_deps.py: platform_flags makes the game look for it next to itself. - if sdl:=project.path(linux_deps(p.engine_root),"sdl3/lib/libSDL3.so.0",allocator=a);os.is_file(sdl) { - append(&copies,Copy{sdl,project.path(folder,"libSDL3.so.0",allocator=a),false}) - } - } else when ODIN_OS==.Darwin { - // The SDL3 of scripts/setup_macos_deps.py: platform_flags makes the game look for it next to itself. - if sdl,relocatable:=macos_sdl(p.engine_root);relocatable { - append(&copies,Copy{project.path(sdl,MACOS_SDL,allocator=a),project.path(folder,MACOS_SDL,allocator=a),false}) - } + append(&copies,Copy{ + from=capacities,to=project.cache_path(folder,project.CAPACITY_CACHE,allocator=a), + }) } + if sdl:=sdl_copy(p.engine_root,folder,compiler,a);sdl.from!="" {append(&copies,sdl)} return copies[:] } @@ -68,10 +55,14 @@ job_ship :: proc(job:^Job) -> Failure { @(private) copies_run :: proc(copies:[]Copy) -> bool { for copy in copies { - parent:=copy.to[:strings.last_index_byte(copy.to,'/')] + parent,name:=os.split_path(copy.to) + if name=="" {return false} + if parent=="" {parent="."} if os.make_directory_all(parent)!=nil && !os.is_dir(parent) {return false} if copy.folder { if base.copy_directory_all(copy.to,copy.from)!=nil {return false} + } else if copy.binary { + if !copy_binary(copy.to,copy.from) {return false} } else if os.copy_file(copy.to,copy.from)!=nil { return false } diff --git a/cook/build/generate.odin b/cook/build/generate.odin index ca5ab1ed5..89f625636 100644 --- a/cook/build/generate.odin +++ b/cook/build/generate.odin @@ -41,8 +41,8 @@ generate :: proc(p:^project.Project,target:Target,allocator:=context.allocator) } // Writes the generated packages. The errors of out are those of the attributes of the project's packages. -generated_write :: proc(p:^project.Project,target:Target) -> (out:Generated,ok:bool) { - return world_write_files(p,target==.Game) +generated_write :: proc(p:^project.Project,target:Target,store_root:="") -> (out:Generated,ok:bool) { + return world_write_files(p,target==.Game,store_root) } // The flags every compilation of a project's generated code needs. diff --git a/cook/build/imports.odin b/cook/build/imports.odin index f75d9b5f6..b1e3ee9bd 100644 --- a/cook/build/imports.odin +++ b/cook/build/imports.odin @@ -3,35 +3,60 @@ package build import "core:fmt" import "core:os" import "core:strings" +import "mirai:asset" import "mirai:project" import "mirai:std/subprocess" -// Brings the imported folder of each root (a project, a plugin) up to date -// with the sources of its assets folder (cook/import/sources.sync), with the -// import program of the engine (Engine_Paths.import_program, of the defines -// the engine was built with), which reads the formats the plugins of the -// project add from the library of its editor: the tool builds without the -// importers. Textures are built Final for a build that cooks them (final) -// and Fast, as the editor imports them, for one that does not. What the program -// says of each source it imports goes to the output of this process as it -// says it: a large source imports for minutes. message, in the temp -// allocator: why it failed. -imports_sync :: proc(program:string,roots:[]string,final:bool) -> (ok:bool,message:string) { +// Brings the packages each root (a project, a plugin) pins in its +// packages.mirai to builds this engine reads (cook/import/sources.project_refresh) +// in the store at store_root ("": the machine's), with the import program of +// the engine (Engine_Paths.import_program, of the defines the engine was +// built with), which reads the formats the plugins of the project add from +// the library of its editor: the tool builds without the importers. A build +// is taken from the store, then from the store at from ("": none; an +// installed engine adds the store of the builds of its plugins), then built +// from its source when the source is on this machine; a package none of them +// gives fails by name. Textures are required Final for a build that cooks +// them (final) and taken Fast, as the editor imports them, for one that does +// not. What the program says of each source it imports goes to the output +// of this process as it says it: a large source imports for minutes. +// message, in the temp allocator: why it failed, with the source and the +// reason the program gave, or, when it ended without saying (a crash, a +// signal), how it ended and the source it was importing. +imports_refresh :: proc(program:string,roots:[]string,final:bool,store_root:string,from:="") -> (ok:bool,message:string) { if len(roots)==0 {return true,""} command:=make([dynamic]string,context.temp_allocator) append(&command,program) if final {append(&command,"--final")} - for root in roots {append(&command,"--sync",root)} - state,_,errors,error:=subprocess.exec({command=command[:]},context.temp_allocator,echo=os.to_writer(os.stdout)) + if store_root!="" {append(&command,"--store",store_root)} + if from!="" {append(&command,"--from",from)} + for root in roots {append(&command,"--refresh",root)} + state,said,errors,error:=subprocess.exec({command=command[:]},context.temp_allocator,echo=os.to_writer(os.stdout)) if error!=nil {return false,fmt.tprintf("the import program could not run: %v",error)} - if state.exit_code!=0 {return false,strings.trim_space(string(errors))} - return true,"" + if state.exit_code==0 && state.success {return true,""} + return false,imports_failure(state,string(said),string(errors)) } -// The roots of a project whose sources its builds import: the project and -// its active plugins, those with an assets folder. In the temp allocator. +// Why the import program failed: what it said on its error output, else how +// it ended and the last source it said it was importing. +@(private) +imports_failure :: proc(state:os.Process_State,said,errors:string) -> string { + if text:=strings.trim_space(errors);text!="" {return text} + ended:=fmt.tprintf("the import program ended with exit code %d",state.exit_code) + if !state.success {ended=fmt.tprintf("the import program crashed or was killed (code %d)",state.exit_code)} + lines:=strings.split_lines(said,context.temp_allocator) + #reverse for line in lines { + if strings.has_prefix(line,"importing ") {return fmt.tprintf("%s while %s, without saying why: run the import again (mirai import --refresh) to see it",ended,line)} + } + return fmt.tprintf("%s before importing any source, without saying why: run the import again (mirai import --refresh) to see it",ended) +} + +// The roots of a project whose packages its builds refresh: the project and +// its active plugins, those with a packages.mirai. In the temp allocator. imports_roots :: proc(p:^project.Project) -> []string { roots:=make([dynamic]string,context.temp_allocator) - for folder in project.asset_folders(p,context.temp_allocator) {append(&roots,folder.root)} + for folder in project.asset_folders(p,context.temp_allocator) { + if os.is_file(project.path(folder.root,asset.PACKAGES_FILE,allocator=context.temp_allocator)) {append(&roots,folder.root)} + } return roots[:] } diff --git a/cook/build/imports_test.odin b/cook/build/imports_test.odin new file mode 100644 index 000000000..40b572ad0 --- /dev/null +++ b/cook/build/imports_test.odin @@ -0,0 +1,24 @@ +package build + +import "core:os" +import "core:strings" +import "core:testing" + +// A failed refresh says why: what the import program said on its error +// output, else how it ended and the source it was importing. +@(test) +a_failed_refresh_names_the_source_and_how_the_program_ended :: proc(t:^testing.T) { + said:="package 1 of 2\nimporting /p/assets/rock.png: its source changed\n rock: 3 assets, 0.2 s\npackage 2 of 2\nimporting /p/assets/tree.gltf: this engine reads no build of it here\n" + Case :: struct {state:os.Process_State,said,errors:string,wanted:[]string} + cases:=[]Case{ + {{exited=true,exit_code=1,success=true},said,"mirai import: package tree: the file is truncated\n",{"package tree: the file is truncated"}}, + {{exited=true,exit_code=9,success=false},said,"",{"crashed or was killed (code 9)","/p/assets/tree.gltf"}}, + {{exited=true,exit_code=3,success=true},said,"",{"exit code 3","/p/assets/tree.gltf"}}, + {{exited=true,exit_code=9,success=false},"","",{"crashed or was killed (code 9)","before importing any source"}}, + } + for c in cases { + message:=imports_failure(c.state,c.said,c.errors) + for wanted in c.wanted {testing.expectf(t,strings.contains(message,wanted),"%q lacks %q",message,wanted)} + testing.expectf(t,!strings.contains(message,"rock.png"),"only the source it was importing: %q",message) + } +} diff --git a/cook/build/plugins.odin b/cook/build/plugins.odin index 060690f7d..b4b8d9746 100644 --- a/cook/build/plugins.odin +++ b/cook/build/plugins.odin @@ -386,16 +386,15 @@ plugin_cache_trim :: proc(engine_root:string) { cache_trim(workspace.cache_folder(engine_root,"plugins",context.temp_allocator),PLUGIN_CACHE_ENTRIES) } -// Copies the library of an entry to staged. False when the cache has none or +// Publishes the library of an entry. False when the cache has none or // it cannot be copied. @(private) -plugin_cache_take :: proc(engine_root:string,lib:^Plugin_Library,hash:u64,staged:string) -> bool { +plugin_cache_take :: proc(engine_root:string,lib:^Plugin_Library,hash:u64) -> bool { entry:=plugin_cache_entry(engine_root,lib,hash) if !cache_entry_complete(entry) {return false} - if os.copy_file(staged,project.path(entry,project.library_file(lib.file,context.temp_allocator),allocator=context.temp_allocator))!=nil { - _=os.remove(staged) - return false - } + name:=project.library_file(lib.file,context.temp_allocator) + from:=project.path(entry,name,allocator=context.temp_allocator) + if !copy_binary(lib.library,from) {return false} cache_entry_used(entry) return true } diff --git a/cook/build/program_uses.odin b/cook/build/program_uses.odin index a0cd1d8f5..3f180fa77 100644 --- a/cook/build/program_uses.odin +++ b/cook/build/program_uses.odin @@ -87,14 +87,25 @@ program_uses :: proc(p:^project.Project,allocator:=context.allocator) -> (uses:[ } } for folder in project.asset_folders(p,context.temp_allocator) { - for root in ([2]string{folder.path,folder.imported}) { - if root=="" || !os.is_dir(root) {continue} - w:=os.walker_create(root) + if os.is_dir(folder.path) { + w:=os.walker_create(folder.path) for info in os.walker_walk(&w) { if info.type==.Regular {visit(strings.clone(info.fullpath,context.temp_allocator),&types,kinds,&materials,&meshes,&drawings,&looks)} } os.walker_destroy(&w) } + // The meshes of the packages the root pins are blobs of the store: the + // materials each depends on are those its manifest lists. + list,_,_:=asset.packages_load(folder.root,context.temp_allocator) + for &entry in list.packages { + text,read_error:=os.read_entire_file(asset.package_file(folder.root,entry.id),context.temp_allocator) + if read_error!=nil {continue} + pinned,parse_error:=asset.package_parse(string(text),context.temp_allocator) + if parse_error!=.None {continue} + for &a in pinned.assets { + if a.kind==asset.KIND_MESH {meshes[a.id]=a.dependencies[:]} + } + } } found:=make(map[string]abi.Program_Uses,context.temp_allocator) add :: proc(found:^map[string]abi.Program_Uses,materials:map[base.Asset_Id]Material,id:base.Asset_Id,uses:abi.Program_Uses) { diff --git a/cook/build/publish_posix.odin b/cook/build/publish_posix.odin new file mode 100644 index 000000000..f4de1f56c --- /dev/null +++ b/cook/build/publish_posix.odin @@ -0,0 +1,9 @@ +#+build !windows +package build + +import "core:os" + +@(private) +publish :: proc(staged,published:string,retired:="") -> bool { + return os.rename(staged,published)==nil +} diff --git a/cook/build/publish_windows.odin b/cook/build/publish_windows.odin new file mode 100644 index 000000000..d0e868c2a --- /dev/null +++ b/cook/build/publish_windows.odin @@ -0,0 +1,47 @@ +#+build windows +package build + +import "core:os" +import "core:strings" + +@(private) +publish :: proc(staged,published:string,retired:="") -> bool { + retired:=retired + folder:string + if retired=="" { + parent,_:=os.split_path(published) + if parent=="" {parent="."} + path,alloc_error:=strings.concatenate({parent,"/",ENGINE_RETIRED},context.temp_allocator) + if alloc_error!=nil {return false} + folder=path + engine_retired_sweep(folder) + } + if os.rename(staged,published)==nil {return true} + // Windows can rename a loaded image, but cannot replace it at its current name. + info,error:=os.lstat(published,context.temp_allocator) + if error!=nil || info.type!=.Regular {return false} + defer if folder!="" {_=os.remove(folder)} + if retired=="" { + if os.make_directory(folder)!=nil && !os.is_dir(folder) {return false} + name,ok:=binary_temp_path(folder) + if !ok {return false} + retired=name + } + if os.rename(published,retired)!=nil {return false} + if os.rename(staged,published)==nil {return true} + _=os.rename(retired,published) + return false +} + +@(private) +ENGINE_RETIRED :: ".mirai-retired" + +@(private) +engine_retired_sweep :: proc(folder:string) { + entries,error:=os.read_all_directory_by_path(folder,context.temp_allocator) + if error!=nil {return} + for entry in entries { + if strings.has_prefix(entry.name,BINARY_TEMP_PREFIX) {_=os.remove(entry.fullpath)} + } + _=os.remove(folder) +} diff --git a/cook/build/shaders.odin b/cook/build/shaders.odin index f22e96d93..31019d6e5 100644 --- a/cook/build/shaders.odin +++ b/cook/build/shaders.odin @@ -254,6 +254,9 @@ shaders_copies :: proc(copies:^[dynamic]Copy,root,folder:string,a:runtime.Alloca for entry in entries { if entry.type!=.Regular {continue} if entry.name!=project.SHADERS_MANIFEST && entry.name!="catalog.json" && !strings.has_suffix(entry.name,CODE) {continue} - append(copies,Copy{project.path(built,entry.name,allocator=a),project.path(folder,project.SHADERS_SHIPPED,entry.name,allocator=a),false}) + append(copies,Copy{ + from=project.path(built,entry.name,allocator=a), + to=project.path(folder,project.SHADERS_SHIPPED,entry.name,allocator=a), + }) } } diff --git a/cook/build/world.odin b/cook/build/world.odin index 43479a0fd..325e98b22 100644 --- a/cook/build/world.odin +++ b/cook/build/world.odin @@ -303,8 +303,10 @@ generator_read :: proc(p:^project.Project) -> (g:Generator) { } // The source of the world packages and of the main packages. The world of -// the game that ships is generated only when shipped is asked for. -world_generate :: proc(p:^project.Project,allocator:=context.allocator,shipped:=false) -> (out:Generated) { +// the game that ships is generated only when shipped is asked for. The paths +// of assets the code writes are checked against the packages of the project +// and its plugins in the store at store_root ("": the machine's). +world_generate :: proc(p:^project.Project,allocator:=context.allocator,shipped:=false,store_root:="") -> (out:Generated) { context.allocator=allocator g:=generator_read(p) for pkg in g.packages { @@ -317,7 +319,7 @@ world_generate :: proc(p:^project.Project,allocator:=context.allocator,shipped:= for pkg in g.packages {draws_read(&g,pkg)} rules_check(&g) defaults_check(&g) - asset_paths_check(&g,p) + asset_paths_check(&g,p,store_root) out.errors=g.errors[:] if len(g.errors)>0 {return} out.game_world=world_write(&g,false) @@ -382,10 +384,10 @@ PROJECT_INFO_SYMBOL :: "mirai_project_info" // Writes the generated packages under /.mirai/gen when they changed. // out has what was generated, in the temp allocator. -world_write_files :: proc(p:^project.Project,shipped:=false) -> (out:Generated,ok:bool) { +world_write_files :: proc(p:^project.Project,shipped:=false,store_root:="") -> (out:Generated,ok:bool) { if out.errors=kinds_write(p);len(out.errors)>0 {return} if out.errors=records_write(p);len(out.errors)>0 {return} - out=world_generate(p,context.temp_allocator,shipped) + out=world_generate(p,context.temp_allocator,shipped,store_root) if len(out.errors)>0 {return} gen:=project.path(project.folder_path(p.root,.Generated,context.temp_allocator),"gen",allocator=context.temp_allocator) files:=[4][2]string{{"editor_world/world.odin",out.editor_world},{"shipped_world/world.odin",out.shipped_world},{"editor/main.odin",out.editor_main},{"game/main.odin",out.game_main}} @@ -3364,17 +3366,20 @@ defaults_check :: proc(g:^Generator) { // The paths of assets the code of the project writes: a string literal that // ends in the extension of an asset ("pets/Dog.scene") names an asset of the -// project or of its plugins, or the build fails where it is written, with the -// path that has the same name when there is one. +// project, of its plugins or of their packages, or the build fails where it +// is written, with the path that has the same name when there is one. @(private="file") -asset_paths_check :: proc(g:^Generator,p:^project.Project) { +asset_paths_check :: proc(g:^Generator,p:^project.Project,store_root:string) { assets:=project.folder_path(p.root,.Assets,context.temp_allocator) if !os.is_dir(assets) {return} registry:asset.Registry - asset.registry_init(®istry,assets,"",context.temp_allocator,project.imported_path(p.root,context.temp_allocator)) + packages,_:=asset.root_packages(p.root,store_root,context.temp_allocator) + asset.registry_init(®istry,assets,"",context.temp_allocator,packages.listed) defer asset.registry_destroy(®istry) for folder in project.asset_folders(p,context.temp_allocator) { - if folder.prefix!="" {asset.registry_mount(®istry,folder.path,folder.prefix,folder.imported)} + if folder.prefix=="" {continue} + plugin,_:=asset.root_packages(folder.root,store_root,context.temp_allocator) + asset.registry_mount(®istry,folder.path,folder.prefix,plugin.listed) } if asset.registry_scan(®istry)!=.None {return} extensions:=make(map[string]bool,context.temp_allocator) diff --git a/cook/cache/cache.odin b/cook/cache/cache.odin index df6188f63..069f09742 100644 --- a/cook/cache/cache.odin +++ b/cook/cache/cache.odin @@ -1,78 +1,201 @@ package cache import "core:fmt" -import "core:os" -import "core:time" +import "core:strings" import "mirai:asset" import "mirai:std/base" +import "mirai:store" -// What an import or a cook built, kept by what it was made from in a folder -// of the caches of a project (project.cache_path), so that building it again -// reads it instead. A file of the cache is an asset whose id is its key and -// the version of what built it: a builder that gives other bytes misses every -// earlier file. A file that does not read back whole is a miss too. A read -// marks the file used, and a file no build used for UNUSED_AGE is removed: -// whatever its size, the cache keeps what the builds of the last weeks read, -// so a project larger than any fixed budget does not evict what its next -// build reads. Deleting the folder costs only the time of building again. - -// A file of the cache no build used for this long is removed. -UNUSED_AGE :: 30*24*time.Hour -// A folder is looked over for unused files at most this often, after a -// write: one stat of its stamp otherwise. -TRIM_EVERY :: 24*time.Hour -// The file of a folder whose modification time is when it was last looked over. -TRIM_STAMP :: ".trimmed" +// What an import or a cook built, kept by what it was made from in the +// machine store, so that building it again reads it instead, here or on +// another machine that shares the store's objects with this one. A result is +// the file of an asset, kept as an object (asset.object_put), whose id is its +// key and the version of what built it; the ref (cache-, +// -) names it (asset.cache_namespace, ref_name), so a builder that gives +// other bytes misses every earlier result. A ref whose object is not here +// whole or does not read is a miss. The store owns space and checks its +// chunks (its scrub): the cache removes nothing. + +// Asked on a miss before building: for each of names, makes the object of the +// ref (namespace, name) whole in into and sets the ref there; given (as +// long as names, all false when asked) says for which it did. Names known +// together are asked together, so a source that asks peers waits for them once. +Fetch :: #type proc(user:rawptr,into:^store.Store,namespace:string,names:[]string,given:[]bool) + +Source :: struct { + fetch:Fetch, // nil: none + user:rawptr, +} + +Cache :: struct { + store:^store.Store, // nil: nothing is kept + source:Source, +} + +// The layout of what a result is: raised when results are kept otherwise, so a +// result of another layout is never read. 3: an object of the store, as Codecs.None writes it. +RESULTS_VERSION :: 3 // The id of the file of key, built by version. id :: proc "contextless" (key,version:u64) -> base.Asset_Id {return {key,version}} -// Where the file of key goes, in the temp allocator. -file_path :: proc(folder:string,key:u64,extension:string) -> string { - return fmt.tprintf("%s/%016x%s",folder,key,extension) +// The file of key built by version in the cache name, open in file for the +// caller to close; it reads nothing more of the store. On a miss the source is +// asked once. False without one, or with one that does not read whole. +load :: proc(c:Cache,name:string,key,version:u64,file:^asset.Asset,allocator:=context.allocator) -> bool { + namespace_buffer:[store.NAME_MAX]u8 + ref_buffer:[REF_BYTES]u8 + namespace,ref,ok:=names(c,name,key,version,&namespace_buffer,&ref_buffer) + if !ok {return false} + if load_here(c.store,namespace,ref,key,version,file,allocator) {return true} + return fetched(c,namespace,ref) && load_here(c.store,namespace,ref,key,version,file,allocator) } -// The file of key built by version, open in file for the caller to close and -// marked used. False without one, or with one that does not read back whole. -load :: proc(folder:string,key,version:u64,extension:string,file:^asset.Asset,allocator:=context.allocator) -> bool { - if folder=="" {return false} - path:=file_path(folder,key,extension) - if asset.asset_load(file,path,allocator)!=.None {return false} - if file.blob.header.id!=id(key,version) || !asset.blob_verify_hash(&file.blob) { - asset.asset_close(file) - return false +// Whether the cache name keeps a whole object for key built by version, +// without reading it; the source is asked on a miss. +has :: proc(c:Cache,name:string,key,version:u64) -> bool { + namespace_buffer:[store.NAME_MAX]u8 + ref_buffer:[REF_BYTES]u8 + namespace,ref,ok:=names(c,name,key,version,&namespace_buffer,&ref_buffer) + if !ok {return false} + if has_here(c.store,namespace,ref) {return true} + return fetched(c,namespace,ref) && has_here(c.store,namespace,ref) +} + +// Which of keys, each built by version, the cache name keeps whole, into +// here (as long as keys): what is not here is asked of the source in one +// fetch. What a pass needs is looked up so, once, and its builds then load +// with a cache without source. +lookup :: proc(c:Cache,name:string,keys:[]u64,version:u64,here:[]bool) { + for &h in here {h=false} + if len(here)!=len(keys) {return} + namespace_buffer:[store.NAME_MAX]u8 + ref_buffer:[REF_BYTES]u8 + namespace,_,ok:=names(c,name,0,version,&namespace_buffer,&ref_buffer) + if !ok {return} + lacking:=make([dynamic]string,context.temp_allocator) + at:=make([dynamic]int,context.temp_allocator) + for key,k in keys { + name_buffer:[REF_BYTES]u8 + ref:=strings.clone(ref_name(key,version,&name_buffer),context.temp_allocator) + if here[k]=has_here(c.store,namespace,ref);here[k] {continue} + if _,no_room:=append(&lacking,ref);no_room!=nil {return} + if _,no_room:=append(&at,k);no_room!=nil {return} + } + if len(lacking)==0 || c.source.fetch==nil {return} + given:=make([]bool,len(lacking),context.temp_allocator) + c.source.fetch(c.source.user,c.store,namespace,lacking[:],given) + for ref,i in lacking { + if given[i] {here[at[i]]=has_here(c.store,namespace,ref)} } - now:=time.now() - _=os.change_times(path,now,now) - return true } -// Whether a file can be written into folder, which is made when missing. -ready :: proc(folder:string) -> bool { - if folder=="" {return false} - return os.is_dir(folder) || os.make_directory_all(folder)==nil +// Keeps bytes, the file of an asset whose id is id(key,version), as the +// result of key in the cache name, committed. False when it cannot be kept, +// which costs only the time of building it the next time. +put :: proc(c:Cache,name:string,key,version:u64,bytes:[]u8) -> bool { + namespace_buffer:[store.NAME_MAX]u8 + ref_buffer:[REF_BYTES]u8 + namespace,ref,ok:=names(c,name,key,version,&namespace_buffer,&ref_buffer) + if !ok {return false} + object,error:=asset.object_put(c.store,bytes) + return error==.None && store.ref_set(c.store,namespace,ref,object)==.None +} + +@(private) +REF_BYTES :: 33 // -, each 16 hex digits + +// The namespace and the name of the ref of key built by version; not ok +// without a store or for a name the store refuses. +@(private) +names :: proc(c:Cache,name:string,key,version:u64,namespace_buffer:^[store.NAME_MAX]u8,ref_buffer:^[REF_BYTES]u8) -> (namespace,ref:string,ok:bool) { + if c.store==nil || name=="" {return} + namespace=asset.cache_namespace(name,namespace_buffer) + if !store.name_valid(namespace) {return} + return namespace,ref_name(key,version,ref_buffer),true } -// Removes the files of folder no build used for UNUSED_AGE, and marks the -// folder looked over. How many went. -trim :: proc(folder:string) -> (removed:int) { - now:=time.now() - entries,error:=os.read_all_directory_by_path(folder,context.temp_allocator) - if error!=nil {return} - for entry in entries { - if entry.type!=.Regular || entry.name==TRIM_STAMP || time.diff(entry.modification_time,now)<=UNUSED_AGE {continue} - if os.remove(entry.fullpath)==nil {removed+=1} +// -, the version combined with RESULTS_VERSION. +@(private) +ref_name :: proc(key,version:u64,buffer:^[REF_BYTES]u8) -> string { + return fmt.bprintf(buffer[:],"%016x-%016x",key,base.hash_combine(version,RESULTS_VERSION)) +} + +@(private) +fetched :: proc(c:Cache,namespace,ref:string) -> bool { + return source_fetch(c.source,c.store,namespace,ref) +} + +// Asks source for the one ref (namespace, name); false without a source. +source_fetch :: proc(source:Source,into:^store.Store,namespace,name:string) -> bool { + if source.fetch==nil {return false} + names:=[1]string{name} + given:[1]bool + source.fetch(source.user,into,namespace,names[:],given[:]) + return given[0] +} + +@(private) +has_here :: proc(s:^store.Store,namespace,ref:string) -> bool { + object,error:=store.ref_get(s,namespace,ref) + return error==.None && asset.object_whole(s,object) +} + +@(private) +load_here :: proc(s:^store.Store,namespace,ref:string,key,version:u64,file:^asset.Asset,allocator:=context.allocator) -> bool { + object,error:=store.ref_get(s,namespace,ref) + if error!=.None {return false} + if asset.asset_open_object(file,s,object,allocator)!=.None {return false} + if file.blob.header.id==id(key,version) && asset.asset_object_settle(file)==.None {return true} + asset.asset_close(file) + return false +} + +// A source that copies from the store at from, a second root of objects and +// refs: the object of the ref and, for a build (asset.REF_IMPORTS or +// asset.REF_IMPORTS_FAST), every object it lists. What a peer gives, and what an installed engine gives from +// the store of the builds of its plugins. +store_source :: proc(from:^store.Store) -> Source { + fetch :: proc(user:rawptr,into:^store.Store,namespace:string,names:[]string,given:[]bool) { + from:=(^store.Store)(user) + for name,k in names {given[k]=copy_ref(from,into,namespace,name)} } - stamp:=fmt.tprintf("%s/%s",folder,TRIM_STAMP) - if !os.exists(stamp) {_=os.write_entire_file(stamp,nil)} - _=os.change_times(stamp,now,now) - return + copy_ref :: proc(from,into:^store.Store,namespace,name:string) -> bool { + object,error:=store.ref_get(from,namespace,name) + if error!=.None || object_copy(from,into,object)!=.None {return false} + if namespace==asset.REF_IMPORTS || namespace==asset.REF_IMPORTS_FAST { + b,build_error:=asset.build_get(from,object,context.temp_allocator) + if build_error!=.None {return false} + for o in b.outputs { + if object_copy(from,into,o.object)!=.None {return false} + if o.thumbnail!={} && object_copy(from,into,o.thumbnail)!=.None {return false} + } + } + return store.ref_set(into,namespace,name,object)==.None + } + return {fetch=fetch,user=from} } -// Called after a file was written into folder: trims it when it was not -// looked over for TRIM_EVERY. -written :: proc(folder:string) { - stamp,error:=os.stat(fmt.tprintf("%s/%s",folder,TRIM_STAMP),context.temp_allocator) - if error==nil && time.diff(stamp.modification_time,time.now()) store.Error { + if asset.object_whole(into,address) {return .None} + copy_chunk :: proc(from,into:^store.Store,address:store.Hash,level:int,depth:int) -> store.Error { + if depth>store.NODE_LEVEL_MAX+1 {return .Corrupt} + buffer,no_room:=make([]u8,store.CHUNK_MAX,context.temp_allocator) + if no_room!=nil {return .Out_Of_Memory} + defer delete(buffer,context.temp_allocator) + bytes:=store.chunk_read(from,address,buffer) or_return + if store.address_of_bytes(bytes)!=address {return .Corrupt} + if !store.chunk_here(into,address) {store.chunk_put(into,address,bytes) or_return} + if level<0 {return .None} + n:=store.node_parse(bytes) or_return + if level!=DATA_OR_ANY && n.level!=level {return .Corrupt} + for e in n.entries {copy_chunk(from,into,e.address,n.level-1 if n.level>0 else -1,depth+1) or_return} + return .None + } + return copy_chunk(from,into,address,DATA_OR_ANY,0) } + +@(private="file") +DATA_OR_ANY :: 1<<20 // the root: a node of any level diff --git a/cook/cache/cache_test.odin b/cook/cache/cache_test.odin new file mode 100644 index 000000000..dd2dcd334 --- /dev/null +++ b/cook/cache/cache_test.odin @@ -0,0 +1,151 @@ +package cache + +import "core:fmt" +import "core:slice" +import "core:strings" +import "core:testing" +import "mirai:asset" +import "mirai:std/base" +import "mirai:store" + +@(private="file") +NAME :: "textures" + +// The file of a result: a small texture whose id is that of key built by version. +@(private="file") +result_bytes :: proc(types:^asset.Types,key,version:u64,fill:u8) -> []u8 { + pixels:=make([]u8,4*4*4,context.temp_allocator) + for &value in pixels {value=fill} + mips:=[1][]u8{pixels} + doc:asset.Document + if asset.texture_document(&doc,types,{id=id(key,version),width=4,height=4,format=.RGBA8,mips=mips[:]},context.temp_allocator)!=.None {return nil} + defer asset.document_destroy(&doc) + bytes,_:=asset.document_bytes(&doc,context.temp_allocator) + return bytes +} + +@(private="file") +store_at :: proc(t:^testing.T,root:string,loc:=#caller_location) -> (s:store.Store,ok:bool) { + error:store.Error + s,error=store.open(root,context.temp_allocator) + return s,testing.expect_value(t,error,store.Error.None,loc=loc) +} + +// Copies what the store at user keeps, as a peer would give it. +@(private="file") +copy_fetch :: proc(user:rawptr,into:^store.Store,namespace:string,names:[]string,given:[]bool) { + store_source((^store.Store)(user)).fetch(user,into,namespace,names,given) +} + +@(private="file") +object_of :: proc(s:^store.Store,key,version:u64) -> store.Hash { + namespace:[store.NAME_MAX]u8 + name:[REF_BYTES]u8 + hash,_:=store.ref_get(s,asset.cache_namespace(NAME,&namespace),ref_name(key,version,&name)) + return hash +} + +@(test) +a_result_is_found_by_its_ref_through_any_store_of_its_root_and_a_source_fills_a_miss :: proc(t:^testing.T) { + defer free_all(context.temp_allocator) + types:asset.Types;asset.types_init(&types);defer asset.types_destroy(&types) + folder:=base.test_folder(t,"cache-refs") + if folder=="" {return} + root:=fmt.tprintf("%s/a",folder) + writer,writer_ok:=store_at(t,root) + reader,reader_ok:=store_at(t,root) + peer,peer_ok:=store_at(t,fmt.tprintf("%s/b",folder)) + if !writer_ok || !reader_ok || !peer_ok {return} + KEY,VERSION :: 0x1234,7 + file:asset.Asset + + // Kept through one Store value, found through another on the same root. + testing.expect(t,!load({store=&reader},NAME,KEY,VERSION,&file),"an empty cache has nothing") + testing.expect(t,put({store=&writer},NAME,KEY,VERSION,result_bytes(&types,KEY,VERSION,1))) + if testing.expect(t,load({store=&reader},NAME,KEY,VERSION,&file,context.temp_allocator)) { + testing.expect(t,file.blob.header.id==id(KEY,VERSION)) + asset.asset_close(&file) + } + testing.expect(t,has({store=&reader},NAME,KEY,VERSION) && !has({store=&reader},NAME,KEY,VERSION+1),"another version is another result") + + // A miss asks the source, which makes the blob present and sets the ref; one that has none is a miss. + from_writer:=Cache{store=&peer,source={fetch=copy_fetch,user=&writer}} + testing.expect(t,!load({store=&peer},NAME,KEY,VERSION,&file),"without a source the peer's result is not here") + if testing.expect(t,load(from_writer,NAME,KEY,VERSION,&file,context.temp_allocator)) {asset.asset_close(&file)} + testing.expect(t,has({store=&peer},NAME,KEY,VERSION),"what the source gave stays") + testing.expect(t,!load(from_writer,NAME,KEY+1,VERSION,&file) && !has(from_writer,NAME,KEY+1,VERSION)) + + // A ref whose object is not here is a miss, and keeping the result again mends it. + namespace:[store.NAME_MAX]u8 + name:[REF_BYTES]u8 + testing.expect_value(t,store.ref_set(&writer,asset.cache_namespace(NAME,&namespace),ref_name(KEY,VERSION,&name),store.address_of_bytes(transmute([]u8)string("absent"))),store.Error.None) + testing.expect(t,!load({store=&reader},NAME,KEY,VERSION,&file) && !has({store=&reader},NAME,KEY,VERSION)) + testing.expect(t,put({store=&writer},NAME,KEY,VERSION,result_bytes(&types,KEY,VERSION,1))) + if testing.expect(t,load({store=&reader},NAME,KEY,VERSION,&file,context.temp_allocator)) {asset.asset_close(&file)} + + // A whole object of another result under the ref is a miss, and stays: it is whole. + other:=result_bytes(&types,KEY+1,VERSION,2) + testing.expect(t,put({store=&writer},NAME,KEY,VERSION,other)) + testing.expect(t,!load({store=&reader},NAME,KEY,VERSION,&file)) + testing.expect(t,asset.object_whole(&reader,object_of(&reader,KEY,VERSION))) +} + +// A lookup of many keys asks the source once for all it lacks, and says which +// are here after it. +@(test) +a_lookup_of_many_keys_asks_the_source_once :: proc(t:^testing.T) { + defer free_all(context.temp_allocator) + types:asset.Types;asset.types_init(&types);defer asset.types_destroy(&types) + folder:=base.test_folder(t,"cache-lookup") + if folder=="" {return} + here,here_ok:=store_at(t,fmt.tprintf("%s/here",folder)) + peer,peer_ok:=store_at(t,fmt.tprintf("%s/peer",folder)) + if !here_ok || !peer_ok {return} + VERSION :: 3 + keys:=[]u64{10,11,12,13,14} + testing.expect(t,put({store=&peer},NAME,10,VERSION,result_bytes(&types,10,VERSION,1))) + testing.expect(t,put({store=&here},NAME,11,VERSION,result_bytes(&types,11,VERSION,2))) + testing.expect(t,put({store=&peer},NAME,12,VERSION,result_bytes(&types,12,VERSION,3))) + Counted :: struct {from:^store.Store,calls,names:int} + counted:=Counted{from=&peer} + fetch :: proc(user:rawptr,into:^store.Store,namespace:string,names:[]string,given:[]bool) { + c:=(^Counted)(user) + c.calls+=1 + c.names+=len(names) + copy_fetch(c.from,into,namespace,names,given) + } + found:=make([]bool,len(keys),context.temp_allocator) + lookup({store=&here,source={fetch=fetch,user=&counted}},NAME,keys,VERSION,found) + testing.expectf(t,counted.calls==1 && counted.names==4,"one fetch of what is not here: %v",counted) + testing.expectf(t,slice.equal(found,[]bool{true,true,true,false,false}),"%v",found) + file:asset.Asset + if testing.expect(t,load({store=&here},NAME,12,VERSION,&file,context.temp_allocator)) {asset.asset_close(&file)} + lookup({store=&here,source={fetch=fetch,user=&counted}},NAME,keys[:3],VERSION,found[:3]) + testing.expectf(t,counted.calls==1,"what is here asks nothing: %v",counted) +} + +@(test) +hostile_names_are_refused_and_any_key_is_kept_without_a_crash :: proc(t:^testing.T) { + defer free_all(context.temp_allocator) + types:asset.Types;asset.types_init(&types);defer asset.types_destroy(&types) + folder:=base.test_folder(t,"cache-hostile") + if folder=="" {return} + s,ok:=store_at(t,folder) + if !ok {return} + c:=Cache{store=&s,source={fetch=copy_fetch,user=&s}} + bytes:=result_bytes(&types,1,1,3) + file:asset.Asset + for name in ([]string{"","Textures","../up","a/b","a\\b","nul\x00","..",strings.repeat("x",store.NAME_MAX,context.temp_allocator)}) { + testing.expectf(t,!put(c,name,1,1,bytes) && !load(c,name,1,1,&file) && !has(c,name,1,1),"%q is refused",name) + } + for key in ([]u64{0,1,max(u64)}) { + for version in ([]u64{0,max(u64)}) { + kept:=result_bytes(&types,key,version,4) + testing.expect(t,put(c,NAME,key,version,kept)) + if testing.expectf(t,load(c,NAME,key,version,&file,context.temp_allocator),"%x-%x",key,version) {asset.asset_close(&file)} + } + } + testing.expect(t,!put({},NAME,1,1,bytes) && !load({},NAME,1,1,&file) && !has({},NAME,1,1),"without a store nothing is kept") + _=put(c,NAME,9,9,nil) + testing.expect(t,!load(c,NAME,9,9,&file),"bytes that are no asset are no result") +} diff --git a/cook/import/cook.odin b/cook/import/cook.odin index b0fbcd2ec..4d1cf307a 100644 --- a/cook/import/cook.odin +++ b/cook/import/cook.odin @@ -15,10 +15,12 @@ import "mirai:std/jobs" // builds when it gets to it. Either way they are the same bytes. An import // whose files of the cache are current builds nothing (writer_current). // -// The meshes built are kept by what they are made from in the mesh cache of -// the project (Import_Options.mesh_cache, cook/cache), as textures are in the -// texture cache: building the same geometry with the same settings again -// reads it. +// The meshes built are kept by what they are made from in the cache +// MESH_CACHE of the machine store (Import_Options.store, cook/cache), as +// textures are in theirs: building the same geometry with the same settings +// again reads it. + +MESH_CACHE :: "meshes" // The version of what builds a mesh: its file of the cache is built again // when the writer, the format of .mesh or the key of its settings change. @@ -109,12 +111,12 @@ mesh_cook :: proc(mesh:^Imported_Mesh,settings:Import_Settings,options:Import_Op types:asset.Types asset.types_init(&types,context.temp_allocator) defer asset.types_destroy(&types) - folder:=options.mesh_cache + c:=import_cache(options) key:u64 - if folder!="" { + if c.store!=nil { key=mesh_cache_key(cooked.content,settings) file:asset.Asset - if !options.rebuild && cache.load(folder,key,MESH_CACHE_VERSION,".mesh",&file,allocator) { + if !options.rebuild && cache.load(c,MESH_CACHE,key,MESH_CACHE_VERSION,&file,allocator) { if view,error:=asset.mesh_view(&file,&types);error==.None { cooked.geometry=view.geometry cooked.collision={view.colliders,view.collision_points,view.collision_indices} @@ -131,7 +133,7 @@ mesh_cook :: proc(mesh:^Imported_Mesh,settings:Import_Settings,options:Import_Op clustered:=levels_geometry(levels,mesh.flags,&cooked.geometry,allocator) base.profile_end(clusters_zone) if !clustered {cooked.failure=.Clusters;return} - if folder!="" && cache.ready(folder) { + if c.store!=nil { // The slots of the file of the cache name no material: an import names its own. slots:=make([]asset.Material_Slot,len(mesh.slots),context.temp_allocator) for slot,at in mesh.slots {slots[at].id=slot.id} @@ -139,10 +141,21 @@ mesh_cook :: proc(mesh:^Imported_Mesh,settings:Import_Settings,options:Import_Op id=cache.id(key,MESH_CACHE_VERSION),geometry=cooked.geometry,slots=slots,joints=mesh.joints, colliders=cooked.collision.colliders,collision_points=cooked.collision.points,collision_indices=cooked.collision.indices, } - if asset.mesh_save(cache.file_path(folder,key,".mesh"),&types,desc)==.None {cache.written(folder)} + doc:asset.Document + if asset.mesh_document(&doc,&types,desc,context.temp_allocator)!=.None {return} + defer asset.document_destroy(&doc) + if bytes,error:=asset.document_bytes(&doc);error==.None { + _=cache.put(c,MESH_CACHE,key,MESH_CACHE_VERSION,bytes) + mem.free_bytes(bytes) + } } } +// The caches of the store an import reads and fills. +import_cache :: proc(options:Import_Options) -> cache.Cache { + return {store=options.store,source=options.cache_source} +} + // Builds the levels of a texture, or reads them from the texture cache, into // the arena of cooked. bytes are the encoded image, the texels of made when // its width is not 0, or what make makes the texels of; decoded, when given, @@ -158,40 +171,47 @@ texture_cook :: proc(bytes:[]u8,made:textures.Source,decoded:textures.Image,buil asset.types_init(&types,context.temp_allocator) defer asset.types_destroy(&types) // The same image with the same settings, built before by any import, is read from the cache. - folder:=options.texture_cache + c:=import_cache(options) key:u64 - if folder!="" { + if c.store!=nil { key=textures.cache_key_build(bytes,made,build) if !options.rebuild { - if built,cached:=textures.cache_read(folder,key,&types,allocator);cached {cooked.built=built;return} + if built,cached:=textures.cache_read(c,key,&types,allocator);cached {cooked.built=built;return} } } if options.fast_textures { build.speed=.Fast cooked.settings=textures.settings_hash(build) - if folder!="" { + if c.store!=nil { key=textures.cache_key_build(bytes,made,build) if !options.rebuild { - if built,cached:=textures.cache_read(folder,key,&types,allocator);cached {cooked.built=built;return} + if built,cached:=textures.cache_read(c,key,&types,allocator);cached {cooked.built=built;return} } } } jobs.worker_zone(options.worker,"texture.build") switch { case make!=nil && decoded.pixels!=nil: - cooked.built,cooked.error=textures.build(textures.rgba8(decoded.pixels,decoded.width,decoded.height),build,allocator,options.worker) + cooked_set(cooked,textures.build(textures.rgba8(decoded.pixels,decoded.width,decoded.height),build,allocator,options.worker)) case make!=nil: source,ok:=make(bytes,context.temp_allocator) if !ok {cooked.error=.Unsupported;return} - cooked.built,cooked.error=textures.build(source,build,allocator,options.worker) + cooked_set(cooked,textures.build(source,build,allocator,options.worker)) case made.width>0: - cooked.built,cooked.error=textures.build(made,build,allocator,options.worker) + cooked_set(cooked,textures.build(made,build,allocator,options.worker)) case decoded.pixels!=nil && !textures.source_wide(bytes): - cooked.built,cooked.error=textures.build(textures.rgba8(decoded.pixels,decoded.width,decoded.height),build,allocator,options.worker) + cooked_set(cooked,textures.build(textures.rgba8(decoded.pixels,decoded.width,decoded.height),build,allocator,options.worker)) case: - cooked.built,cooked.error=textures.from_bytes(bytes,build,allocator,options.worker) + cooked_set(cooked,textures.from_bytes(bytes,build,allocator,options.worker)) } - if cooked.error==.None {textures.cache_write(folder,key,&types,&cooked.built)} + if cooked.error==.None {textures.cache_write(c,key,&types,&cooked.built)} +} + +// One store per value until odin-lang/Odin#7937 (see ODIN_WORKAROUNDS). +@(private="file") +cooked_set :: proc(cooked:^Cooked_Texture,built:textures.Built,error:textures.Error) { + cooked.built=built + cooked.error=error } // The settings hash of a texture of build built Fast. diff --git a/cook/import/cutout.odin b/cook/import/cutout.odin index 4f44039e2..17803fa7c 100644 --- a/cook/import/cutout.odin +++ b/cook/import/cutout.odin @@ -233,7 +233,12 @@ model_cutouts :: proc(r:^Reader) { for made,at in meshes { if !made.cut {continue} mesh:=&r.meshes[at] - mesh.vertices,mesh.indices,mesh.submeshes,mesh.slots,mesh.outlines=made.vertices,made.indices,made.submeshes,made.slots,made.outlines + // One store per value until odin-lang/Odin#7937 (see ODIN_WORKAROUNDS). + mesh.vertices=made.vertices + mesh.indices=made.indices + mesh.submeshes=made.submeshes + mesh.slots=made.slots + mesh.outlines=made.outlines if len(mesh.skin)>0 {mesh.skin=made.skin} if len(mesh.morphs)>0 {mesh.morphs,mesh.morph_deltas=made.morphs,made.morph_deltas} if len(mesh.vertices)>0 { diff --git a/cook/import/documents.odin b/cook/import/documents.odin new file mode 100644 index 000000000..855f7db72 --- /dev/null +++ b/cook/import/documents.odin @@ -0,0 +1,246 @@ +package importer + +import "core:fmt" +import "core:os" +import "core:path/filepath" +import "core:strings" +import "mirai:asset" +import anim "mirai:std/anim" +import "mirai:std/base" +import "mirai:std/serial" +import "mirai:store" + +// The documents of a package (materials, skeletons, clips, scenes) are edited +// in each project that uses it. Its build keeps them as the source gives them, +// their baseline; a project gets a file of each, written from the baseline the +// first time and followed on every later import: the file keeps what was +// changed in it after the last import. A material keeps the fields changed +// over what its source gave, its overlays and hidden names; a skeleton keeps +// the body parts set by hand and the sockets added to it on bones it still +// has; a clip keeps the root set by hand while its bone is still in its +// skeleton. Any document keeps the owner its file has, so one taken out of the +// package stays out. Any other document, and what of a skeleton or a clip is +// not kept, is written over: one whose file changed since it was last written +// (documents_changed) is left as it is unless the changes are discarded. + +// The baseline of a skeleton the clips followed with it play on: the bytes of its file. +Followed_Skeleton :: struct { + id:base.Asset_Id, + data:[]u8, +} + +// The extension of the files of a kind of a plugin, as its manifest declares it. +Kind_Extension :: struct { + kind:asset.Fourcc, + extension:string, +} + +// A document documents_follow wrote. +Followed :: struct { + id:base.Asset_Id, + kind:asset.Fourcc, + path:string, + warning:string, // what of the file could not be kept; "" when all + hash:u64, // what the file holds as it is left (asset.Package_Written); 0 for a material, followed field by field + kept:bool, // it changed since it was last written and was left as it is +} + +// Writes into folder the documents of the build b of the package p, or +// follows those whose ids registry has where it has them. A new file is +// named as the package names its document, under folder. recorded: what the +// project's pin says its files held when last written; one that changed since +// is left as it is, with a warning, unless changed is Discard. What it wrote, +// in allocator. +// kinds: the extensions of the kinds of the plugins of the project, which a new +// file of one of their documents takes. +documents_follow :: proc(s:^store.Store,p:^asset.Package,b:^asset.Build,folder:string,registry:^asset.Registry,recorded:[]asset.Package_Written,changed:Changed,kinds:[]Kind_Extension={},allocator:=context.allocator) -> (written:[dynamic]Followed,error:Error) { + written=make([dynamic]Followed,allocator) + types:asset.Types + asset.types_init(&types,context.temp_allocator) + defer asset.types_destroy(&types) + skeletons:=make([dynamic]Followed_Skeleton,context.temp_allocator) + for o in b.outputs { + if o.kind!=asset.KIND_SKELETON {continue} + if data,read_error:=asset.object_file(s,o.object,context.temp_allocator);read_error==.None {append(&skeletons,Followed_Skeleton{o.id,data})} + } + taken:=make(map[string]bool,context.temp_allocator) + kept:[]base.Asset_Id + if changed!=.Discard {kept=documents_changed(p,recorded,registry,context.temp_allocator)} + each: for o in b.outputs { + if asset.import_cached(o.kind) {continue} + for id in kept { + if id!=o.id {continue} + file,_:=asset.registry_find(registry,id) + warning:=fmt.tprintf("assets/%s changed since it was last written and was left as it is: import the source of %s with discard_changes to write over it",file.path,p.name) + append(&written,Followed{id=o.id,kind=o.kind,path=strings.clone(asset.registry_file_path(registry,file.path,context.temp_allocator),allocator),warning=strings.clone(warning,allocator),hash=written_of(recorded,id),kept=true}) + continue each + } + baseline,read_error:=asset.object_file(s,o.object,context.temp_allocator) + if read_error!=.None {return written,error_make(.Not_Found,"the store at %s has no object of the document %v of the package %v whole (%v): fetch it from a peer that has it",s.root,o.id,p.id,read_error,allocator=context.temp_allocator)} + target:string + if registry!=nil { + if file,found:=asset.registry_find(registry,o.id);found && file.kind==o.kind {target=asset.registry_file_path(registry,file.path,context.temp_allocator)} + } + if target=="" { + name:="unnamed" + for &a in p.assets { + if a.id==o.id {name=asset.safe_path(a.name)} + } + extension:=kind_extension(o.kind,kinds,registry) + for at:=1;;at+=1 { + candidate:=fmt.tprintf("%s/%s%s",folder,name,extension) if at==1 else fmt.tprintf("%s/%s (%d)%s",folder,name,at,extension) + key:=strings.to_lower(candidate,context.temp_allocator) + if taken[key] || os.exists(candidate) {continue} + taken[key]=true + target=candidate + break + } + if under:=filepath.dir(target);!os.is_dir(under) {_=os.make_directory_all(under)} + } + warning,follow_error:=document_follow(&types,o.kind,baseline,target,skeletons[:],registry) + if follow_error!=.None {return written,error_make(.Write,"%s could not be written: %v",target,follow_error,allocator=context.temp_allocator)} + hash:=written_hash(target) if o.kind!=asset.KIND_MATERIAL else 0 + append(&written,Followed{id=o.id,kind=o.kind,path=strings.clone(target,allocator),warning=strings.clone(warning,allocator),hash=hash}) + } + return written,{} +} + +// The extension of the files of kind: of the engine, of a plugin as kinds +// says, else of a file of that kind registry has; "" for none of them. +@(private="file") +kind_extension :: proc(kind:asset.Fourcc,kinds:[]Kind_Extension,registry:^asset.Registry) -> string { + if info,known:=asset.kind_info(kind);known {return info.extension} + for k in kinds { + if k.kind==kind {return k.extension} + } + if registry!=nil { + for &file in registry.files { + if file.kind!=kind || file.state!=.Asset {continue} + dot:=strings.last_index_byte(file.path,'.') + if dot>strings.last_index_byte(file.path,'/') {return file.path[dot:]} + } + } + return "" +} + +// Writes the document whose baseline is the file in baseline to target, or +// follows the file target has. skeletons: the baselines the clips of an +// animation may play on, else registry has them. decided: the owner of the +// baseline wins over the one of the file (Import_Settings.parts decides it +// again). The warning, in the temp allocator, says what of the file could not +// be kept. +document_follow :: proc(types:^asset.Types,kind:asset.Fourcc,baseline:[]u8,target:string,skeletons:[]Followed_Skeleton,registry:^asset.Registry,decided:=false) -> (warning:string,error:asset.Error) { + given:asset.Asset + asset.asset_open(&given,baseline,context.temp_allocator) or_return + defer asset.asset_close(&given) + previous:asset.Asset + existed:=os.exists(target) && asset.asset_load(&previous,target,context.temp_allocator)==.None + defer if existed {asset.asset_close(&previous)} + owner:=previous.blob.header.owner if existed && !decided else given.blob.header.owner + id:=given.blob.header.id + switch kind { + case asset.KIND_MATERIAL: + view:=asset.material_view(&given,types) or_return + desc:=asset.Material_Desc{id=id,owner=owner,info=view.info^,imported=view.imported,parameters=view.parameters,imported_parameters=view.imported_parameters,source=view.source} + if existed && previous.blob.header.id==id { + if kept,kept_error:=asset.material_view(&previous,types);kept_error==.None { + desc.info=asset.material_follow(types,kept,view.imported^ if view.imported!=nil else view.info^) + desc.parameters=asset.material_parameters_follow(kept,view.imported_parameters,context.temp_allocator) + desc.overlays,desc.hidden=kept.overlays,kept.hidden + } + } + return "",asset.material_save(target,types,desc) + case asset.KIND_SKELETON: + view:=asset.skeleton_view(&given,types) or_return + kept:^asset.Skeleton_View + kept_view:asset.Skeleton_View + if existed { + view_error:asset.Error + kept_view,view_error=asset.skeleton_view(&previous,types) + if view_error==.None {kept=&kept_view} + if view_error==.Version {warning=fmt.tprintf("%s is of a version this build does not read: the body parts, axes and sockets set by hand on it were not kept; set them again",target)} + } + body:asset.Skeleton_Body + body_parts:[len(anim.Body_Part)]asset.Skeleton_Body_Part + count,_:=skeleton_body(kept,view.bones,&body,&body_parts) + sockets:=make([dynamic]asset.Skeleton_Socket,context.temp_allocator) + append(&sockets,..view.sockets) + skeleton_sockets_kept(kept,view.bones,&sockets) + return warning,asset.skeleton_save(target,types,{id=id,owner=owner,bones=view.bones,sockets=sockets[:],body=&body if count>0 else nil,body_parts=body_parts[:count],source=view.source}) + case asset.KIND_ANIMATION: + view:=asset.animation_view(&given,types) or_return + clips,editable:=asset.animation_descs(view,context.temp_allocator) + if !editable {break} + if existed { + kept,view_error:=asset.animation_view(&previous,types) + if view_error==.Version {warning=fmt.tprintf("%s is of a version this build does not read: the root bones set by hand on its clips were not kept; set them again",target)} + for &clip in clips { + for &old in kept.clips { + if base.text_get(old.name[:])!=clip.name || !old.root.translation && !old.root.rotation {continue} + bones,found:=skeleton_bones(types,clip.skeleton,skeletons,registry) + if !found || asset.skeleton_bone_index(bones,old.root.bone)>=0 {clip.root=old.root} + break + } + } + } + return warning,asset.animation_save(target,types,{id=id,owner=owner,clips=clips,source=view.source}) + } + // Any other document is the baseline, with the owner of its file. + data:=asset.asset_bytes_clone(baseline,context.temp_allocator) or_return + if owner!=given.blob.header.owner {asset.header_owner_set(data,owner) or_return} + if _,write_error:=serial.file_write_changed(target,data);write_error!=.None {return "",.IO} + return "",.None +} + +// The bones of the skeleton id: a baseline of skeletons, else the file +// registry has. In the temp allocator. +@(private="file") +skeleton_bones :: proc(types:^asset.Types,id:base.Asset_Id,skeletons:[]Followed_Skeleton,registry:^asset.Registry) -> (bones:[]asset.Skeleton_Bone,found:bool) { + data:[]u8 + for s in skeletons { + if s.id==id {data=s.data} + } + if data==nil && registry!=nil { + if file,known:=asset.registry_find(registry,id);known {data,_=asset.registry_read(registry,file.path,context.temp_allocator)} + } + if data==nil {return} + a:asset.Asset + if asset.asset_open(&a,data,context.temp_allocator)!=.None {return} + view,error:=asset.skeleton_view(&a,types) + if error!=.None {return} + return view.bones,true +} + +// The sockets of the file an import replaces that it does not make again (one a person or an agent +// added, such as where a weapon hangs), on a bone it still has. Those it makes (named by the +// source, bone.*, grips) are made again from the source. +@(private) +skeleton_sockets_kept :: proc(previous:^asset.Skeleton_View,bones:[]asset.Skeleton_Bone,sockets:^[dynamic]asset.Skeleton_Socket) { + if previous==nil {return} + made:=len(sockets) + for &socket in previous.sockets { + name:=base.text_get(socket.name[:]) + if strings.has_prefix(name,"bone.") {continue} + again:=false + for &other in sockets[:made] {if other.name==socket.name {again=true;break}} + if again {continue} + for &bone in bones {if bone.path==socket.bone {append(sockets,socket);break}} + } +} + +// The body of an imported skeleton: found on its bones, keeping the parts a +// person set in the file it replaces when their bones are still there. +// Returns how many parts it wrote, 0 for a skeleton without a body, and the map they come from. +@(private) +skeleton_body :: proc(previous_file:^asset.Skeleton_View,bones:[]asset.Skeleton_Bone,body:^asset.Skeleton_Body,parts:^[len(anim.Body_Part)]asset.Skeleton_Body_Part) -> (count:int,map_:anim.Body_Map) { + previous:=anim.body_map_empty() + if previous_file!=nil { + previous=asset.skeleton_body_map(bones,previous_file.body,previous_file.body_parts) + for part in anim.Body_Part {if part not_in previous.manual {previous.bones[part],previous.confidence[part]=-1,0}} + } + found:bool + map_,found=asset.skeleton_body_detect(bones,previous) + if !found {return 0,map_} + count,_=asset.skeleton_body_rows(bones,map_,body,parts) + return count,map_ +} diff --git a/cook/import/font.odin b/cook/import/font.odin index ab1937e6e..f7d79f765 100644 --- a/cook/import/font.odin +++ b/cook/import/font.odin @@ -2,7 +2,6 @@ package importer import "core:fmt" import "core:mem" -import "core:os" import "core:path/filepath" import "mirai:asset" import "mirai:std/base" @@ -49,20 +48,20 @@ import_font :: proc(path:string,options:Import_Options={},allocator:=context.all report_init(&report,allocator) w:Writer ok:=writer_init(&w,path,"",options,&report,allocator) - defer asset.types_destroy(&w.types) + defer writer_close(&w) if ok { source:=w.source name:=filepath.stem(path) info:=import_info(source,name,0,u64(len(bytes)),base.hash64(bytes),0,import_version(0)) desc:=asset.Font_Desc{file=bytes,source=&info,compression=.Stream} target:string - desc.id,target=root_take(&w,asset.KIND_FONT,name,".font") - existed:=os.exists(target) + output:int + desc.id,target,output=root_take(&w,asset.KIND_FONT,name,".font") if save_error:=asset.font_save(target,&w.types,desc,context.temp_allocator);save_error!=.None { ok=save_failed(&w,save_error,target) - } else { - record(&w,desc.id,asset.KIND_FONT,name,target,source,existed,fmt.tprintf("%v outlines, %d glyphs of the atlas",outlines,glyphs)) - ok=writer_finish(&w) + } else if ok=writer_put(&w,output,target);ok { + record(&w,output,name,fmt.tprintf("%v outlines, %d glyphs of the atlas",outlines,glyphs)) + ok=writer_finish(&w,name,"font") } } if !ok { diff --git a/cook/import/gltf/animation_test.odin b/cook/import/gltf/animation_test.odin index 1f5c25b3d..e096e6bd8 100644 --- a/cook/import/gltf/animation_test.odin +++ b/cook/import/gltf/animation_test.odin @@ -54,7 +54,7 @@ gltf_animation_import_preserves_tangents_and_unambiguous_targets :: proc(t:^test testing.expect_value(t,anim.clip_evaluate(&library,bound[0],1),position) testing.expect_value(t,anim.clip_evaluate(&library,bound[1],1),rotation) // Imported again over its files, it keeps their ids. - again,again_error:=importer.import_write(&model,{root=filepath.dir(folder)}) + again,again_error:=write_again(t,&model,filepath.dir(folder)) defer importer.report_destroy(&again) if !testing.expect_value(t,again_error.code,importer.Error_Code.None) {return} testing.expect_value(t,again.animation_id,report.animation_id) diff --git a/cook/import/gltf/support_test.odin b/cook/import/gltf/support_test.odin index 1805bb6b9..06735051c 100644 --- a/cook/import/gltf/support_test.odin +++ b/cook/import/gltf/support_test.odin @@ -10,6 +10,7 @@ import "mirai:asset" import importer "mirai:cook/import" import "mirai:std/base" import mm "mirai:std/math" +import "mirai:store" FIXTURES :: #directory+"/../../../assets/fixtures/" DEMO :: #directory+"/../../../demo/assets/" @@ -34,7 +35,27 @@ write_model :: proc(t:^testing.T,model:^importer.Imported_Model,name:string) -> data,_:=os.read_entire_file(model.source,context.temp_allocator) _=os.write_entire_file(copied,data) model.source=copied - report,error=importer.import_write(model,{root=root}) + report,error=write_again(t,model,root) + return +} + +// Writes model again as the import of the project at root, into the store of +// that project, with the documents beside its source; each output of the cache +// gets the file of its object as its path. +write_again :: proc(t:^testing.T,model:^importer.Imported_Model,root:string) -> (report:importer.Import_Report,error:importer.Error) { + opened,store_error:=store.open(fmt.tprintf("%s/store",root),context.temp_allocator) + if !testing.expect_value(t,store_error,store.Error.None) {return {},{code=.Write}} + s:=new(store.Store,context.temp_allocator) + s^=opened + report,error=importer.import_write(model,{root=root,store=s,documents=filepath.dir(importer.path_key(model.source))}) + for &written in report.assets { + if written.path!="" || written.object=={} {continue} + digits:[64]u8 + path:=fmt.tprintf("%s-objects/%s",s.root,store.hex(written.object,&digits)) + os.make_directory_all(filepath.dir(path)) + if !os.exists(path) {asset.object_file_write(s,written.object,path)} + written.path=strings.clone(path) + } return } diff --git a/cook/import/identity.odin b/cook/import/identity.odin index 9d892d53e..d899b8033 100644 --- a/cook/import/identity.odin +++ b/cook/import/identity.odin @@ -9,13 +9,16 @@ import "mirai:asset" import "mirai:project" import "mirai:std/base" import "mirai:std/serial" +import "mirai:store" -// Which asset each part of a source is and where its file is. The sidecar of -// the source says the asset: the output of its kind and name (of its kind for -// the root of the import), else a new one. An asset of the cache is a file of -// the imported folder of the root named by its id and the key of what the -// import builds from; a document is where the registry of the root has its -// id, else a new file in the folder of the import. +// Which asset each part of a source is and where it is written. The manifest +// of the package the source was ingested as (package_previous) says the +// asset: the one of its kind and name (of its kind for the root of the +// import), else a new one. Every part is written to a file of the scratch +// folder of the import and put in the store as an object of its build; a +// document is also written, or followed, in the folder of the documents +// (Import_Options.documents): where the registry of the root has its id, else +// a new file there. // What an import writes, before its file and id are known. @(private) @@ -25,164 +28,187 @@ Part :: struct { index:int, // among those of its kind in the source extension:string, only:bool, // the one of its kind, whatever its name, as the root (the clips of a model, named after it) - output:int, // what the sidecar the import writes lists for it (Writer.outputs) + output:int, // what the manifest the import writes lists for it (Writer.outputs) given:bool, // an output this import gave before (Writer.given), written then decided:bool, // whether it is a part of the root was decided by an import before: part says it part:bool, id:base.Asset_Id, owner:base.Asset_Id, - path:string, + path:string, // where it is written, in the scratch folder of the import + target:string, // of a document: its file in the folder of the documents; "": none } -// Reads the sidecar of the source at path, which says which asset each thing -// of the source is, and the folder of the source new documents go to: the -// one its sidecar keeps, else folder. The source is in the assets of the -// project or the plugin at options.root. +// Starts the import of the source at path: the manifest it was ingested as, +// the scratch folder its parts are written in and, with +// options.documents, the folder of the documents (folder under it: where +// those of a first import go, "" in it) and the assets of the root they are +// found among. @(private) writer_source :: proc(w:^Writer,path,folder:string) -> bool { w.path=path w.keep=context.temp_allocator - w.source=source_path(w.options,path) + w.origin=path_key(path) + w.source=filepath.base(w.origin) w.inputs.allocator=context.temp_allocator - names:=project.FOLDER_NAMES - if w.options.root=="" {return writer_fail(w,.Not_Found,"%s: an import names the project or the plugin whose assets the source is in (root)",path)} - if !strings.has_prefix(w.source,names[.Assets]) || !strings.has_prefix(w.source[len(names[.Assets]):],"/") { - return writer_fail(w,.Not_Found,"%s is not in the %s folder of %s: move it there, with its %s when it has one, beside what it makes",path,names[.Assets],w.options.root,SIDECAR_EXTENSION) - } - // One store per value until odin-lang/Odin#7937 (see ODIN_WORKAROUNDS). - sidecar,found,error:=sidecar_load(path,context.temp_allocator) + w.outputs.allocator=context.temp_allocator + w.targets.allocator=context.temp_allocator + w.paths.allocator=context.temp_allocator + w.objects.allocator=context.temp_allocator + w.fixed=false + s:=w.options.store + if s==nil {return writer_fail(w,.Write,"%s: an import puts what it builds in a store: give it one (Import_Options.store)",path)} + previous,manifest,found,error:=package_previous(s,w.options.manifest,w.origin,context.temp_allocator) if error.code!=.None {return writer_fail(w,error.code,"%s",error.message)} - w.sidecar=sidecar + // One store per value until odin-lang/Odin#7937 (see ODIN_WORKAROUNDS). + w.previous=previous + w.previous_manifest=manifest w.stored=found - if !found {w.sidecar.folder=folder} - w.folder=filepath.dir(path) if w.sidecar.folder=="" else fmt.tprintf("%s/%s",filepath.dir(path),w.sidecar.folder) - w.taken=make([]bool,len(w.sidecar.outputs),context.temp_allocator) - w.outputs=make([dynamic]Output,context.temp_allocator) - w.paths=make([dynamic]string,context.temp_allocator) - w.imported=project.imported_path(w.options.root,context.temp_allocator) - w.registry=w.options.registry - if w.registry==nil { - w.registry=new(asset.Registry,context.temp_allocator) - asset.registry_init(w.registry,project.folder_path(w.options.root,.Assets,context.temp_allocator),"",context.temp_allocator,w.imported) - asset.registry_scan(w.registry) + w.package_id=previous.id if found else writer_id(w) + w.taken=make([]bool,len(previous.assets),context.temp_allocator) + if documents:=w.options.documents;documents!="" { + w.folder=documents if folder=="" else fmt.tprintf("%s/%s",documents,folder) + w.registry=w.options.registry + if w.registry==nil && w.options.root!="" { + w.registry=new(asset.Registry,context.temp_allocator) + asset.registry_init(w.registry,project.folder_path(w.options.root,.Assets,context.temp_allocator),"",context.temp_allocator) + asset.registry_scan(w.registry) + } } + scratch,scratch_error:=os.make_directory_temp("","mirai-import-*",w.keep) + if scratch_error!=nil {return writer_fail(w,.Write,"%s: the folder its parts are written in before the store takes them could not be made: %v",path,scratch_error)} + w.scratch=scratch return true } -// The file of an output of the cache: in the imported folder, by its id and -// the key of what the import builds from (import_key). +// The file a part is written to before the store takes it. @(private) -imported_file :: proc(w:^Writer,id:base.Asset_Id,extension:string) -> string { - if w.key==0 { - slice.sort(w.inputs[:]) - w.key=import_key(w.path,{settings=w.report.settings,inputs=w.inputs},w.version,w.options.importer_code,w.options.fast_textures) - } - return asset.imported_file(w.imported,id,w.key,extension,context.temp_allocator) +scratch_path :: proc(w:^Writer,id:base.Asset_Id,extension:string) -> string { + text:[base.ASSET_ID_TEXT]u8 + return fmt.tprintf("%s/%s%s",w.scratch,base.asset_id_format(id,text[:]),extension) } -// Notes a file the import reads besides its source, as the sidecar lists it. -// False for a new one once the key of the import is fixed: the files of the -// cache are named without it. +// Notes a file the import reads besides its source, as the manifest lists it. +// False for a new one once what the import is made from is fixed +// (writer_identity). @(private) writer_input :: proc(w:^Writer,file:string) -> bool { - relative,error:=filepath.rel(filepath.dir(path_key(w.path)),path_key(file),context.temp_allocator) + relative,error:=filepath.rel(filepath.dir(w.origin),path_key(file),context.temp_allocator) if error!=.None {return true} slashed,_:=strings.replace_all(relative,"\\","/",context.temp_allocator) if slice.contains(w.inputs[:],slashed) {return true} - if w.key!=0 {return false} + if w.fixed {return false} append(&w.inputs,strings.clone(slashed,w.keep)) return true } -// Gives each part its asset and its file: the output of the sidecar of its -// kind and name (the one of its index first), or of its kind for the only one -// of its kind, else a new id. What the import gives is listed in w.outputs, the sidecar it -// writes when it ends. A part of the kind and name of one of the first -// w.given outputs, or of the kind, name and index of a later one, is that one -// (given): an output is listed once however many times it is asked for. +// Gives each part its asset and its files: the asset of the previous manifest +// of its kind and name (the one of its index first), or of its kind for the +// only one of its kind, else a new id. What the import gives is listed in +// w.outputs, the assets of the manifest it writes. A part of the kind and name +// of one of the first w.given outputs, or of the kind, name and index of a +// later one, is that one (given): an output is listed once however many times +// it is asked for. Planning a part of the cache fixes what the import is made +// from. @(private) parts_plan :: proc(w:^Writer,parts:[]Part) { each: for &p in parts { if !p.only { for output,at in w.outputs { if output.kind!=p.kind || output.name!=p.name || at>=w.given && output.index!=p.index {continue} - p.id,p.output,p.path,p.given=output.id,at,w.paths[at],true + p.id,p.output,p.path,p.target,p.given=output.id,at,w.paths[at],w.targets[at],true continue each } } entry:=-1 - for output,at in w.sidecar.outputs { - if w.taken[at] || output.kind!=p.kind || !p.only && output.name!=p.name {continue} + for a,at in w.previous.assets { + if w.taken[at] || a.kind!=p.kind || !p.only && a.name!=p.name {continue} entry=at - if p.only || output.index==p.index {break} + if p.only || a.index==p.index {break} } p.id=writer_id(w) written:u64 if entry>=0 { w.taken[entry]=true - p.id,written=w.sidecar.outputs[entry].id,w.sidecar.outputs[entry].written + a:=w.previous.assets[entry] + p.id,p.decided=a.id,true + p.part=a.part if asset.import_cached(p.kind) else a.owner!={} } - switch { - case asset.import_cached(p.kind): - p.decided=entry>=0 - p.part=p.decided && w.sidecar.outputs[entry].part - p.path=imported_file(w,p.id,p.extension) - case: - if file,found:=asset.registry_find(w.registry,p.id);found && file.kind==p.kind && file.imported==0 { - p.path=asset.registry_file_path(w.registry,file.path,context.temp_allocator) - p.decided,p.part=true,file.owner!={} + p.path=scratch_path(w,p.id,p.extension) + p.target="" + if asset.import_cached(p.kind) { + writer_identity(w) + } else if w.folder!="" { + if w.registry!=nil { + if file,found:=asset.registry_find(w.registry,p.id);found && file.kind==p.kind {p.target=asset.registry_file_path(w.registry,file.path,context.temp_allocator)} + } + if _,keeping:=w.keeping[p.id];keeping { + // A document the import keeps as it is: its object alone is written. + p.target="" } else { - p.path=fresh_path(w,fmt.tprintf("%s%s",asset.safe_path(p.name),p.extension)) + if p.target=="" {p.target=fresh_path(w,fmt.tprintf("%s%s",asset.safe_path(p.name),p.extension))} + if folder:=filepath.dir(p.target);!os.is_dir(folder) {_=os.make_directory_all(folder)} } - // What is written of a document the import keeps goes nowhere. - if kept,keeping:=w.keeping[p.id];keeping {p.path=kept} } - if folder:=filepath.dir(p.path);!os.is_dir(folder) {_=os.make_directory_all(folder)} p.output=len(w.outputs) - append(&w.outputs,Output{kind=p.kind,name=strings.clone(p.name,w.keep),index=p.index,id=p.id,written=written}) + append(&w.outputs,asset.Package_Asset{kind=p.kind,name=strings.clone(p.name,w.keep),index=p.index,id=p.id}) append(&w.paths,strings.clone(p.path,w.keep)) + append(&w.targets,strings.clone(p.target,w.keep)) + append(&w.objects,store.Hash{}) } } -// The id and the file of the root of an import: the output of kind its -// sidecar lists, else a new one named name. +// The id, the file and the output of the root of an import: the asset of +// kind the previous manifest lists, else a new one named name. @(private) -root_take :: proc(w:^Writer,kind:asset.Fourcc,name,extension:string) -> (id:base.Asset_Id,path:string) { +root_take :: proc(w:^Writer,kind:asset.Fourcc,name,extension:string) -> (id:base.Asset_Id,path:string,output:int) { parts:=[1]Part{{kind=kind,name=name,extension=extension,only=true}} parts_plan(w,parts[:]) - return parts[0].id,parts[0].path + return parts[0].id,parts[0].path,parts[0].output } -// Records which parts of the cache name the root of the import as their -// owner, once that is decided: what the sidecar keeps for the next import. +// Takes the file of an output written at file as an object of the build, with the +// owner and the dependencies its header gives, and its document written or +// followed at its target (document_follow). False, with the writer failed, +// when the store does not take it or the target cannot be written. @(private) -parts_record :: proc(w:^Writer,parts:[]Part) { - for &p in parts { - if asset.import_cached(p.kind) && !p.given {w.outputs[p.output].part=p.owner!={}} +writer_put :: proc(w:^Writer,output:int,file:string) -> bool { + s:=w.options.store + object,error:=asset.object_put_file(s,file) + if error!=.None {return writer_fail(w,.Write,"%s could not be put in the store at %s: %v",file,s.root,error)} + return writer_take(w,output,object) +} + +// What the project's pin records its file of the document id held when the +// import or the refresh last wrote it (asset.Package_Written); 0: nothing. +written_of :: proc(written:[]asset.Package_Written,id:base.Asset_Id) -> u64 { + for entry in written { + if entry.id==id {return entry.hash} } + return 0 } -// The documents among outputs that an import writes over and that changed -// since it last wrote them: the file of each in registry (the assets of their -// root) holds other content than the sidecar records (Output.written). A +// The documents of the package p that the import or the refresh writes over +// and that changed since it last wrote them: the file of each in registry +// (the assets of the project) holds other content than written records. A // change of owner alone (asset.detach) is no change. Their ids, in allocator. -outputs_changed :: proc(outputs:[]Output,registry:^asset.Registry,allocator:=context.allocator) -> []base.Asset_Id { +documents_changed :: proc(p:^asset.Package,written:[]asset.Package_Written,registry:^asset.Registry,allocator:=context.allocator) -> []base.Asset_Id { changed:=make([dynamic]base.Asset_Id,allocator) - for output in outputs { - if output.written==0 || asset.import_cached(output.kind) {continue} - if file,found:=asset.registry_find(registry,output.id);found && file.kind==output.kind && file.imported==0 && file.content_hash!=output.written {append(&changed,output.id)} + if registry==nil {return changed[:]} + for &a in p.assets { + recorded:=written_of(written,a.id) + if recorded==0 || asset.import_cached(a.kind) {continue} + if file,found:=asset.registry_find(registry,a.id);found && file.kind==a.kind && file.content_hash!=recorded {append(&changed,a.id)} } return changed[:] } // Applies Import_Options.changed to the documents of the import that changed -// since it last wrote them: Refuse fails naming them, Keep sends what would be -// written of each to a file of the imported folder that goes when the import -// ends, and warns of it. +// since it last wrote them: Refuse fails naming them, Keep leaves each as it +// is (Writer.keeping), writing its object alone, and warns of it. @(private) writer_changed :: proc(w:^Writer) -> bool { - if w.options.changed==.Discard {return true} - changed:=outputs_changed(w.sidecar.outputs[:],w.registry,context.temp_allocator) + if w.folder=="" || w.options.changed==.Discard {return true} + changed:=documents_changed(&w.previous,w.options.written,w.registry,context.temp_allocator) if len(changed)==0 {return true} names:=project.FOLDER_NAMES paths:=make([]string,len(changed),context.temp_allocator) @@ -196,12 +222,9 @@ writer_changed :: proc(w:^Writer) -> bool { return writer_fail(w,.Changed,"%s changed since %s was last imported, and importing it again writes over %s: import with discard_changes to write over %s, or leave it as it is to keep %s", strings.join(paths,", ",context.temp_allocator),w.source,one?"it":"them",one?"that change":"those changes",one?"it":"them") case .Keep: - w.keeping=make(map[base.Asset_Id]string,context.temp_allocator) - buffer:[base.ASSET_ID_TEXT]u8 + w.keeping=make(map[base.Asset_Id]u64,context.temp_allocator) for id,at in changed { - file,_:=asset.registry_find(w.registry,id) - info,_:=asset.kind_info(file.kind) - w.keeping[id]=fmt.tprintf("%s/%s.kept%s",w.imported,base.asset_id_format(id,buffer[:]),info.extension) + w.keeping[id]=written_of(w.options.written,id) writer_warn(w,"%s changed since its import last wrote it and was left as it is: import %s with discard_changes to write over it",paths[at],w.source) } case .Discard: @@ -211,7 +234,6 @@ writer_changed :: proc(w:^Writer) -> bool { // What the file of a document holds: the content_hash of its header; 0 when // it is not an asset. -@(private) written_hash :: proc(file:string) -> u64 { head:[asset.HEADER_SIZE]u8 bytes,_,read_error:=serial.file_read_head(file,head[:]) @@ -220,68 +242,89 @@ written_hash :: proc(file:string) -> u64 { return header.content_hash if header_error==.None else 0 } -// Records in the sidecar of the source at source what the document id its -// import wrote at file holds (Output.written): for one written once the import -// ended, the prefab of a model. -output_recorded :: proc(source:string,id:base.Asset_Id,file:string) -> bool { - sidecar,found,error:=sidecar_load(source,context.temp_allocator) - if !found || error.code!=.None {return false} - for &output in sidecar.outputs { - if output.id!=id {continue} - output.written=written_hash(file) - return sidecar_save(source,sidecar) - } - return false +// Records in the report what the project's file of the document id holds as +// the import leaves it (Import_Report.written): hash, else, for one it keeps +// as it is, what the pin recorded. A material records nothing: it is followed +// field by field. +@(private) +writer_written :: proc(w:^Writer,id:base.Asset_Id,kind:asset.Fourcc,hash:u64) { + if kind==asset.KIND_MATERIAL || asset.import_cached(kind) {return} + hash:=hash + if recorded,keeping:=w.keeping[id];keeping {hash=recorded} + if hash!=0 {append(&w.report.written,asset.Package_Written{id,hash})} } -// Ends the outputs of an import that wrote its assets: what its sidecar -// listed and the source no longer gives leaves it (a document stays, a plain -// asset from then on), the files of the imported folder its outputs had under -// another key go, and the sidecar is written with the settings the source was -// imported with. An import of documents alone leaves its stamp (STAMP_EXTENSION). +// Takes the object hash as the output of the build, from the store: its owner and +// dependencies, and its document at its target. @(private) -writer_finish :: proc(w:^Writer) -> bool { - current:=make(map[base.Asset_Id]string,context.temp_allocator) - for output in w.outputs { - info,known:=asset.kind_info(output.kind) - if known && asset.import_cached(output.kind) {current[output.id]=filepath.base(imported_file(w,output.id,info.extension))} +writer_take :: proc(w:^Writer,output:int,hash:store.Hash) -> bool { + s:=w.options.store + dependencies:=make([dynamic]base.Asset_Id,context.temp_allocator) + header,peek_error:=asset.object_header(s,hash,&dependencies) + digits:[64]u8 + if peek_error!=.None {return writer_fail(w,.Write,"the object %s of the store at %s, taken for %s, does not read: %v",store.hex(hash,&digits),s.root,w.outputs[output].name,peek_error)} + o:=&w.outputs[output] + w.objects[output]=hash + baseline:[]u8 + if w.targets[output]!="" || o.kind==asset.KIND_SKELETON { + read_error:asset.Error + baseline,read_error=asset.object_file(s,hash,w.keep) + if read_error!=.None {return writer_fail(w,.Write,"the object %s of the store at %s, taken for %s, does not read: %v",store.hex(hash,&digits),s.root,o.name,read_error)} } - stamp:="" - if len(current)==0 && len(w.outputs)>0 { - stamp=imported_file(w,w.outputs[0].id,STAMP_EXTENSION) - current[w.outputs[0].id]=filepath.base(stamp) + o.owner=header.owner + o.part=asset.import_cached(o.kind) && header.owner!={} + o.dependencies=make([dynamic]base.Asset_Id,w.keep) + for d in dependencies { + if d!={} && d!=o.id && !slice.contains(o.dependencies[:],d) {append(&o.dependencies,d)} } - for output,at in w.sidecar.outputs { + if target:=w.targets[output];target!="" { + existed:=os.exists(target) + warning,follow_error:=document_follow(&w.types,o.kind,baseline,target,w.followed[:],w.registry,decided=w.report.settings.parts!=.Auto) + if follow_error!=.None {return save_failed(w,follow_error,target)} + if warning!="" {writer_warn(w,"%s",warning)} + if !existed {append(&w.created,strings.clone(target,w.keep))} + writer_written(w,o.id,o.kind,written_hash(target)) + } else if _,keeping:=w.keeping[o.id];keeping { + writer_written(w,o.id,o.kind,0) + } + if o.kind==asset.KIND_SKELETON {append(&w.followed,Followed_Skeleton{o.id,baseline})} + return true +} + +// Ends the outputs of an import that wrote its assets: what the previous +// manifest listed and the source no longer gives leaves the package (a +// document stays, a plain asset from then on), and the manifest and the build +// are put in the store and named (writer_publish). +@(private) +writer_finish :: proc(w:^Writer,name,importer:string) -> bool { + for a,at in w.previous.assets { if w.taken[at] {continue} - if asset.import_cached(output.kind) { - current[output.id]="" - writer_warn(w,"%s %s is no longer in the source: its asset is gone, and what named it names nothing",kind_name(output.kind),output.name) - } else if file,found:=asset.registry_find(w.registry,output.id);found { - writer_warn(w,"%s is no longer in the source: it was left as it is",file.path) + if asset.import_cached(a.kind) { + writer_warn(w,"%s %s is no longer in the source: its asset is gone, and what named it names nothing",kind_name(a.kind),a.name) + } else if w.registry!=nil { + if file,found:=asset.registry_find(w.registry,a.id);found {writer_warn(w,"%s is no longer in the source: it was left as it is",file.path)} } } - entries,_:=os.read_all_directory_by_path(w.imported,context.temp_allocator) - for entry in entries { - id,_,named:=asset.imported_name(entry.name) - if name,listed:=current[id];named && listed && name!=entry.name {os.remove(entry.fullpath)} - } - for _,kept in w.keeping {os.remove(kept)} - slice.sort(w.inputs[:]) - if !sidecar_save(w.path,{settings=w.report.settings,folder=w.sidecar.folder,inputs=w.inputs,outputs=w.outputs}) { - return writer_fail(w,.Write,"%s%s, which keeps the ids of the import, could not be written",w.path,SIDECAR_EXTENSION) - } - w.report.saved=strings.concatenate({w.path,SIDECAR_EXTENSION},w.allocator) - // Without it the next sync imports the source again: nothing is lost. - if stamp!="" { - _=os.make_directory_all(w.imported) - _=os.write_entire_file(stamp,nil) - } - return true + return writer_publish(w,name,importer) +} + +// Removes the documents the import created and the scratch folder; the objects it +// put stay, named by nothing. +@(private) +writer_rollback :: proc(w:^Writer) { + for path in w.created {os.remove(path)} +} + +// Lets the writer go: its types and its scratch folder. +@(private) +writer_close :: proc(w:^Writer) { + if w.scratch!="" {_=os.remove_all(w.scratch)} + asset.types_destroy(&w.types) } -// The first of file (a path under the folder of the import) there, then file -// with " (2)" and on before its extension, that no part of this write took -// and no file has. +// The first of file (a path under the folder of the documents) there, then +// file with " (2)" and on before its extension, that no part of this write +// took and no file has. @(private) fresh_path :: proc(w:^Writer,file:string) -> string { slash:=strings.last_index_byte(file,'/') @@ -301,8 +344,8 @@ fresh_path :: proc(w:^Writer,file:string) -> string { @(private) writer_id :: proc(w:^Writer) -> base.Asset_Id {return base.asset_id_random(&w.rng)} -// An absolute path with forward slashes: how files are compared. -@(private) +// An absolute path with forward slashes: how files are compared, and the +// origin of a package (asset.Package_Origin.path). path_key :: proc(path:string) -> string { absolute,error:=filepath.abs(path,context.temp_allocator) if error!=nil {absolute=path} @@ -311,6 +354,40 @@ path_key :: proc(path:string) -> string { return strings.clone(slashed,context.temp_allocator) } +// The project or the plugin whose assets folder has the file at path: the +// folder that holds the nearest assets folder above it. False for a file in +// no assets folder. +source_root :: proc(path:string,allocator:=context.allocator) -> (root:string,ok:bool) { + names:=project.FOLDER_NAMES + folder:=project.path_absolute(path,context.temp_allocator) + for { + parent:=filepath.dir(folder) + if parent==folder {return} + if filepath.base(parent)==names[.Assets] {return strings.clone(filepath.dir(parent),allocator),true} + folder=parent + } +} + +// Where a manifest says its source is (asset.Package_Origin.path), for the +// source at origin (path_key): relative to the root whose assets hold it +// (source_root), so every copy of that root names it alike; else origin. In +// the temp allocator. +origin_of :: proc(origin:string) -> string { + root,inside:=source_root(origin,context.temp_allocator) + under:=fmt.tprintf("%s/",path_key(root)) + if !inside || !strings.has_prefix(origin,under) {return origin} + return origin[len(under):] +} + +// The source the origin of a manifest names, as path_key gives it: a relative +// one under root, the project or plugin that pins its package. "" for a +// relative one without a root. In the temp allocator. +origin_path :: proc(origin,root:string) -> string { + if origin=="" || filepath.is_abs(origin) || strings.has_prefix(origin,"/") {return origin} + if root=="" {return ""} + return path_key(fmt.tprintf("%s/%s",root,origin)) +} + @(private) kind_name :: proc(kind:asset.Fourcc) -> string { // The scene an import writes has one root: the prefab of its model. diff --git a/cook/import/import.odin b/cook/import/import.odin index 630bf3858..1e22d3692 100644 --- a/cook/import/import.odin +++ b/cook/import/import.odin @@ -20,7 +20,7 @@ import_model :: proc(path:string,formats:[]Format,options:Import_Options={},allo caller:=context.temp_allocator settings,stored:=options.settings,false if !options.ignore_stored { - kept,found,settings_error:=settings_load(path) + kept,found,settings_error:=settings_load(options,path) if settings_error.code!=.None {return {},settings_error} if found {settings,stored=kept,true} } diff --git a/cook/import/interface_test.odin b/cook/import/interface_test.odin index 0c10792b3..be92d800e 100644 --- a/cook/import/interface_test.odin +++ b/cook/import/interface_test.odin @@ -8,52 +8,66 @@ import "mirai:asset" import cimage "mirai:cook/image" import "mirai:std/base" import "mirai:std/serial" +import "mirai:store" @(private="file") CFF_OTF :: #load("../../assets/fixtures/cff.otf") @(private="file") INTER :: #load("../../assets/fonts/Inter-Regular.ttf") // A folder of a test as the root of a project: its assets folder, where the -// sources of an import are. "" when there is none. -test_assets :: proc(t:^testing.T,name:string) -> (root,assets:string) { +// sources of an import are, and a store of its own (Import_Options.store), +// in the temp allocator. "" when there is none. +test_assets :: proc(t:^testing.T,name:string) -> (root,assets:string,s:^store.Store) { root=base.test_folder(t,name) if root=="" {return} assets=fmt.tprintf("%s/assets",root) os.make_directory_all(assets) + s=test_store(t,fmt.tprintf("%s/store",root)) + if s==nil {return "","",nil} return } +// A store at folder, in the temp allocator; nil when it does not open. +test_store :: proc(t:^testing.T,folder:string) -> ^store.Store { + s:=new(store.Store,context.temp_allocator) + opened,error:=store.open(folder,context.temp_allocator) + if !testing.expect_value(t,error,store.Error.None) {return nil} + s^=opened + return s +} + @(test) fonts_import_as_the_file_they_are_and_collections_are_refused :: proc(t:^testing.T) { - root,folder:=test_assets(t,"fonts") + _,folder,s:=test_assets(t,"fonts") if folder=="" {return} for file in ([2]string{"face.otf","inter.ttf"}) { source:=fmt.tprintf("%s/%s",folder,file) _=os.write_entire_file(source,CFF_OTF if file=="face.otf" else INTER) - report,error:=import_font(source,{root=root}) + report,error:=import_font(source,{store=s}) if !testing.expectf(t,error.code==.None,"%s imports: %s",file,error.message) {continue} defer report_destroy(&report) - testing.expect(t,len(report.assets)==1 && report.assets[0].kind==asset.KIND_FONT && strings.has_suffix(report.assets[0].path,".font")) + testing.expect(t,len(report.assets)==1 && report.assets[0].kind==asset.KIND_FONT && report.assets[0].object!={} && report.assets[0].path=="") types:asset.Types;asset.types_init(&types);defer asset.types_destroy(&types) a:asset.Asset - if !testing.expect_value(t,asset.asset_load(&a,report.assets[0].path),asset.Error.None) {continue} + if !testing.expect_value(t,written_open(s,report.assets[0],&a),asset.Error.None) {continue} defer asset.asset_close(&a) view,view_error:=asset.font_view(&a,&types) testing.expect_value(t,view_error,asset.Error.None) testing.expect(t,view.info.outlines==(.CFF if file=="face.otf" else .TrueType) && len(view.file)==len(CFF_OTF if file=="face.otf" else INTER)) // Again: the same asset, with the same id. - again,again_error:=import_font(source,{root=root}) - testing.expect(t,again_error.code==.None && len(again.assets)==1 && again.assets[0].id==report.assets[0].id && again.assets[0].path==report.assets[0].path) + again,again_error:=import_font(source,{store=s}) + testing.expect(t,again_error.code==.None && len(again.assets)==1 && again.assets[0].id==report.assets[0].id && again.assets[0].object==report.assets[0].object) report_destroy(&again) } collection:=fmt.tprintf("%s/family.ttc",folder) _=os.write_entire_file(collection,transmute([]u8)string("ttcf\x00\x02\x00\x00\x00\x00\x00\x02............")) - _,error:=import_font(collection,{root=root}) + _,error:=import_font(collection,{store=s}) testing.expect(t,error.code==.Unsupported && strings.contains(error.message,"collection") && strings.contains(error.message,"fonttools ttLib -y 0"),error.message) broken:=fmt.tprintf("%s/broken.otf",folder) _=os.write_entire_file(broken,transmute([]u8)string("OTTO\x00\x09\x00\x80\x00\x03\x00\x10 not tables at all")) - _,error=import_font(broken,{root=root}) + _,error=import_font(broken,{store=s}) testing.expect(t,error.code==.Malformed,error.message) - testing.expect(t,!os.exists(fmt.tprintf("%s/broken.font",folder)),"A refused font writes nothing") + refs,_:=store.ref_list(s,asset.REF_PACKAGES,context.temp_allocator) + testing.expect(t,len(refs)==2,"A refused font records nothing") } @(test) @@ -155,7 +169,7 @@ prompt_images_pack_into_power_of_two_atlases :: proc(t:^testing.T) { @(test) prompts_import_packs_the_images_of_a_manifest :: proc(t:^testing.T) { - root,folder:=test_assets(t,"prompts") + root,folder,s:=test_assets(t,"prompts") if folder=="" {return} os.make_directory_all(fmt.tprintf("%s/images",folder)) paint :: proc(path:string,width,height:int,color:[4]u8) { @@ -172,12 +186,12 @@ prompts_import_packs_the_images_of_a_manifest :: proc(t:^testing.T) { {"family": "PlayStation", "input": "South", "image": "images/a.png"}, {"family": "Keyboard", "input": "Key", "key": "Space", "image": "images/space.png"} ]}`)) - report,error:=import_prompts(manifest,{root=root}) + report,error:=import_prompts(manifest,{store=s,root=root,documents=folder}) if !testing.expectf(t,error.code==.None,"%s",error.message) {return} defer report_destroy(&report) if !testing.expect_value(t,len(report.assets),2) {return} atlas,pack:=report.assets[0],report.assets[1] - testing.expect(t,atlas.kind==asset.KIND_TEXTURE && pack.kind==asset.KIND_PROMPTS && pack.path==fmt.tprintf("%s/pad.prompts",folder) && strings.contains(atlas.path,"/.mirai/imported/")) + testing.expect(t,atlas.kind==asset.KIND_TEXTURE && pack.kind==asset.KIND_PROMPTS && pack.path==fmt.tprintf("%s/pad.prompts",folder) && atlas.path=="" && atlas.object!={}) types:asset.Types;asset.types_init(&types);defer asset.types_destroy(&types) a,b:asset.Asset if !testing.expect_value(t,asset.asset_load(&a,pack.path),asset.Error.None) {return} @@ -187,7 +201,7 @@ prompts_import_packs_the_images_of_a_manifest :: proc(t:^testing.T) { testing.expect(t,len(view.images)==3 && view.images[0].texture==atlas.id && view.images[0].uv==view.images[1].uv,"One image shown by two entries is packed once") space:=view.images[2] testing.expect(t,serial.text_field(space.key[:])=="Space" && space.aspect==2 && space.family==0 && space.input==asset.PROMPT_INPUT_KEY) - if !testing.expect_value(t,asset.asset_load(&b,atlas.path),asset.Error.None) {return} + if !testing.expect_value(t,written_open(s,atlas,&b),asset.Error.None) {return} defer asset.asset_close(&b) texture,texture_error:=asset.texture_view(&b,&types) if !testing.expect_value(t,texture_error,asset.Error.None) {return} @@ -199,7 +213,7 @@ prompts_import_packs_the_images_of_a_manifest :: proc(t:^testing.T) { testing.expect_value(t,texel(level,width,x,y),[4]u8{255,255,255,128}) testing.expect_value(t,texel(level,width,x-PROMPTS_PADDING,y-PROMPTS_PADDING),[4]u8{255,255,255,128}) // the padding repeats the edge, alpha straight // Again: the same files and ids. - again,again_error:=import_prompts(manifest,{root=root}) + again,again_error:=import_prompts(manifest,{store=s,root=root,documents=folder}) testing.expect(t,again_error.code==.None && len(again.assets)==2 && again.assets[0].id==atlas.id && again.assets[1].id==pack.id && again.assets[0].kept) report_destroy(&again) @@ -215,7 +229,7 @@ prompts_import_packs_the_images_of_a_manifest :: proc(t:^testing.T) { }) { bad:=fmt.tprintf("%s/bad.prompts.json",folder) _=os.write_entire_file(bad,transmute([]u8)c.json) - _,refused:=import_prompts(bad,{root=root}) + _,refused:=import_prompts(bad,{store=s,root=root,documents=folder}) testing.expectf(t,refused.code!=.None && strings.contains(refused.message,c.says),"%s: %s",c.json,refused.message) } } diff --git a/cook/import/model.odin b/cook/import/model.odin index dd0690ec5..849de6f41 100644 --- a/cook/import/model.odin +++ b/cook/import/model.odin @@ -4,10 +4,12 @@ import "core:fmt" import "core:mem" import "mirai:asset" import "mirai:core" +import "mirai:cook/cache" import "mirai:cook/textures" import "mirai:std/base" import "mirai:std/jobs" import mm "mirai:std/math" +import mstore "mirai:store" // What an importer reads from a source file, as plain data that depends on no // engine component: the geometry as .mesh is built of it, the materials with @@ -48,30 +50,42 @@ Error :: struct { } Import_Options :: struct { - // Replaced by the settings the sidecar of the source keeps when it has - // one, unless ignore_stored. + // Replaced by the settings the package of the source keeps when it was + // ingested before, unless ignore_stored. using settings:Import_Settings, ignore_stored:bool, - // The project or the plugin whose assets folder the source is in: sources - // are named relative to it in Import_Info, its documents are found among - // its assets, and what it builds of the cache kinds goes to its imported - // folder (sidecar.odin). + // The machine store: where the import puts the blobs of its build and the + // records of its package (package.odin), and whose caches keep textures + // and meshes built before by what they are built from (textures.cache_read, + // mesh_cook). + store:^mstore.Store, + cache_source:cache.Source, // asked on a miss of those caches before building + // The manifest of the package the source was ingested as, whose ids and + // settings the import keeps; zero: the one the store names for the path + // of the source (package_previous), else a new package. + manifest:mstore.Hash, + // The folder of the assets of a project the documents of the import are + // written or followed in (documents.odin), and the prefab of a model; + // "": they are blobs of the build alone. + documents:string, + // The project or the plugin the import is for: the codecs of its + // assets, and the assets its documents are found among by id. root:string, registry:^asset.Registry, // the scanned assets of root, which the import does not change; nil: it scans them world:^core.World_Info, // the types of the game of root, which the prefab of a model is written in rebuild:bool, // writes every asset, also one whose file already is the one the import would write - changed:Changed, // what becomes of a document it writes over that was changed since it last wrote it - texture_cache:string, // folder of textures built before, kept by image and settings (textures.cache_read); "": none + changed:Changed, // what becomes of a document it writes over in documents that was changed since it last wrote it + written:[]asset.Package_Written, // what the files of its documents in documents held when it last wrote them (the project's pin) // A texture is built Fast unless the texture cache keeps it as Final // (textures.Encode_Speed): the imports of the editor, which a cook // encodes again as Final, and mirai import. False: Final, what a build // that cooks and the package of the engine build. fast_textures:bool, - mesh_cache:string, // folder of meshes built before, kept by geometry and settings (mesh_cook); "": none // The identity of the code of the plugin that imports the source whole // (sources.code_identity), 0 for the engine's: part of what its outputs - // are built from (import_key). + // are made from (identity_of). importer_code:u64, + hashes:^Source_Hashes, // the files of the source known by size and time (identity_of); nil: every one is read material_index:int, // the index of the first material among those of the source: imports that write its materials apart give each its own // Where the import cooks: the worker that calls it (the owner of its // job system, or the job that imports), which runs its builds in @@ -81,10 +95,10 @@ Import_Options :: struct { } // What an import does with the documents it writes over that were changed -// since it last wrote them (outputs_changed). Refuse: it writes nothing and +// since it last wrote them (documents_changed). Refuse: it writes nothing and // fails with Error_Code.Changed, naming them, so whoever imports decides. // Discard: it writes over them. Keep: it leaves them as they are, imports the -// rest and warns of each: what an import nobody asked for (sync) does. +// rest and warns of each: what an import nobody asked for (a refresh) does. Changed :: enum u8 {Refuse,Discard,Keep} // A zone of the import that base.profile_end ends. @@ -217,7 +231,7 @@ Imported_Model :: struct { warnings:[]string, // what the importer left out or could not honor programs:[]string, // the Surface programs reading it wrote to the shaders of the project, their files files:[]string, // the files reading it read besides the source and its images (buffers, material libraries) - settings:Import_Settings, // those of the sidecar of the source when stored is set, else those of the options it was read with + settings:Import_Settings, // those of the package of the source when stored is set, else those of the options it was read with stored:bool, arena:^mem.Dynamic_Arena, allocator:mem.Allocator, diff --git a/cook/import/outputs.odin b/cook/import/outputs.odin index 65f233e61..7d5a0287a 100644 --- a/cook/import/outputs.odin +++ b/cook/import/outputs.odin @@ -2,75 +2,83 @@ package importer import "core:fmt" import "core:mem" +import "core:path/filepath" import "core:strings" import "mirai:asset" import "mirai:std/base" // The assets of an import that is not that of one model: the scene of a // level, the source a plugin imports whole with the models it reads -// (output_model). They are the assets the sidecar of their source lists as -// those of a model are (identity.odin): the root by output_root, each other -// one by kind and name (then index). A new one gets a random id and a file of -// its own, a name with '/' in that folder under the one of the import, and -// outputs_end writes the sidecar. Outputs does not move while it is open: its +// (output_model). They are the assets of the package of their source as those +// of a model are (identity.odin): the root by output_root, each other one by +// kind and name (then index). Each is written to the file its part gives and +// taken by output_written: a blob of the build, and a document also written +// or followed in the folder of the documents. outputs_end puts the manifest +// and the build in the store. Outputs does not move while it is open: its // search keeps memory of its own. One thread at a time. Outputs :: struct { w:Writer, report:Import_Report, source:string, // the source as imports name it + importer:string, size,hash:u64, version:u32, - current:Maybe(bool), // every output of the cache the sidecar lists is current: models cook nothing ahead + current:Maybe(bool), // the store has every output of the cache of the package, built from what it is made from now: models cook nothing ahead scratch:mem.Dynamic_Arena, // the temp memory of the search } // Starts the outputs of an import of the file at source, whose bytes have // size and hash, read by a writer of version (import_version), with the -// settings its sidecar keeps unless options.ignore_stored. folder: under -// the folder of the source, where the new documents of a first import go -// ("" beside it), unless output_part names another. -outputs_begin :: proc(o:^Outputs,source,folder:string,size,hash:u64,version:u32,options:Import_Options,allocator:=context.allocator) -> Error { +// settings its package keeps unless options.ignore_stored. folder: under +// the folder of the documents, where the new documents of a first import go +// ("" in it), unless output_part names another. importer: what reads it, as +// its package records it (asset.Package_Origin.importer). +outputs_begin :: proc(o:^Outputs,source,folder:string,size,hash:u64,version:u32,options:Import_Options,importer:="",allocator:=context.allocator) -> Error { o^={size=size,hash=hash,version=version} caller:=context.temp_allocator mem.dynamic_arena_init(&o.scratch,allocator,allocator,block_size=1<<20,out_band_size=1<<20) context.temp_allocator=base.dynamic_arena_allocator(&o.scratch) report_init(&o.report,allocator) - o.source=source_path(options,source) + o.source=source_path(source) + o.importer=strings.clone(importer,context.temp_allocator) if !writer_init(&o.w,source,folder,options,&o.report,allocator) { - error:=outputs_error(o,caller) - outputs_end(o,true) + error:=writer_error(&o.w,caller) + _,_=outputs_end(o,true) return error } - // The settings kept with the source are those of every import of it. - if o.w.stored && !options.ignore_stored {o.w.options.settings=o.w.sidecar.settings} + o.w.version=version + // The settings its package keeps are those of every import of it. + if o.w.stored && !options.ignore_stored { + settings,valid:=settings_parse(o.w.previous.settings) + if !valid { + writer_fail(&o.w,.Malformed,"the settings the package of %s keeps (%s) are not those of this build: import it as a new package",source,o.w.previous.settings) + error:=writer_error(&o.w,caller) + _,_=outputs_end(o,true) + return error + } + o.w.options.settings=settings + } o.report.settings=o.w.options.settings return {} } -// The error of the writer, its message in allocator. -@(private="file") -outputs_error :: proc(o:^Outputs,allocator:mem.Allocator) -> Error { - return writer_error(&o.w,allocator) -} - // The file (in the temp allocator) and id of the root of the import: the -// output of kind the sidecar of the source lists, of that name when it lists +// asset of kind the package of the source lists, of that name when it lists // several of the kind, else a new one. output_root :: proc(o:^Outputs,kind:asset.Fourcc,name,extension:string) -> (id:base.Asset_Id,path:string,error:Error) { caller:=context.temp_allocator context.temp_allocator=base.dynamic_arena_allocator(&o.scratch) listed:=0 - for output in o.w.sidecar.outputs {listed+=int(output.kind==kind)} + for a in o.w.previous.assets {listed+=int(a.kind==kind)} parts:=[1]Part{{kind=kind,name=name,extension=extension,only=listed<=1}} parts_plan(&o.w,parts[:]) return parts[0].id,strings.clone(parts[0].path,caller),{} } -// A file the import reads besides its source: the sidecar lists it, and its -// bytes are part of what the outputs of the cache are built from -// (import_key), so sync imports the source again when it changes. Every -// input is given before the first output of the cache is planned, when that -// key is fixed: a new one after it is an error. +// A file the import reads besides its source: the package lists it, and its +// bytes are part of what the import is made from (identity_of). Every input +// is given before the first output of the cache is planned, when that is +// fixed: a new one after it is an error. output_input :: proc(o:^Outputs,file:string) -> Error { caller:=context.temp_allocator context.temp_allocator=base.dynamic_arena_allocator(&o.scratch) @@ -78,23 +86,13 @@ output_input :: proc(o:^Outputs,file:string) -> Error { return input_late(file,caller) } -// The file and id of another output of kind: the one the sidecar of the -// source lists with that name (the same index first), else a new one, a -// document in folder ("": that of the import). Asked for again with the same -// name and index, it is the same. +// The file to write and the id of another output of kind: the asset the +// package of the source lists with that name (the same index first), else a +// new one, a document of folder ("": that of the import) in the folder of the +// documents. Asked for again with the same name and index, it is the same. output_part :: proc(o:^Outputs,kind:asset.Fourcc,name:string,index:int,extension:string,folder:="") -> (id:base.Asset_Id,path:string) { - caller:=context.temp_allocator - context.temp_allocator=base.dynamic_arena_allocator(&o.scratch) - parts:=[1]Part{{kind=kind,name=name,index=index,extension=extension}} - if folder!="" { - kept:=o.w.folder - o.w.folder=folder - defer o.w.folder=kept - parts_plan(&o.w,parts[:]) - } else { - parts_plan(&o.w,parts[:]) - } - return parts[0].id,strings.clone(parts[0].path,caller) + ids,paths:=output_parts(o,kind,{name},index,extension,folder) + return ids[0],paths[0] } // The files and ids of other outputs of kind, by name, as output_part gives @@ -105,7 +103,7 @@ output_parts :: proc(o:^Outputs,kind:asset.Fourcc,names:[]string,index:int,exten parts:=make([]Part,len(names),context.temp_allocator) for name,i in names {parts[i]={kind=kind,name=name,index=index,extension=extension}} kept:=o.w.folder - if folder!="" {o.w.folder=folder} + if folder!="" && o.w.folder!="" {o.w.folder=folder} parts_plan(&o.w,parts) o.w.folder=kept ids=make([]base.Asset_Id,len(parts),caller) @@ -120,42 +118,55 @@ output_info :: proc(o:^Outputs,name:string,index:int,settings:u64=0) -> asset.Im return import_info(o.source,name,index,o.size,o.hash,settings,o.version) } -// Records an output written at path: the report lists it, and a failure of -// the import removes it when it is new. -output_written :: proc(o:^Outputs,id:base.Asset_Id,kind:asset.Fourcc,name,path:string,existed:bool,detail:="") { +// Takes the output id, written to the file its part gave: a blob of the +// build, a document also written or followed in the folder of the documents, +// and the report lists it as name. +output_written :: proc(o:^Outputs,id:base.Asset_Id,name:string,detail:="") -> Error { + caller:=context.temp_allocator context.temp_allocator=base.dynamic_arena_allocator(&o.scratch) - record(&o.w,id,kind,name,path,o.source,existed,detail) + for output,at in o.w.outputs { + if output.id!=id {continue} + if !writer_put(&o.w,at,o.w.paths[at]) { + error:=writer_error(&o.w,caller) + o.w.error={} + return error + } + record(&o.w,at,name,detail) + return {} + } + return error_make(.Malformed,"%s: no part of the import is the asset %v: write the outputs output_root and output_part give",o.source,id,allocator=caller) } output_warn :: proc(o:^Outputs,format:string,args:..any) { append(&o.report.warnings,fmt.aprintf(format,..args,allocator=o.report.allocator)) } -// Whether every output of the cache the sidecar lists has its file under the -// key of what the import builds from, or one under another key it may take: -// then the writer keeps each or builds it, and a model read ahead for it needs -// nothing cooked (model_cook). It fixes that key: -// every input is given (output_input) before. +// Whether the store has every output of the cache of the package, built from +// what the import is made from now: then the writer takes each, and a model +// read ahead for it needs nothing cooked (model_cook). It fixes what the +// import is made from: every input is given (output_input) before. outputs_current :: proc(o:^Outputs) -> bool { if current,known:=o.current.?;known {return current} + context.temp_allocator=base.dynamic_arena_allocator(&o.scratch) o.current=writer_current(&o.w) return o.current.? } -// Ends the outputs: writes the sidecar of the source and gives the report, -// with a warning for each output the sidecar listed that none took. failed -// removes the files the outputs created, leaves the sidecar as it was and -// gives no report. -outputs_end :: proc(o:^Outputs,failed:bool) -> (report:Import_Report) { +// Ends the outputs: puts the manifest of the package and its build in the +// store, names them, and gives the report, with a warning for each asset the +// package listed that none took. failed removes the documents the outputs +// created, records nothing and gives no report. +outputs_end :: proc(o:^Outputs,failed:bool,allocator:=context.allocator) -> (report:Import_Report,error:Error) { + caller:=allocator context.temp_allocator=base.dynamic_arena_allocator(&o.scratch) - if failed { + if failed || !writer_finish(&o.w,filepath.stem(o.source),o.importer if o.importer!="" else "whole") { + if !failed {error=writer_error(&o.w,caller)} writer_rollback(&o.w) report_destroy(&o.report) } else { - if !writer_finish(&o.w) {writer_warn(&o.w,"%s",o.w.error.message)} report=o.report } - asset.types_destroy(&o.w.types) + writer_close(&o.w) delete(o.w.error.message,o.w.allocator) mem.dynamic_arena_destroy(&o.scratch) o^={} diff --git a/cook/import/outputs_test.odin b/cook/import/outputs_test.odin index dda99e784..77de62272 100644 --- a/cook/import/outputs_test.odin +++ b/cook/import/outputs_test.odin @@ -3,75 +3,103 @@ package importer import "core:fmt" import "core:mem" import "core:os" +import "core:slice" import "core:strings" import "core:testing" import "mirai:asset" import "mirai:cook/textures" import "mirai:std/base" import mm "mirai:std/math" +import "mirai:store" + +// The package an import wrote, from the store. +@(private="file") +test_package :: proc(t:^testing.T,s:^store.Store,report:Import_Report) -> (p:asset.Package,b:asset.Build,ok:bool) { + error:asset.Error + if p,error=asset.package_get(s,report.manifest,context.temp_allocator);!testing.expect_value(t,error,asset.Error.None) {return} + if b,error=asset.build_get(s,report.build,context.temp_allocator);!testing.expect_value(t,error,asset.Error.None) {return} + return p,b,true +} // What a plugin that imports a source whole writes with Outputs: the root and -// the parts are the assets the sidecar of the source lists by kind and name, -// each in the folder its kind goes to, a part asked for twice is one, a part -// renamed in the source is another, and a run that fails leaves the sidecar as -// it was. +// the parts are the assets the package of the source lists by kind and name, +// a document also in the folder its kind goes to, a part asked for twice is +// one, a part renamed in the source is another, and a run that fails records +// nothing. @(test) -the_outputs_of_a_source_are_the_assets_its_sidecar_lists :: proc(t:^testing.T) { - root,assets:=test_assets(t,"outputs") +the_outputs_of_a_source_are_the_assets_its_package_lists :: proc(t:^testing.T) { + root,assets,s:=test_assets(t,"outputs") if root=="" {return} source:=fmt.tprintf("%s/level.map",assets) bytes:="a level" testing.expect(t,os.write_entire_file(source,bytes)==nil) - Pass :: struct {scene,floor:base.Asset_Id,materials:[2]base.Asset_Id,scene_path,floor_path:string,warnings:int} - pass :: proc(t:^testing.T,root,source,bytes:string,names:[]string,failed:=false) -> (p:Pass,ok:bool) { + Pass :: struct {scene,floor:base.Asset_Id,materials:[2]base.Asset_Id,floor_blob:store.Hash,report:Import_Report,warnings:int} + pass :: proc(t:^testing.T,s:^store.Store,root,source,bytes:string,names:[]string,failed:=false) -> (p:Pass,ok:bool) { o:Outputs - error:=outputs_begin(&o,source,"level",u64(len(bytes)),base.hash64(transmute([]u8)bytes),import_version(0),{root=root}) + error:=outputs_begin(&o,source,"level",u64(len(bytes)),base.hash64(transmute([]u8)bytes),import_version(0),{store=s,root=root,documents=fmt.tprintf("%s/assets",root)},allocator=context.temp_allocator) if !testing.expectf(t,error.code==.None,"%s",error.message) {return} root_error:Error - p.scene,p.scene_path,root_error=output_root(&o,asset.KIND_SCENE,"level",".scene") + p.scene,_,root_error=output_root(&o,asset.KIND_SCENE,"level",".scene") testing.expect_value(t,root_error.code,Error_Code.None) - p.floor,p.floor_path=output_part(&o,asset.KIND_MESH,"Floor",0,".mesh") + floor_path:string + p.floor,floor_path=output_part(&o,asset.KIND_TEXTURE,"Floor",0,".tex") ids,paths:=output_parts(&o,asset.KIND_MATERIAL,names,0,".mat",fmt.tprintf("%s/assets/level/materials",root)) for id,at in ids { if at0 && string(after)==string(kept),"a failed import leaves the sidecar as it was") - - // A part renamed in the source is a new one, and the one that is gone leaves the sidecar. - renamed,renamed_ok:=pass(t,root,source,bytes,{"Wood","Granite"}) + manifest,build,read:=test_package(t,s,first.report) + if !read {return} + testing.expect(t,len(manifest.assets)==4 && len(build.outputs)==3,"the scene planned is listed; what was written is built") + for o in build.outputs { + listed:=false + for a in manifest.assets {listed||=a.id==o.id && a.kind==o.kind} + testing.expect(t,listed && asset.object_whole(s,o.object),"each output of the build is an asset of its package, in the store") + } + testing.expect(t,os.exists(fmt.tprintf("%s/level/materials/Stone.mat",assets)),"a document is written in the folder of the documents") + + // Again: the same assets. A run that fails records nothing. + again,again_ok:=pass(t,s,root,source,bytes,{"Wood","Stone"}) + testing.expect(t,again_ok && again.scene==first.scene && again.floor==first.floor && again.floor_blob==first.floor_blob && again.materials=={first.materials[1],first.materials[0]} && again.warnings==0,"each output is found by its kind and name") + id_text:[base.ASSET_ID_TEXT]u8 + named,_:=store.ref_get(s,asset.REF_PACKAGES,base.asset_id_format(manifest.id,id_text[:])) + _,_=pass(t,s,root,source,bytes,{"Clay"},failed=true) + after,_:=store.ref_get(s,asset.REF_PACKAGES,base.asset_id_format(manifest.id,id_text[:])) + testing.expect(t,named==again.report.manifest && after==named,"a failed import leaves the package as it was") + + // A part renamed in the source is a new one, and the one that is gone leaves the package. + renamed,renamed_ok:=pass(t,s,root,source,bytes,{"Wood","Granite"}) testing.expect(t,renamed_ok && renamed.materials[0]==first.materials[1] && renamed.materials[1]!=first.materials[0],"Granite is not what Stone was") - _,gone_ok:=pass(t,root,source,bytes,{"Granite"}) - sidecar,found,_=sidecar_load(source,context.temp_allocator) - testing.expect(t,gone_ok && found && len(sidecar.outputs)==3,"Wood is no longer listed") + gone,gone_ok:=pass(t,s,root,source,bytes,{"Granite"}) + left,_,_:=test_package(t,s,gone.report) + testing.expect(t,gone_ok && len(left.assets)==3,"Wood is no longer listed") } // A quad of one material, which a model of the test draws. @@ -113,20 +141,20 @@ model_of :: proc(name,mesh,image:string,materials:..string) -> Imported_Model { // import: one part of the kind and name another model gave is that asset, // written once, and every part keeps its asset when the source is imported // again. A name with '/' goes to that folder, made safe; the files the import -// reads are what its cache is built from, given before the cache is. +// reads are what it is made from, given before the cache is planned. @(test) the_models_of_an_import_share_their_parts_by_name :: proc(t:^testing.T) { - root,assets:=test_assets(t,"outputs-models") + root,assets,s:=test_assets(t,"outputs-models") if root=="" {return} source:=fmt.tprintf("%s/level.map",assets) bytes:="a level of two props" testing.expect(t,os.write_entire_file(source,bytes)==nil) package_file:=fmt.tprintf("%s/props.pkg",assets) testing.expect(t,os.write_entire_file(package_file,"the props")==nil) - Pass :: struct {level:base.Asset_Id,crate,barrel:Import_Report,textures,materials,meshes:int,wood,stone:string} - pass :: proc(t:^testing.T,root,source,bytes,package_file:string,root_first:bool) -> (p:Pass,ok:bool) { + Pass :: struct {level:base.Asset_Id,crate,barrel,report:Import_Report,textures,materials,meshes:int,wood,stone:string} + pass :: proc(t:^testing.T,s:^store.Store,root,source,bytes,package_file:string,root_first:bool) -> (p:Pass,ok:bool) { o:Outputs - error:=outputs_begin(&o,source,"level",u64(len(bytes)),base.hash64(transmute([]u8)bytes),import_version(0),{root=root,textures=.Uncompressed}) + error:=outputs_begin(&o,source,"level",u64(len(bytes)),base.hash64(transmute([]u8)bytes),import_version(0),{store=s,root=root,documents=fmt.tprintf("%s/assets",root),textures=.Uncompressed},allocator=context.temp_allocator) if !testing.expectf(t,error.code==.None,"%s",error.message) {return} testing.expect_value(t,output_input(&o,package_file).code,Error_Code.None) if root_first {p.level,_,_=output_root(&o,asset.KIND_SCENE,"level",".scene")} @@ -139,8 +167,8 @@ the_models_of_an_import_share_their_parts_by_name :: proc(t:^testing.T) { late:=output_input(&o,fmt.tprintf("%s.more",package_file)) testing.expect(t,late.code!=.None && output_input(&o,package_file).code==.None,"a new input once the cache is planned is refused, a known one is not") if !root_first {p.level,_,_=output_root(&o,asset.KIND_SCENE,"level",".scene")} - report:=outputs_end(&o,false) - defer report_destroy(&report) + report,end_error:=outputs_end(&o,false,context.temp_allocator) + testing.expectf(t,end_error.code==.None,"%s",end_error.message) for written in report.assets { switch written.kind { case asset.KIND_TEXTURE: p.textures+=1 @@ -150,87 +178,267 @@ the_models_of_an_import_share_their_parts_by_name :: proc(t:^testing.T) { if written.name=="Materials/M_Wood" {p.wood=strings.clone(written.path,context.temp_allocator)} if written.name=="../Materials/./M_Stone" {p.stone=strings.clone(written.path,context.temp_allocator)} } + p.report=report return p,true } - first,ok:=pass(t,root,source,bytes,package_file,false) + first,ok:=pass(t,s,root,source,bytes,package_file,false) if !ok {return} crate,barrel:=first.crate,first.barrel testing.expect(t,crate.material_ids[0]==barrel.material_ids[0] && crate.image_ids[0]==barrel.image_ids[0],"the material and the texture of both models are one asset") testing.expect(t,crate.mesh_ids[0]!=barrel.mesh_ids[0] && crate.prefab!=barrel.prefab && crate.prefab!={} && barrel.material_ids[1]!={}) testing.expectf(t,first.textures==1 && first.materials==2 && first.meshes==2,"what is shared is written once: %v",first) - testing.expectf(t,strings.has_suffix(crate.prefab_path,"/assets/level/Crate.scene"),"%s",crate.prefab_path) testing.expectf(t,strings.has_suffix(first.wood,"/assets/level/Materials/M_Wood.mat"),"%s",first.wood) testing.expectf(t,strings.has_suffix(first.stone,"/assets/level/Materials/M_Stone.mat"),"%s goes nowhere else",first.stone) - // The registry lists the cache where the names say. - r:asset.Registry - asset.registry_init(&r,assets,allocator=context.temp_allocator,imported=fmt.tprintf("%s/.mirai/imported",root)) - testing.expect_value(t,asset.registry_scan(&r),asset.Error.None) - listed:=make(map[string]base.Asset_Id,context.temp_allocator) - for file in r.files {listed[file.path]=file.id} - testing.expectf(t,listed["level/Meshes/SM_Crate.mesh"]==crate.mesh_ids[0] && listed["level/Textures/T_Wood.tex"]==crate.image_ids[0],"%v",listed) + // The package lists the parts by the names the models give them, the prefab of each among them. + manifest,build,read:=test_package(t,s,first.report) + if !read {return} + named:=make(map[string]base.Asset_Id,context.temp_allocator) + for a in manifest.assets {named[a.name]=a.id} + testing.expectf(t,named["Meshes/SM_Crate"]==crate.mesh_ids[0] && named["Textures/T_Wood"]==crate.image_ids[0] && named["Crate"]==crate.prefab,"%v",named) + built:=0 + for o in build.outputs {built+=int(o.id==crate.mesh_ids[0] || o.id==crate.image_ids[0])} + testing.expect(t,built==2,"the build has the mesh and the texture") - // Again, with the root asked for first among the scenes its sidecar lists: the same assets. - again,again_ok:=pass(t,root,source,bytes,package_file,true) + // Again, with the root asked for first among the scenes its package lists: the same assets. + again,again_ok:=pass(t,s,root,source,bytes,package_file,true) testing.expect(t,again_ok && again.level==first.level && again.crate.mesh_ids[0]==crate.mesh_ids[0] && again.barrel.material_ids[1]==barrel.material_ids[1] && again.crate.image_ids[0]==crate.image_ids[0] && again.crate.prefab==crate.prefab,"each part keeps its asset") - // An input that changed changes what the cache is built from. - sidecar,_,_:=sidecar_load(source,context.temp_allocator) - before:=import_key(source,sidecar,import_version(0),0,false) + // An input that changed changes what the import is made from. + testing.expect(t,len(manifest.inputs)==1 && manifest.inputs[0]=="props.pkg","the input is listed") + settings,_:=settings_parse(manifest.settings) + before:=identity_of(manifest.id,path_key(source),manifest.inputs[:],settings,0) + testing.expect(t,before==manifest.identity,"the identity is that of the source, its input and its settings") testing.expect(t,os.write_entire_file(package_file,"other props")==nil) - testing.expect(t,len(sidecar.inputs)==1 && sidecar.inputs[0]=="props.pkg" && import_key(source,sidecar,import_version(0),0,false)!=before,"the input is listed and counts in the key") + testing.expect(t,identity_of(manifest.id,path_key(source),manifest.inputs[:],settings,0)!=before,"the input counts in the identity") } // An image only its format decodes is keyed by its data: a texture the // texture cache keeps for that data is written without making its texels. @(test) an_image_only_its_format_decodes_is_taken_from_the_cache_without_decoding :: proc(t:^testing.T) { - root,assets:=test_assets(t,"outputs-made") + root,assets,s:=test_assets(t,"outputs-made") if root=="" {return} - cache:=fmt.tprintf("%s/cache",root) + // A store of its own holds no texture of the data. + empty:=test_store(t,fmt.tprintf("%s/empty",root)) + if empty==nil {return} texels_of :: proc(data:[]u8,allocator:mem.Allocator) -> (made:textures.Source,ok:bool) { texels:=make([]u8,4*4*4,allocator) for &texel,at in texels {texel=u8(at*7)+data[0]} return {texels=texels,width=4,height=4,format=.RGBA8},true } refused :: proc(data:[]u8,allocator:mem.Allocator) -> (made:textures.Source,ok:bool) {return} - written :: proc(t:^testing.T,root,assets,name:string,make:Image_Make,cache:string) -> (texels:string) { + written :: proc(t:^testing.T,s:^store.Store,root,assets,name:string,make:Image_Make) -> (texels:string,report:Import_Report) { source:=fmt.tprintf("%s/%s.map",assets,name) testing.expect(t,os.write_entire_file(source,name)==nil) o:Outputs - error:=outputs_begin(&o,source,name,u64(len(name)),base.hash64(transmute([]u8)name),import_version(0),{root=root,textures=.Uncompressed,texture_cache=cache}) + error:=outputs_begin(&o,source,name,u64(len(name)),base.hash64(transmute([]u8)name),import_version(0),{store=s,root=root,textures=.Uncompressed},allocator=context.temp_allocator) if !testing.expectf(t,error.code==.None,"%s",error.message) {return} model:=model_of("Crate","SM_Crate","T_Wood","M_Wood") model.images[0]={name="T_Wood",data=transmute([]u8)string("packed"),make=make,srgb=true} _,error=output_model(&o,&model,context.temp_allocator) testing.expectf(t,error.code==.None,"%s",error.message) - report:=outputs_end(&o,false) - defer report_destroy(&report) + report,_=outputs_end(&o,false,context.temp_allocator) for w in report.assets { if w.kind!=asset.KIND_TEXTURE {continue} types:asset.Types;asset.types_init(&types,context.temp_allocator) a:asset.Asset - if asset.asset_load(&a,w.path,context.temp_allocator)!=.None {return} + if written_open(s,w,&a,context.temp_allocator)!=.None {return} view,view_error:=asset.texture_view(&a,&types) if view_error==.None {texels=strings.clone(string(asset.texture_mip(view,0)),context.temp_allocator)} } return } - first:=written(t,root,assets,"first",texels_of,cache) + first,report:=written(t,s,root,assets,"first",texels_of) testing.expect(t,len(first)==4*4*4,"its format makes the texels") - testing.expect(t,written(t,root,assets,"second",refused,cache)==first,"the cache gives the texture of the same data") - testing.expect(t,written(t,root,assets,"third",refused,"")=="","without the cache the texels are made, and here they cannot be") + second,_:=written(t,s,root,assets,"second",refused) + testing.expect(t,second==first,"the cache gives the texture of the same data") + third,_:=written(t,empty,root,assets,"third",refused) + testing.expect(t,third=="","without the cache the texels are made, and here they cannot be") // Imported again unchanged, its outputs are current: a model read ahead for it cooks nothing. - current :: proc(root,assets:string) -> bool { + current :: proc(s:^store.Store,root,assets:string) -> bool { o:Outputs source:=fmt.tprintf("%s/first.map",assets) - if outputs_begin(&o,source,"first",5,base.hash64(transmute([]u8)string("first")),import_version(0),{root=root,textures=.Uncompressed}).code!=.None {return false} + if outputs_begin(&o,source,"first",5,base.hash64(transmute([]u8)string("first")),import_version(0),{store=s,root=root,textures=.Uncompressed}).code!=.None {return false} defer outputs_end(&o,true) return outputs_current(&o) } - testing.expect(t,current(root,assets),"nothing changed") - entries,_:=os.read_all_directory_by_path(fmt.tprintf("%s/.mirai/imported",root),context.temp_allocator) - for entry in entries {_=os.remove(entry.fullpath)} - testing.expect(t,!current(root,assets),"its files are gone") + testing.expect(t,current(s,root,assets),"nothing changed") + _,build,read:=test_package(t,s,report) + if !read {return} + // A build whose mesh is an object the store does not hold is no current one. + for &o in build.outputs { + if o.kind==asset.KIND_MESH {o.object=store.address_of_bytes(transmute([]u8)string("gone"))} + } + stale,_:=asset.build_put(s,&build) + digits:[64]u8 + testing.expect_value(t,store.ref_set(s,asset.REF_IMPORTS,store.hex(report.key,&digits),stale),store.Error.None) + testing.expect(t,!current(s,root,assets),"an object is gone") +} + +// A source a model was read from, in a folder of its own that no project +// holds, its bytes those of name. +@(private="file") +model_source :: proc(t:^testing.T,folder,name:string,version:u32) -> (model:Imported_Model) { + model=model_of("Crate","SM_Crate","T_Wood","M_Wood") + model.source=path_key(fmt.tprintf("%s/%s.model",folder,name)) + testing.expect(t,os.write_entire_file(model.source,name)==nil) + model.source_size,model.source_hash=u64(len(name)),base.hash64(transmute([]u8)name) + model.format,model.folder,model.version="test",name,version + return +} + +// The blobs of a build by asset, sorted by id. +@(private="file") +build_blobs :: proc(t:^testing.T,s:^store.Store,report:Import_Report) -> []asset.Build_Output { + _,b,_:=test_package(t,s,report) + slice.sort_by(b.outputs[:],proc(a,b:asset.Build_Output) -> bool {return a.id[0] (report:Import_Report,ok:bool) { + model:=model_source(t,outside,"crate",import_version(0)) + error:Error + report,error=import_write(&model,{store=s,textures=.Uncompressed},context.temp_allocator) + return report,testing.expectf(t,error.code==.None,"%s",error.message) + } + first,ok:=import_once(t,s,outside) + if !ok {return} + entries,_:=os.read_all_directory_by_path(outside,context.temp_allocator) + testing.expect_value(t,len(entries),1) + again,again_ok:=import_once(t,s,outside) + if !again_ok {return} + testing.expect(t,again.package_id==first.package_id && again.manifest==first.manifest && again.build==first.build,"the same package, manifest and build") + testing.expect(t,again.mesh_ids[0]==first.mesh_ids[0] && again.material_ids[0]==first.material_ids[0] && again.image_ids[0]==first.image_ids[0] && again.prefab==first.prefab,"the same ids") + kept:=0 + for w in again.assets {kept+=int(w.kept)} + testing.expect(t,kept==2,"the mesh and the texture are taken from the build, not built") + p,_,_:=test_package(t,s,first) + testing.expect(t,p.origin.path==path_key(fmt.tprintf("%s/crate.model",outside)) && p.origin.importer=="test","its origin is where the source was") + for &a in p.assets { + if a.kind==asset.KIND_MESH {testing.expect(t,a.owner==first.prefab && a.part && slice.contains(a.dependencies[:],first.material_ids[0]),"a mesh is a part of the prefab and depends on its material")} + } +} + +// The identity of a package is what its source is: the settings change it, a +// version of the engine does not, and changes the key of its build instead. +@(test) +a_changed_setting_changes_the_identity_and_a_changed_engine_does_not :: proc(t:^testing.T) { + folder:=base.test_folder(t,"outputs-identity") + if folder=="" {return} + s:=test_store(t,fmt.tprintf("%s/store",folder)) + if s==nil {return} + written :: proc(t:^testing.T,s:^store.Store,folder:string,version:u32,options:Import_Options) -> (p:asset.Package,b:asset.Build) { + model:=model_source(t,folder,"crate",version) + options:=options + options.store,options.textures,options.ignore_stored=s,.Uncompressed,true + report,error:=import_write(&model,options,context.temp_allocator) + if !testing.expectf(t,error.code==.None,"%s",error.message) {return} + p,b,_=test_package(t,s,report) + return + } + first,first_build:=written(t,s,folder,import_version(0),{}) + upgraded,upgraded_build:=written(t,s,folder,import_version(9),{}) + testing.expect(t,first.identity!={} && upgraded.identity==first.identity && upgraded_build.key!=first_build.key,"another version of the reader keeps the identity and changes the key") + lods,_:=written(t,s,folder,import_version(0),{lods=.None}) + testing.expect(t,lods.identity!=first.identity && lods.id==first.id,"a setting changes the identity of the same package") + testing.expect(t,strings.contains(lods.settings,"lods"),"the package keeps the settings that differ from the defaults") +} + +// A build of Fast textures is named apart from the Final builds, where only a +// refresh that takes Fast looks, while the machine chooses it for the package. +@(test) +a_fast_build_is_named_apart_from_the_final_builds :: proc(t:^testing.T) { + folder:=base.test_folder(t,"outputs-fast") + if folder=="" {return} + s:=test_store(t,fmt.tprintf("%s/store",folder)) + if s==nil {return} + model:=model_source(t,folder,"crate",import_version(0)) + report,error:=import_write(&model,{store=s,textures=.Uncompressed,fast_textures=true},context.temp_allocator) + if !testing.expectf(t,error.code==.None,"%s",error.message) {return} + testing.expect(t,!report.final,"its texture is Fast") + digits:[64]u8 + _,shared:=store.ref_get(s,asset.REF_IMPORTS,store.hex(report.key,&digits)) + fast,local:=store.ref_get(s,asset.REF_IMPORTS_FAST,store.hex(report.key,&digits)) + testing.expect(t,shared==.Missing && local==.None && fast==report.build,"published under the namespace of Fast builds alone") + p,_,_:=test_package(t,s,report) + chosen,_:=store.ref_get(s,asset.REF_BUILDS,store.hex(asset.build_select(report.manifest,&p),&digits)) + testing.expect(t,chosen==report.build,"the build this machine chose") + list:=asset.Packages{packages=make([dynamic]asset.Project_Package,context.temp_allocator)} + append(&list.packages,asset.Project_Package{id=p.id,manifest=report.manifest}) + resolved,resolve_error:=asset.packages_resolve(s,"",&list,context.temp_allocator) + testing.expect(t,resolve_error==.None && len(resolved.problems)==0 && len(resolved.listed)==2,"a project lists the mesh and the texture of the package") + for l in resolved.listed {testing.expectf(t,l.state==.Built && asset.object_whole(s,l.object) && l.header.id==l.id,"%s is built",l.path)} +} + +// Two machines that import one source as the same package write the same +// blobs: what an import writes depends on no machine. +@(test) +two_stores_that_import_one_source_hold_the_same_blobs :: proc(t:^testing.T) { + folder:=base.test_folder(t,"outputs-stores") + if folder=="" {return} + here:=test_store(t,fmt.tprintf("%s/here",folder)) + there:=test_store(t,fmt.tprintf("%s/there",folder)) + if here==nil || there==nil {return} + model:=model_source(t,folder,"crate",import_version(0)) + first,error:=import_write(&model,{store=here,textures=.Uncompressed},context.temp_allocator) + if !testing.expectf(t,error.code==.None,"%s",error.message) {return} + // The other machine has the manifest, as a peer gives it, and the source. + p,_,_:=test_package(t,here,first) + manifest,put_error:=asset.package_put(there,&p) + testing.expect(t,put_error==.None && manifest==first.manifest,"the manifest is the same blob") + model=model_source(t,folder,"crate",import_version(0)) + second:Import_Report + second,error=import_write(&model,{store=there,manifest=manifest,textures=.Uncompressed},context.temp_allocator) + if !testing.expectf(t,error.code==.None,"%s",error.message) {return} + testing.expect(t,second.manifest==first.manifest && second.build==first.build,"the same manifest and build") + a,b:=build_blobs(t,here,first),build_blobs(t,there,second) + testing.expect(t,len(a)==len(b) && len(a)>0) + for o,at in a { + if at string { + settings,defaults:=settings,Import_Settings{} + schema:serial.Schema + serial.schema_init(&schema,context.temp_allocator) + defer serial.schema_destroy(&schema) + type,_:=serial.schema_add(&schema,Import_Settings) + w:serial.Writer + serial.writer_init(&w,512,allocator) + serial.text_write_changed(&w,&schema,type,([^]u8)(&settings)[:size_of(Import_Settings)],([^]u8)(&defaults)[:size_of(Import_Settings)],.Pretty,depth=1) + return string(serial.writer_take(&w)) +} + +// False for a text that is not settings this build reads: a field or a value +// it does not know, or one out of its range. "" is the defaults. +settings_parse :: proc(text:string) -> (settings:Import_Settings,ok:bool) { + if text!="" { + schema:serial.Schema + serial.schema_init(&schema,context.temp_allocator) + defer serial.schema_destroy(&schema) + type,_:=serial.schema_add(&schema,Import_Settings) + r:serial.Text_Reader + serial.text_reader_init(&r,text,context.temp_allocator) + defer serial.text_reader_destroy(&r) + serial.text_read(&r,&schema,type,([^]u8)(&settings)[:size_of(Import_Settings)]) or_return + } + return settings,settings_valid(settings) +} + +@(private="file") +settings_valid :: proc(settings:Import_Settings) -> bool { + return settings.textures<=max(textures.Encoding) && settings.rdo_lambda<=textures.RDO_LAMBDA_MAX && + settings.image<=max(Image_Use) && settings.alpha_coverage>=0 && settings.alpha_coverage<=1 && settings.lods<=max(Lods) && settings.parts<=max(Parts) && settings.front<=max(Front) && settings.rig<=max(Rig) && + settings.rig_crotch>=0 && settings.rig_neck>=0 && (settings.rig==.Humanoid || settings.rig_crotch==0 && settings.rig_neck==0) +} + +// What the outputs of the package package_id are made from, without any +// version of the engine (asset.Package.identity): the package, whose ids its +// blobs hold, so two packages of the same bytes have builds of their own; the +// bytes of the source at path and of each input, named from the folder of the +// source; the text of its settings and the code of the plugin that imports it +// whole (Import_Options.importer_code). A file that cannot be read, such as a +// folder, counts as no bytes. With hashes, a file whose size and modification +// time it knows is not read. +identity_of :: proc(package_id:base.Asset_Id,path:string,inputs:[]string,settings:Import_Settings,code:u64,hashes:^Source_Hashes=nil) -> store.Hash { + TAG :: "mirai import identity" + bytes:=make([dynamic]u8,context.temp_allocator) + append(&bytes,TAG) + package_id:=package_id + append(&bytes,..([^]u8)(&package_id)[:size_of(package_id)]) + address :: proc(bytes:^[dynamic]u8,file:string,hashes:^Source_Hashes) { + hash:=store.address_of_bytes(nil) + if !unsafe_path(file) {hash=file_address(hashes,file)} + append(bytes,..hash[:]) + } + address(&bytes,path,hashes) + folder:=path[:max(strings.last_index_byte(path,'/'),0)] + for input in inputs { + append(&bytes,input) + append(&bytes,0) + address(&bytes,fmt.tprintf("%s/%s",folder,input),hashes) + } + append(&bytes,settings_text(settings,context.temp_allocator)) + append(&bytes,0) + code:=code + append(&bytes,..([^]u8)(&code)[:size_of(code)]) + return store.address_of_bytes(bytes[:]) +} + +// The addresses of the files the identity of one source read, by path, as +// they were when read: kept by the library of the machine per source +// (asset.REF_SOURCES), so finding whether a source changed reads only the +// files whose size or modification time changed. The bytes still decide: a +// file that changed is read again. +Source_Hashes :: struct { + files:map[string]Source_Hash, + changed:bool, +} +Source_Hash :: struct {size,modified:i64,hash:store.Hash} + +// A file modified this close to when it was read may change again within the +// same modification time (2 s on FAT, a coarse network file system): what was +// learned of it is not kept, and it is read again the next time. +SETTLED_NANOSECONDS :: 2_000_000_000 + +// The hashes kept for the source at origin (path_key), in allocator; none +// when the store has none or they do not read. +source_hashes_load :: proc(s:^store.Store,origin:string,allocator:=context.allocator) -> (hashes:Source_Hashes) { + hashes.files=make(map[string]Source_Hash,allocator=allocator) + name:[64]u8 + hash,ref_error:=store.ref_get(s,asset.REF_SOURCES,asset.origin_name(origin,&name)) + if ref_error!=.None {return} + text,read_error:=asset.record_get(s,hash) + if read_error!=.None {return} + for line in strings.split_lines_iterator(&text) { + parts:=strings.split_n(line," ",4,context.temp_allocator) + if len(parts)!=4 {continue} + address,address_ok:=store.parse_hex(parts[0]) + file_size,size_ok:=strconv.parse_i64(parts[1]) + modified,modified_ok:=strconv.parse_i64(parts[2]) + if address_ok && size_ok && modified_ok {hashes.files[strings.clone(parts[3],allocator)]={file_size,modified,address}} + } + return +} + +// Keeps the hashes of the source at origin when reading it changed them. +source_hashes_save :: proc(s:^store.Store,origin:string,hashes:^Source_Hashes) { + if !hashes.changed {return} + files:=make([dynamic]string,0,len(hashes.files),context.temp_allocator) + for file in hashes.files {append(&files,file)} + slice.sort(files[:]) + b:=strings.builder_make(context.temp_allocator) + digits:[64]u8 + for file in files { + known:=hashes.files[file] + fmt.sbprintf(&b,"%s %d %d %s\n",store.hex(known.hash,&digits),known.size,known.modified,file) + } + hash,error:=asset.record_put(s,transmute([]u8)strings.to_string(b)) + if error!=.None {return} + name:[64]u8 + _=store.ref_set(s,asset.REF_SOURCES,asset.origin_name(origin,&name),hash) +} + +// The BLAKE3 of the bytes of the file at path, read once; false when it does not read. +@(private="file") +file_hash :: proc(path:string) -> (hash:store.Hash,read:bool) { + file,open_error:=os.open(path) + if open_error!=nil {return} + defer os.close(file) + size,size_error:=os.file_size(file) + if size_error!=nil {return} + hasher:blake3.Hasher + buffer:[64<<10]u8=--- + for at:=i64(0);at store.Hash { + info,stat_error:=os.stat(file,context.temp_allocator) + size,modified:=info.size,time.time_to_unix_nano(info.modification_time) + if hashes!=nil && stat_error==nil { + if known,found:=hashes.files[file];found && known.size==size && known.modified==modified {return known.hash} + } + hash,read:=file_hash(file) + if !read {return store.address_of_bytes(nil)} + if hashes!=nil && stat_error==nil && time.time_to_unix_nano(time.now())-modified>SETTLED_NANOSECONDS { + if _,found:=hashes.files[file];!found {hashes.files[strings.clone(file,hashes.files.allocator)]={}} + hashes.files[file]={size,modified,hash} + hashes.changed=true + } + return hash +} + +// The versions of what reads a source with a format of version +// (import_version) and writes its assets, as asset.import_key combines them: +// the writer and the format, every kind of asset and what builds textures. +import_versions :: proc(version:u32) -> [3]u64 { + kinds:=make([dynamic]u64,0,len(asset.KINDS),context.temp_allocator) + for info in asset.KINDS {append(&kinds,u64(info.kind)<<16|u64(info.version))} + return {u64(version),base.hash64(slice.to_bytes(kinds[:])),u64(textures.BUILD_VERSION)} +} + +// The manifest of the package the source at origin (an absolute path with +// forward slashes) was ingested as: the one manifest names, else the one the +// ref REF_ORIGINS of this machine names for origin. found is +// false without one. In allocator, an arena. +package_previous :: proc(s:^store.Store,manifest:store.Hash,origin:string,allocator:=context.allocator) -> (p:asset.Package,hash:store.Hash,found:bool,error:Error) { + digits:[64]u8 + if manifest!={} { + read_error:asset.Error + p,read_error=asset.package_get(s,manifest,allocator) + if read_error==.Not_Found {return {},{},false,error_make(.Not_Found,"the manifest %s of the package of %s is not in the store at %s: fetch it from a peer that has it, or import the source as a new package",store.hex(manifest,&digits),origin,s.root)} + if read_error!=.None {return {},{},false,error_make(.Malformed,"the manifest %s of the package of %s does not read (%v): import the source as a new package",store.hex(manifest,&digits),origin,read_error)} + return p,manifest,true,{} + } + name:[64]u8 + named,ref_error:=store.ref_get(s,asset.REF_ORIGINS,asset.origin_name(origin,&name)) + if ref_error!=.None {return} + read_error:asset.Error + if p,read_error=asset.package_get(s,named,allocator);read_error!=.None || p.origin.path!=origin_of(origin) {return {},{},false,{}} + return p,named,true,{} +} + +// The settings of the package the source at path was ingested as +// (package_previous with options.manifest), the defaults (found false) when it +// was not or without options.store. +settings_load :: proc(options:Import_Options,path:string) -> (settings:Import_Settings,found:bool,error:Error) { + if options.store==nil {return} + p:asset.Package + p,_,found,error=package_previous(options.store,options.manifest,path_key(path),context.temp_allocator) + if !found || error.code!=.None {return} + valid:bool + if settings,valid=settings_parse(p.settings);!valid { + return {},false,error_make(.Malformed,"the settings the package of %s keeps (%s) are not those of this build: import it as a new package",path,p.settings) + } + return +} + +// The asset the package the source at path was ingested as on this machine +// gave what the source names name, of kind; false when it gave none. +output_id :: proc(s:^store.Store,path:string,kind:asset.Fourcc,name:string) -> (id:base.Asset_Id,found:bool) { + p,_,known,error:=package_previous(s,{},path_key(path),context.temp_allocator) + if !known || error.code!=.None {return} + for a in p.assets { + if a.kind==kind && a.name==name {return a.id,true} + } + return +} + +// Puts the manifest of the package and its build in the store and names +// them: the package with REF_PACKAGES and by its origin (REF_ORIGINS), the build with REF_IMPORTS when its +// textures are Final, else REF_IMPORTS_FAST, and as the build this machine +// chose for the manifest (REF_BUILDS). +@(private) +writer_publish :: proc(w:^Writer,name,importer:string) -> bool { + s:=w.options.store + writer_identity(w) + p:=asset.Package{id=w.package_id,name=name,identity=w.identity,inputs=w.inputs,assets=w.outputs} + p.origin={path=origin_of(w.origin),folder=os.is_dir(w.path),importer=importer} + if settings:=settings_text(w.report.settings,context.temp_allocator);settings!="{}" {p.settings=settings} + slice.sort(p.inputs[:]) + b:=asset.Build{package_id=w.package_id,key=w.key,final=!w.fast} + b.outputs=make([dynamic]asset.Build_Output,context.temp_allocator) + for output,at in w.outputs { + if w.objects[at]=={} {continue} + info,_:=asset.kind_info(output.kind) + append(&b.outputs,asset.Build_Output{id=output.id,kind=output.kind,kind_version=info.version,object=w.objects[at]}) + } + manifest,manifest_error:=asset.package_put(s,&p) + if manifest_error!=.None {return writer_fail(w,.Write,"the manifest of the package of %s could not be put in the store at %s: %v",w.path,s.root,manifest_error)} + build,build_error:=asset.build_put(s,&b) + if build_error!=.None {return writer_fail(w,.Write,"the build of the package of %s could not be put in the store at %s: %v",w.path,s.root,build_error)} + id_text:[base.ASSET_ID_TEXT]u8 + key_text,select_text,origin_text:[64]u8 + refs:=[4]struct{namespace,name:string}{ + {asset.REF_PACKAGES,base.asset_id_format(w.package_id,id_text[:])}, + {asset.REF_ORIGINS,asset.origin_name(w.origin,&origin_text)}, + {asset.REF_IMPORTS if b.final else asset.REF_IMPORTS_FAST,store.hex(w.key,&key_text)}, + {asset.REF_BUILDS,store.hex(asset.build_select(manifest,&p),&select_text)}, + } + for ref,at in refs { + if error:=store.ref_set(s,ref.namespace,ref.name,manifest if at<2 else build);error!=.None { + return writer_fail(w,.Write,"the ref %s/%s of the package of %s could not be set in the store at %s: %v",ref.namespace,ref.name,w.path,s.root,error) + } + } + w.report.package_id,w.report.manifest,w.report.build,w.report.key,w.report.final=w.package_id,manifest,build,w.key,b.final + return true +} + +// Fixes what the import is made from once its inputs are whole: its identity +// and the key of its build (asset.import_key). +@(private) +writer_identity :: proc(w:^Writer) { + if w.fixed {return} + w.fixed=true + slice.sort(w.inputs[:]) + w.identity=identity_of(w.package_id,w.origin,w.inputs[:],w.report.settings,w.options.importer_code,w.options.hashes) + w.key=build_key(w.identity,w.version,w.options.importer_code) +} + +// The key of the build of a package of identity (asset.import_key): the +// versions of what reads its source with a format of version (import_version) +// and, for a source a plugin imports whole, the identity of the code of that +// plugin (code; 0: none), so a changed importer builds again also where its +// source is not. +build_key :: proc(identity:store.Hash,version:u32,code:u64) -> store.Hash { + versions:=import_versions(version) + all:=[4]u64{versions[0],versions[1],versions[2],code} + key,_:=asset.import_key(identity,all[:3 if code==0 else 4]) + return key +} diff --git a/cook/import/progress.odin b/cook/import/progress.odin index 359867d14..19b73d391 100644 --- a/cook/import/progress.odin +++ b/cook/import/progress.odin @@ -5,7 +5,7 @@ import "core:io" import "core:time" // What an import says of how far it is while it runs, for a sync to print -// under the line of its source: the assets it wrote, of those its sidecar +// under the line of its source: the assets it wrote, of those its package // listed (what the previous import gave), and how long it has run; a line at // most every PROGRESS_EVERY. The caller owns it; the thread that writes the // import updates it. diff --git a/cook/import/prompts.odin b/cook/import/prompts.odin index d1301f1e9..b65a09544 100644 --- a/cook/import/prompts.odin +++ b/cook/import/prompts.odin @@ -220,10 +220,12 @@ import_prompts :: proc(path:string,options:Import_Options={},allocator:=context. w:Writer ok:=writer_init(&w,path,"",options,&report,allocator) w.options.textures=.Uncompressed - defer asset.types_destroy(&w.types) + defer writer_close(&w) if ok { for file in files {writer_input(&w,file)} - ok=prompts_write(&w,path,bytes,entries,of_entry,images[:],places,pages) && writer_finish(&w) + name:string + ok,name=prompts_write(&w,path,bytes,entries,of_entry,images[:],places,pages) + ok=ok && writer_finish(&w,name,"prompts") } if ok && skipped>0 {append(&report.warnings,fmt.aprintf("%d images of %s say no input by their names and were left out; Project Settings > UI gives any input an image of its own",skipped,path,allocator=allocator))} if !ok { @@ -236,13 +238,14 @@ import_prompts :: proc(path:string,options:Import_Options={},allocator:=context. } @(private="file") -prompts_write :: proc(w:^Writer,path:string,bytes:[]u8,entries:[]Prompts_Entry,of_entry:[]int,images:[]textures.Image,places:[]Prompts_Place,pages:[][2]int) -> bool { +prompts_write :: proc(w:^Writer,path:string,bytes:[]u8,entries:[]Prompts_Entry,of_entry:[]int,images:[]textures.Image,places:[]Prompts_Place,pages:[][2]int) -> (ok:bool,name:string) { PAD :: PROMPTS_PADDING source:=w.source file:=filepath.base(path) - name:=file[:len(file)-len(PROMPTS_MANIFEST)] if strings.has_suffix(strings.to_lower(file,context.temp_allocator),PROMPTS_MANIFEST) else filepath.stem(file) + name=file[:len(file)-len(PROMPTS_MANIFEST)] if strings.has_suffix(strings.to_lower(file,context.temp_allocator),PROMPTS_MANIFEST) else filepath.stem(file) target:string - w.owner,target=root_take(w,asset.KIND_PROMPTS,name,".prompts") + root:int + w.owner,target,root=root_take(w,asset.KIND_PROMPTS,name,".prompts") // The texels of each page: the images, each with its edge repeated around it. jobs:=make([]Texture_Job,len(pages),context.temp_allocator) @@ -267,12 +270,10 @@ prompts_write :: proc(w:^Writer,path:string,bytes:[]u8,entries:[]Prompts_Entry,o parts[page]=Part{kind=asset.KIND_TEXTURE,name=job.name,index=page,extension=".tex"} } parts_plan(w,parts) - for &part in parts {part.owner=w.owner} - parts_record(w,parts) for &part,page in parts { - if !write_texture(w,jobs[page],part.id,w.owner,part.path,source) { - if w.error.code==.None {writer_fail(w,.Write,"the atlas %s could not be built",part.path)} - return false + if !write_texture(w,jobs[page],part.id,w.owner,part.path,source,part.output) { + if w.error.code==.None {writer_fail(w,.Write,"the atlas %s of %s could not be built",part.name,path)} + return false,name } } @@ -288,8 +289,8 @@ prompts_write :: proc(w:^Writer,path:string,bytes:[]u8,entries:[]Prompts_Entry,o row.aspect=f32(image.width)/f32(image.height) } info:=import_info(source,name,0,u64(len(bytes)),base.hash64(bytes),0,import_version(0)) - existed:=os.exists(target) - if error:=asset.prompts_save(target,&w.types,{id=w.owner,images=rows,source=&info},context.temp_allocator);error!=.None {return save_failed(w,error,target)} - record(w,w.owner,asset.KIND_PROMPTS,name,target,source,existed,fmt.tprintf("%d prompts of %d images in %d atlas%s",len(rows),len(images),len(pages),len(pages)==1?"":"es")) - return true + if error:=asset.prompts_save(target,&w.types,{id=w.owner,images=rows,source=&info},context.temp_allocator);error!=.None {return save_failed(w,error,target),name} + if !writer_put(w,root,target) {return false,name} + record(w,root,name,fmt.tprintf("%d prompts of %d images in %d atlas%s",len(rows),len(images),len(pages),len(pages)==1?"":"es")) + return true,name } diff --git a/cook/import/reader.odin b/cook/import/reader.odin index 9808c0c1c..5c8d48421 100644 --- a/cook/import/reader.odin +++ b/cook/import/reader.odin @@ -19,7 +19,7 @@ import mm "mirai:std/math" // space when the file says it (source_space), and returns false after fail. Reader :: struct { out:mem.Allocator, - options:Import_Options, // its settings are those the import applies: .import when it has one + options:Import_Options, // its settings are those the import applies: those its package keeps when it has one data:rawptr, // Format.data: the options of the format path:string, // of the source: absolute, with forward slashes directory:string, diff --git a/cook/import/reader_test.odin b/cook/import/reader_test.odin index 5df0119d8..af9637f68 100644 --- a/cook/import/reader_test.odin +++ b/cook/import/reader_test.odin @@ -3,7 +3,9 @@ package importer import "core:fmt" import "core:os" import "core:testing" +import "mirai:asset" import "mirai:std/base" +import "mirai:store" // What the reader of the test below saw in its options. @(private="file") @@ -22,23 +24,29 @@ SETTINGS_FORMAT :: Format{name="Settings test",extensions={0=".seen"},space=ENGI }} // A reader that acts on a setting (Import_Settings.skeleton: clips that keep no skeleton of their -// own) reads it from the options: those of .import when the source has one, as the writer -// applies them, else those the import was given. +// own) reads it from the options: those the package of the source keeps when it was ingested, as +// the writer applies them, else those the import was given. @(test) a_reader_sees_the_settings_kept_with_its_source :: proc(t:^testing.T) { folder:=base.test_folder(t,"reader-settings") if folder=="" {return} + s:=test_store(t,fmt.tprintf("%s/store",folder)) + if s==nil {return} path:=fmt.tprintf("%s/model.seen",folder) _=os.write_entire_file(path,"anything") formats:=[1]Format{SETTINGS_FORMAT} given:=base.Asset_Id{1,2} - model,error:=import_model(path,formats[:],{root=folder,settings={skeleton=given}}) + model,error:=import_model(path,formats[:],{store=s,settings={skeleton=given}}) if !testing.expectf(t,error.code==.None,"%s",error.message) {return} model_destroy(&model) testing.expect_value(t,seen.skeleton,given) kept:=base.Asset_Id{3,4} - if !testing.expect(t,sidecar_save(path,{settings={skeleton=kept}})) {return} - model,error=import_model(path,formats[:],{root=folder,settings={skeleton=given}}) + ingested:=asset.Package{id={5,6},name="model",origin={path=path_key(path)},settings=settings_text({skeleton=kept},context.temp_allocator)} + manifest,put_error:=asset.package_put(s,&ingested) + if !testing.expect_value(t,put_error,asset.Error.None) {return} + origin:[64]u8 + testing.expect_value(t,store.ref_set(s,asset.REF_ORIGINS,asset.origin_name(ingested.origin.path,&origin),manifest),store.Error.None) + model,error=import_model(path,formats[:],{store=s,settings={skeleton=given}}) if !testing.expectf(t,error.code==.None,"%s",error.message) {return} model_destroy(&model) testing.expect_value(t,seen.skeleton,kept) diff --git a/cook/import/settings.odin b/cook/import/settings.odin index d43706d2b..1429012b8 100644 --- a/cook/import/settings.odin +++ b/cook/import/settings.odin @@ -6,10 +6,11 @@ import "core:strings" import "mirai:cook/textures" import "mirai:std/base" -// How a source becomes assets. The settings of a source are kept in its -// sidecar (sidecar.odin), so importing it again gives the same assets whoever -// does it: an import reads them when the sidecar exists and uses them instead -// of those of its options. The zero value is the default of every setting. +// How a source becomes assets. The settings of a source are kept in the +// manifest of its package (package.odin), so importing it again gives the same +// assets whoever does it: an import of a package ingested before uses them +// instead of those of its options. The zero value is the default of every +// setting. Lods :: enum u8 { Auto, // simplified levels are stored after level 0 (mesh_levels) diff --git a/cook/import/sidecar.odin b/cook/import/sidecar.odin deleted file mode 100644 index aafd9c5fa..000000000 --- a/cook/import/sidecar.odin +++ /dev/null @@ -1,209 +0,0 @@ -package importer - -import "core:fmt" -import "core:os" -import "core:path/filepath" -import "core:slice" -import "core:strconv" -import "core:strings" -import "core:time" -import "mirai:asset" -import "mirai:cook/textures" -import "mirai:std/base" -import "mirai:std/serial" - -// .import, the sidecar (asset/sidecar.odin): what is versioned of an -// import with its source, beside it in the assets of a project or a plugin. -// It says how the source becomes assets (settings), where its outputs go and -// which asset each thing of the source is (outputs): nothing else decides an -// id. The meshes, textures, sounds and fonts (asset.import_cached), which -// nobody edits, are files of the imported folder of the root named by their -// ids, never versioned: the registry lists them by where the sidecar is, and -// sources.sync builds them again. The other kinds are documents of the -// assets, found by their id. - -SIDECAR_EXTENSION :: asset.SIDECAR_EXTENSION -// Of the empty file of the imported folder that says an import of documents -// alone is current, named as a file of the cache is by the id of its first -// output and its key: what sync looks for when no output is of the cache. -STAMP_EXTENSION :: ".stamp" - -Output :: asset.Import_Output - -Sidecar :: struct { - settings:Import_Settings, - folder:string, // under the folder of the source, where its outputs go; "": beside it - inputs:[dynamic]string, // the files the import read besides the source, from the folder of the source: with it, what the outputs are built from - outputs:[dynamic]Output, -} - -@(private="file") -settings_valid :: proc(settings:Import_Settings) -> bool { - return settings.textures<=max(textures.Encoding) && settings.rdo_lambda<=textures.RDO_LAMBDA_MAX && - settings.image<=max(Image_Use) && settings.alpha_coverage>=0 && settings.alpha_coverage<=1 && settings.lods<=max(Lods) && settings.parts<=max(Parts) && settings.front<=max(Front) && settings.rig<=max(Rig) && - settings.rig_crotch>=0 && settings.rig_neck>=0 && (settings.rig==.Humanoid || settings.rig_crotch==0 && settings.rig_neck==0) -} - -// The settings as the text of their object, in the sidecar: only where they -// differ from their defaults (the zero value), so a sidecar shows what was -// chosen; reading fills the rest. -@(private="file") -settings_text :: proc(settings:Import_Settings,allocator:=context.allocator) -> string { - settings,defaults:=settings,Import_Settings{} - schema:serial.Schema - serial.schema_init(&schema,context.temp_allocator) - defer serial.schema_destroy(&schema) - type,_:=serial.schema_add(&schema,Import_Settings) - w:serial.Writer - serial.writer_init(&w,512,allocator) - serial.text_write_changed(&w,&schema,type,([^]u8)(&settings)[:size_of(Import_Settings)],([^]u8)(&defaults)[:size_of(Import_Settings)],.Pretty,depth=1) - return string(serial.writer_take(&w)) -} - -sidecar_text :: proc(sidecar:Sidecar,allocator:=context.allocator) -> string { - return asset.sidecar_text(settings_text(sidecar.settings,context.temp_allocator),{folder=sidecar.folder,inputs=sidecar.inputs,outputs=sidecar.outputs},allocator) -} - -// False for a text that is not the sidecar of a source (asset.sidecar_parse) -// or whose settings this build does not read. What it holds lives in -// allocator, an arena: nothing frees it piece by piece. -sidecar_parse :: proc(text:string,allocator:=context.allocator) -> (sidecar:Sidecar,ok:bool) { - parsed:=asset.sidecar_parse(text,allocator) or_return - sidecar.folder,sidecar.inputs,sidecar.outputs=parsed.folder,parsed.inputs,parsed.outputs - if parsed.settings!="" { - schema:serial.Schema - serial.schema_init(&schema,context.temp_allocator) - defer serial.schema_destroy(&schema) - type,_:=serial.schema_add(&schema,Import_Settings) - r:serial.Text_Reader - serial.text_reader_init(&r,parsed.settings,context.temp_allocator) - defer serial.text_reader_destroy(&r) - serial.text_read(&r,&schema,type,([^]u8)(&sidecar.settings)[:size_of(Import_Settings)]) or_return - } - return sidecar,settings_valid(sidecar.settings) -} - -// What the outputs of an import are built from, as one number: the bytes of -// the source at path and of each input its sidecar lists, its settings, the -// version of what reads and writes it, the code of the plugin that imports it -// whole (Import_Options.importer_code), the versions of the kinds of the -// cache, the encoders of the program and whether its textures are built Fast. -// An output of the cache is current when its file is named by this key. With -// hashes, a file whose size and modification time it knows is not read. -import_key :: proc(path:string,sidecar:Sidecar,version:u32,code:u64,fast:bool,hashes:^Source_Hashes=nil) -> u64 { - words:=make([dynamic]u64,0,7+len(sidecar.inputs),context.temp_allocator) - append(&words,u64(version)<<1|u64(fast),u64(asset.SOUND_VERSION)<<48|u64(asset.MESH_VERSION)<<32|u64(asset.TEXTURE_VERSION)<<16|u64(textures.BUILD_VERSION),u64(asset.FONT_VERSION),code) - append(&words,base.hash64(transmute([]u8)settings_text(sidecar.settings,context.temp_allocator)),textures.encoders_id()) - folder:=filepath.dir(path) - for at in -1.. (hashes:Source_Hashes) { - hashes.files=make(map[string]Source_Hash,allocator=allocator) - data,error:=serial.file_read(path,context.temp_allocator) - if error!=.None {return} - text:=string(data) - for line in strings.split_lines_iterator(&text) { - parts:=strings.split_n(line," ",4,context.temp_allocator) - if len(parts)!=4 {continue} - hash,hash_ok:=strconv.parse_u64(parts[0],16) - size,size_ok:=strconv.parse_i64(parts[1]) - modified,modified_ok:=strconv.parse_i64(parts[2]) - if hash_ok && size_ok && modified_ok {hashes.files[strings.clone(parts[3],allocator)]={size,modified,hash}} - } - return -} - -// Writes the hashes to path when a build changed them. -source_hashes_save :: proc(hashes:^Source_Hashes,path:string) { - if !hashes.changed {return} - b:=strings.builder_make(context.temp_allocator) - for file,known in hashes.files {fmt.sbprintf(&b,"%016x %d %d %s\n",known.hash,known.size,known.modified,file)} - if folder:=filepath.dir(path);!os.is_dir(folder) {_=os.make_directory_all(folder)} - _=serial.file_write_atomic(path,transmute([]u8)strings.to_string(b)) -} - -// The hash of the bytes of a file, from hashes when its size and modification -// time are those it knows; 0 for a file that cannot be read. -@(private="file") -file_hash :: proc(hashes:^Source_Hashes,file:string) -> u64 { - info,stat_error:=os.stat(file,context.temp_allocator) - size,modified:=info.size,time.time_to_unix_nano(info.modification_time) - if hashes!=nil && stat_error==nil { - if known,found:=hashes.files[file];found && known.size==size && known.modified==modified {return known.hash} - } - bytes,_:=serial.file_read(file,context.temp_allocator,limit=MAX_SOURCE_BYTES) - hash:=base.hash64(bytes) - delete(bytes,context.temp_allocator) - if hashes!=nil && stat_error==nil && time.time_to_unix_nano(time.now())-modified>SETTLED_NANOSECONDS { - allocator:=hashes.files.allocator - if _,found:=hashes.files[file];!found {hashes.files[strings.clone(file,allocator)]={}} - hashes.files[file]={size,modified,hash} - hashes.changed=true - } - return hash -} - -// The sidecar of the source at source, in allocator. found is false, with an -// empty one, when the file is not there; a file that cannot be read as a -// sidecar is an error, not the defaults, so a source is never imported in a -// way nobody chose nor given new ids over the ones it had. -sidecar_load :: proc(source:string,allocator:=context.allocator) -> (sidecar:Sidecar,found:bool,error:Error) { - sidecar.outputs=make([dynamic]Output,allocator) - path:=strings.concatenate({source,SIDECAR_EXTENSION},context.temp_allocator) - // A source that names a device has no file next to it. - if unsafe_path(path) {return} - data,file_error:=serial.file_read(path,context.temp_allocator) - if file_error==.Not_Found {return} - if file_error!=.None {return sidecar,false,error_make(.Not_Found,"%s could not be read",path)} - parsed,valid:=sidecar_parse(string(data),allocator) - if !valid {return sidecar,false,error_make(.Malformed,"%s is not the settings and outputs of an import (version %d): fix it, or delete it and import its source again, which gives its assets new ids",path,asset.SIDECAR_VERSION)} - delete(sidecar.outputs) - return parsed,true,{} -} - -// Writes the sidecar of the source at source, unless the file already says the same. -sidecar_save :: proc(source:string,sidecar:Sidecar) -> bool { - path:=strings.concatenate({source,SIDECAR_EXTENSION},context.temp_allocator) - text:=sidecar_text(sidecar,context.temp_allocator) - if data,read_error:=os.read_entire_file(path,context.temp_allocator);read_error==nil && string(data)==text {return true} - return serial.file_write_atomic(path,transmute([]u8)text)==.None -} - -// The settings kept with a source, the defaults (found false) when it has no sidecar. -settings_load :: proc(source:string) -> (settings:Import_Settings,found:bool,error:Error) { - sidecar:Sidecar - sidecar,found,error=sidecar_load(source,context.temp_allocator) - return sidecar.settings,found,error -} - -// The asset the import of a source gave what the source names name, of kind; -// false when its sidecar lists none. -output_id :: proc(source:string,kind:asset.Fourcc,name:string) -> (id:base.Asset_Id,found:bool) { - sidecar,_,error:=sidecar_load(source,context.temp_allocator) - if error.code!=.None {return} - for output in sidecar.outputs { - if output.kind==kind && output.name==name {return output.id,true} - } - return -} diff --git a/cook/import/sound.odin b/cook/import/sound.odin index 0fa4d7102..cccd7f036 100644 --- a/cook/import/sound.odin +++ b/cook/import/sound.odin @@ -2,7 +2,6 @@ package importer import "core:fmt" import "core:mem" -import "core:os" import "core:path/filepath" import "core:strings" import "mirai:asset" @@ -75,7 +74,7 @@ i16_of :: proc(sample:f32) -> i16 { } // Imports a sound file as .sound. The sound is the root of the import: -// importing the file again writes the asset its sidecar lists (identity.odin). +// importing the file again gives the asset its package lists (identity.odin). import_sound :: proc(path:string,sound:Sound_Options={},options:Import_Options={},allocator:=context.allocator) -> (report:Import_Report,error:Error) { caller:=context.temp_allocator scratch:mem.Dynamic_Arena @@ -101,20 +100,20 @@ import_sound :: proc(path:string,sound:Sound_Options={},options:Import_Options={ report_init(&report,allocator) w:Writer ok:=writer_init(&w,path,"",options,&report,allocator) - defer asset.types_destroy(&w.types) + defer writer_close(&w) if ok { source:=w.source name:=filepath.stem(path) info:=import_info(source,name,0,u64(len(bytes)),base.hash64(bytes),u64(.Stream in desc.flags),import_version(0)) target:string - desc.id,target=root_take(&w,asset.KIND_SOUND,name,".sound") + output:int + desc.id,target,output=root_take(&w,asset.KIND_SOUND,name,".sound") desc.source=&info - existed:=os.exists(target) if save_error:=asset.sound_save(target,&w.types,desc,context.temp_allocator);save_error!=.None { ok=save_failed(&w,save_error,target) - } else { - record(&w,desc.id,asset.KIND_SOUND,name,target,source,existed) - ok=writer_finish(&w) + } else if ok=writer_put(&w,output,target);ok { + record(&w,output,name) + ok=writer_finish(&w,name,"sound") } } if !ok { diff --git a/cook/import/sound_test.odin b/cook/import/sound_test.odin index 2bcc4a06a..a11602351 100644 --- a/cook/import/sound_test.odin +++ b/cook/import/sound_test.odin @@ -11,6 +11,7 @@ import "mirai:asset" import audio "mirai:audio" import "mirai:std/base" import "mirai:std/serial" +import "mirai:store" @(private="file") SINE_OGG :: #load("../../audio/testdata/sine_mono.ogg") // 0.5 s of 440 Hz at 44.1 kHz, amplitude 0.5 @(private="file") THREE_OGG :: #load("../../audio/testdata/three_channels.ogg") @@ -46,8 +47,8 @@ lossless_same :: proc(t:^testing.T,view:asset.Sound_View,samples:[]i16,loc:=#cal } @(private="file") -sound_open :: proc(t:^testing.T,path:string,types:^asset.Types,a:^asset.Asset) -> (view:asset.Sound_View,ok:bool) { - if !testing.expect_value(t,asset.asset_load(a,path),asset.Error.None) {return} +sound_open :: proc(t:^testing.T,s:^store.Store,written:Written,types:^asset.Types,a:^asset.Asset) -> (view:asset.Sound_View,ok:bool) { + if !testing.expect_value(t,written_open(s,written,a),asset.Error.None) {return} error:asset.Error view,error=asset.sound_view(a,types) return view,testing.expect_value(t,error,asset.Error.None) @@ -56,7 +57,7 @@ sound_open :: proc(t:^testing.T,path:string,types:^asset.Types,a:^asset.Asset) - @(test) a_wav_and_an_ogg_become_sounds_decoded_or_streamed_by_their_length :: proc(t:^testing.T) { defer free_all(context.temp_allocator) - root,folder:=test_assets(t,"sound") + _,folder,s:=test_assets(t,"sound") if folder=="" {return} types:asset.Types;asset.types_init(&types);defer asset.types_destroy(&types) @@ -67,14 +68,14 @@ a_wav_and_an_ogg_become_sounds_decoded_or_streamed_by_their_length :: proc(t:^te path:=fmt.tprintf("%s/Step Left.wav",folder) _=os.write_entire_file(path,wav) - report,error:=import_sound(path,options={root=root}) + report,error:=import_sound(path,options={store=s}) testing.expect_value(t,error.code,Error_Code.None) testing.expect_value(t,len(report.assets),1) if len(report.assets)==1 { written:=report.assets[0] - testing.expect(t,written.kind==asset.KIND_SOUND && written.id!={} && strings.has_suffix(written.path,".sound") && strings.contains(written.path,"/.mirai/imported/") && written.bytes time.Duration { } @(private="file") -Large :: struct {path,out:string,model_bytes,asset_bytes:int,triangles,vertices:int} +Large :: struct {t:^testing.T,path,out:string,model_bytes,asset_bytes:int,triangles,vertices:int} @(test) bench_import_and_write_of_a_large_obj :: proc(t:^testing.T) { @@ -42,7 +42,7 @@ bench_import_and_write_of_a_large_obj :: proc(t:^testing.T) { fmt.sbprintf(&text,"f %d/%d/%d %d/%d/%d %d/%d/%d %d/%d/%d\n",a,a,a,b,b,b,c,c,c,d,d,d) } } - large:=Large{path=write_text(folder,"grid.obj",string(text.buf[:])),out=fmt.tprintf("%s/out",folder),model_bytes=len(text.buf)} + large:=Large{t=t,path=write_text(folder,"grid.obj",string(text.buf[:])),out=fmt.tprintf("%s/out",folder),model_bytes=len(text.buf)} read:=best_of(proc(data:rawptr) { large:=(^Large)(data) model,error:=importer.import_model(large.path,FORMATS[:]) @@ -58,7 +58,7 @@ bench_import_and_write_of_a_large_obj :: proc(t:^testing.T) { model,_:=importer.import_model(large.path,FORMATS[:]) defer importer.model_destroy(&model) // Every round builds the mesh: the one before left the same file. - report,error:=write_into(&model,large.out,{rebuild=true}) + report,error:=write_into(large.t,&model,large.out,{rebuild=true}) assert(error.code==.None) large.asset_bytes=report.assets[0].bytes importer.report_destroy(&report) diff --git a/cook/import/sources/cache_test.odin b/cook/import/sources/cache_test.odin index 75f323f32..4ca539e5d 100644 --- a/cook/import/sources/cache_test.odin +++ b/cook/import/sources/cache_test.odin @@ -6,9 +6,12 @@ import "core:os" import "core:strings" import "core:testing" import "mirai:asset" +import "mirai:cook/cache" import importer "mirai:cook/import" import "mirai:cook/prefab" +import "mirai:cook/textures" import "mirai:std/base" +import "mirai:store" @(test) sources_are_known_by_their_content :: proc(t:^testing.T) { @@ -31,18 +34,18 @@ sources_are_known_by_their_content :: proc(t:^testing.T) { testing.expectf(t,source_kind(path)==c.kind,"%s is a %v source, not %v",c.name,c.kind,source_kind(path)) } // A GLB is read as one whatever its name says. - report,error:=import_source(fmt.tprintf("%s/model.bin",folder),{world=prefab.import_world(t)}) + report,error:=import_test(t,fmt.tprintf("%s/model.bin",folder),{world=prefab.import_world(t)}) testing.expectf(t,error.code==.None && len(report.assets)>0,"%v %s",error.code,error.message) importer.report_destroy(&report) - _,refused:=import_source(fmt.tprintf("%s/mesh.txt",folder)) + _,refused:=import_test(t,fmt.tprintf("%s/mesh.txt",folder)) testing.expect(t,refused.code==.Unsupported && strings.contains(refused.message,"glTF, USD, FBX and OBJ"),refused.message) - _,missing:=import_source(fmt.tprintf("%s/none.obj",folder)) + _,missing:=import_test(t,fmt.tprintf("%s/none.obj",folder)) testing.expect_value(t,missing.code,importer.Error_Code.Not_Found) - // Every source has a sidecar, next to it, which keeps its id. - picture,picture_error:=import_source(fmt.tprintf("%s/picture.dat",folder)) - testing.expect(t,picture_error.code==.None && len(picture.assets)==1 && picture.assets[0].kind==asset.KIND_TEXTURE && os.exists(fmt.tprintf("%s/picture.dat.import",folder))) + // An image is known by its bytes, and nothing is written beside it. + picture,picture_error:=import_test(t,fmt.tprintf("%s/picture.dat",folder)) + testing.expect(t,picture_error.code==.None && len(picture.assets)==1 && picture.assets[0].kind==asset.KIND_TEXTURE) importer.report_destroy(&picture) - _,collection:=import_source(fmt.tprintf("%s/family.ttc",folder)) + _,collection:=import_test(t,fmt.tprintf("%s/family.ttc",folder)) testing.expect(t,collection.code==.Unsupported && strings.contains(collection.message,"collection"),collection.message) } @@ -59,15 +62,33 @@ kept_of :: proc(report:^importer.Import_Report) -> (meshes,textures,others:int) return } -// The path and id of the asset of kind a report wrote. +// The path and id of the asset of kind a report wrote: of an object of the +// store, a file of it beside the store named by its address. @(private="file") -written_of :: proc(report:^importer.Import_Report,kind:asset.Fourcc=asset.KIND_MESH) -> (path:string,id:base.Asset_Id) { +written_of :: proc(s:^store.Store,report:^importer.Import_Report,kind:asset.Fourcc=asset.KIND_MESH) -> (path:string,id:base.Asset_Id) { for written in report.assets { - if written.kind==kind {return strings.clone(written.path,context.temp_allocator),written.id} + if written.kind!=kind {continue} + if written.path!="" {return strings.clone(written.path,context.temp_allocator),written.id} + digits:[64]u8 + folder:=fmt.tprintf("%s-objects",s.root) + os.make_directory_all(folder) + path=fmt.tprintf("%s/%s",folder,store.hex(written.object,&digits)) + asset.object_file_write(s,written.object,path) + return path,written.id } return } +// Forgets every build the store names, as a store that lost them: an import +// finds no output to take. +@(private="file") +builds_forget :: proc(s:^store.Store) { + for namespace in ([]string{asset.REF_IMPORTS,asset.REF_IMPORTS_FAST,asset.REF_BUILDS}) { + refs,_:=store.ref_list(s,namespace,context.temp_allocator) + for ref in refs {store.ref_remove(s,namespace,ref.name)} + } +} + // Records in the file at path that it was built with settings, as one that // other encoders or another builder made; gives what it recorded. @(private="file") @@ -83,26 +104,30 @@ built_with :: proc(t:^testing.T,path:string,settings:u64) -> (was:u64) { return } -// What is current is decided by the name of a file of the cache: the key of -// what the import builds from now. A file under another key is taken as it is -// when what it records it was built from is what it would be built from now. +// What is current is a blob a build of the package has: the build of the key +// of what the import is made from now, else the one this machine chose for +// its previous manifest, when what the blob records it was built from is what +// it would be built from now. @(test) -an_import_keeps_what_its_cache_says_is_current_and_builds_again_what_changed :: proc(t:^testing.T) { +an_import_keeps_what_its_builds_say_is_current_and_builds_again_what_changed :: proc(t:^testing.T) { root,_:=test_project(t,"cache") if root=="" {return} source:=copy_in(root,DEMO,"gallery/vase.obj","gallery/vase.mtl","gallery/vase_color.png") world:=prefab.import_world(t) - Kept :: struct {meshes,textures,others:int,mesh,texture:string,id:base.Asset_Id} + s:=test_store(t,fmt.tprintf("%s/store",root)) + if s==nil {return} + Kept :: struct {meshes,textures,others:int,mesh,texture:string,id:base.Asset_Id,manifest,build:store.Hash} import_kept :: proc(t:^testing.T,source:string,options:importer.Import_Options,loc:=#caller_location) -> (k:Kept,ok:bool) { - report,error:=import_source(source,options) + report,error:=import_test(t,source,options) defer importer.report_destroy(&report) if !testing.expectf(t,error.code==.None,"%v %s",error.code,error.message,loc=loc) {return} k.meshes,k.textures,k.others=kept_of(&report) - k.mesh,k.id=written_of(&report) - k.texture,_=written_of(&report,asset.KIND_TEXTURE) + k.mesh,k.id=written_of(options.store,&report) + k.texture,_=written_of(options.store,&report,asset.KIND_TEXTURE) + k.manifest,k.build=report.manifest,report.build return k,true } - first,ok:=import_kept(t,source,{world=world}) + first,ok:=import_kept(t,source,{world=world,store=s}) if !ok {return} testing.expect(t,first.meshes==0 && first.textures==0 && first.others==0,"a first import builds everything") built,_:=os.read_entire_file(first.mesh,context.temp_allocator) @@ -114,33 +139,44 @@ an_import_keeps_what_its_cache_says_is_current_and_builds_again_what_changed :: // Nothing changed: the mesh and the texture are kept; materials and the prefab, cheap, are written again. again:Kept - again,ok=import_kept(t,source,{world=world}) + again,ok=import_kept(t,source,{world=world,store=s}) testing.expect(t,ok && again.meshes==1 && again.textures==1 && again.others==0 && again.mesh==first.mesh && same(first.mesh,built),"an unchanged source keeps its mesh and its texture") // Another code that imports it (that of a plugin that imports it whole) is another key, and - // the same mesh and image: their files are taken under it as they are, not built nor written. + // the same mesh and image: the blobs of the build before are taken, not built. coded:Kept - coded,ok=import_kept(t,source,{world=world,importer_code=1}) - testing.expect(t,ok && coded.meshes==1 && coded.textures==1 && coded.id==first.id && coded.mesh!=first.mesh && !os.exists(first.mesh) && !os.exists(first.texture),"a file under another key is taken") - testing.expect(t,same(coded.mesh,built) && same(coded.texture,texture),"as it is") + coded,ok=import_kept(t,source,{world=world,store=s,importer_code=1}) + testing.expect(t,ok && coded.meshes==1 && coded.textures==1 && coded.id==first.id && coded.mesh==first.mesh && coded.texture==first.texture,"a blob of another key is taken") // One that records other encoders (a texture) or another builder (a mesh) is built again. - texture_settings:=built_with(t,coded.texture,1) - mesh_settings:=built_with(t,coded.mesh,1) + b,read_error:=asset.build_get(s,coded.build,context.temp_allocator) + p,_:=asset.package_get(s,coded.manifest,context.temp_allocator) + if !testing.expect_value(t,read_error,asset.Error.None) {return} + recorded:[2]u64 + for &o in b.outputs { + if o.kind!=asset.KIND_MESH && o.kind!=asset.KIND_TEXTURE {continue} + other:=fmt.tprintf("%s/other",root) + asset.object_file_write(s,o.object,other) + recorded[int(o.kind==asset.KIND_TEXTURE)]=built_with(t,other,1) + o.object,_=asset.object_put_file(s,other) + } + forged,_:=asset.build_put(s,&b) + digits:[64]u8 + store.ref_set(s,asset.REF_BUILDS,store.hex(asset.build_select(coded.manifest,&p),&digits),forged) other:Kept - other,ok=import_kept(t,source,{world=world,importer_code=2}) + other,ok=import_kept(t,source,{world=world,store=s,importer_code=2}) testing.expect(t,ok && other.meshes==0 && other.textures==0,"what other encoders or another builder made is built again") - testing.expect(t,same(other.mesh,built) && same(other.texture,texture) && texture_settings!=1 && mesh_settings!=1,"to the bytes of now") + testing.expect(t,same(other.mesh,built) && same(other.texture,texture) && recorded[0]!=1 && recorded[1]!=1,"to the bytes of now") - // A setting of the mesh builds it again, under a new name, with the same id; the texture is + // A setting of the mesh builds it again, another blob with the same id; the texture is // built from the same image and settings and is taken. turned:Kept - turned,ok=import_kept(t,source,{world=world,front=.X,ignore_stored=true}) - testing.expect(t,ok && turned.meshes==0 && turned.textures==1 && turned.id==first.id && turned.mesh!=other.mesh && !os.exists(other.mesh),"a setting builds again, and the file of before goes") + turned,ok=import_kept(t,source,{world=world,store=s,front=.X,ignore_stored=true}) + testing.expect(t,ok && turned.meshes==0 && turned.textures==1 && turned.id==first.id && turned.mesh!=other.mesh && os.exists(other.mesh),"a setting builds again, and the blob of before stays in the store") // --rebuild keeps nothing; with the settings of the first import, it gives its bytes. rebuilt:Kept - rebuilt,ok=import_kept(t,source,{world=world,rebuild=true,ignore_stored=true}) + rebuilt,ok=import_kept(t,source,{world=world,store=s,rebuild=true,ignore_stored=true}) testing.expect(t,ok && rebuilt.meshes==0 && rebuilt.textures==0 && rebuilt.others==0 && rebuilt.mesh==first.mesh,"rebuild builds everything") testing.expect(t,same(first.mesh,built)) } @@ -150,25 +186,26 @@ an_import_takes_a_texture_that_another_import_built_from_the_texture_cache :: pr types:asset.Types;asset.types_init(&types);defer asset.types_destroy(&types) root,folder:=test_project(t,"texture-cache") if folder=="" {return} - cache:=fmt.tprintf("%s/cache",root) + kept:=test_store(t,fmt.tprintf("%s/store",root)) + if kept==nil {return} bytes:=png_bytes(16,16,{200,100,50}) // Each import is another image of the project, with its own id: the levels are what the cache gives. imported :: proc(t:^testing.T,types:^asset.Types,folder,name:string,bytes:[]u8,options:importer.Import_Options) -> (format:asset.Texture_Format,mips:int,texels:string,flags:asset.Texture_Flags) { image:=fmt.tprintf("%s/%s.png",folder,name) _=os.write_entire_file(image,bytes) - report,error:=import_source(image,options) + report,error:=import_test(t,image,options) defer importer.report_destroy(&report) if !testing.expectf(t,error.code==.None && len(report.assets)==1,"%v %s",error.code,error.message) {return} a:asset.Asset - if !testing.expect_value(t,asset.asset_load(&a,report.assets[0].path,context.temp_allocator),asset.Error.None) {return} + if !testing.expect_value(t,importer.written_open(options.store,report.assets[0],&a,context.temp_allocator),asset.Error.None) {return} view,view_error:=asset.texture_view(&a,types) if !testing.expect_value(t,view_error,asset.Error.None) {return} return view.info.format,len(view.mips),strings.clone(string(view.texels),context.temp_allocator),view.info.flags } - options:=importer.Import_Options{texture_cache=cache} + options:=importer.Import_Options{store=kept} format,mips,first,_:=imported(t,&types,folder,"a",bytes,options) testing.expect(t,format==.BC1 && mips==5) - entries:=cached_files(cache,".tex") + entries:=cached_refs(kept,textures.CACHE) if !testing.expect_value(t,len(entries),1) {return} // Another import of the same image has the same levels, made from the cache. @@ -176,13 +213,21 @@ an_import_takes_a_texture_that_another_import_built_from_the_texture_cache :: pr testing.expect(t,len(first)>0 && second==first) // What the cache holds is what the import takes: a texture put under its key is the one written. - kept:asset.Asset - testing.expect_value(t,asset.asset_load(&kept,entries[0].fullpath,context.temp_allocator),asset.Error.None) plain:[16*16*4]u8 - testing.expect_value(t,asset.texture_save(entries[0].fullpath,&types,{id=kept.blob.header.id,width=16,height=16,format=.RGBA8,mips={plain[:]}}),asset.Error.None) + planes:=[1][]u8{plain[:]} + flat:=textures.Built{width=16,height=16,format=.RGBA8,mips=planes[:]} + testing.expect(t,textures.cache_write({store=kept},cached_key(entries[0]),&types,&flat)) format,mips,_,_=imported(t,&types,folder,"c",bytes,options) testing.expect(t,format==.RGBA8 && mips==1) + // A store that misses asks its source before building: what a peer built is taken as it is. + peer:=options + peer.store=test_store(t,fmt.tprintf("%s/peer",root)) + peer.cache_source=cache.store_source(kept) + format,mips,_,_=imported(t,&types,folder,"p",bytes,peer) + testing.expect(t,format==.RGBA8 && mips==1,"the levels of the source, not built") + testing.expect_value(t,len(cached_refs(peer.store,textures.CACHE)),1) + // Rebuilding does not read the cache, and builds it again. rebuilt:=options rebuilt.rebuild=true @@ -196,7 +241,7 @@ an_import_takes_a_texture_that_another_import_built_from_the_texture_cache :: pr fast.fast_textures=true _,_,taken,_:=imported(t,&types,folder,"f",bytes,fast) testing.expect(t,taken==first,"the Final the cache holds") - fast.texture_cache=fmt.tprintf("%s/empty",root) + fast.store=test_store(t,fmt.tprintf("%s/empty",root)) flags:asset.Texture_Flags format,_,_,flags=imported(t,&types,folder,"g",bytes,fast) testing.expect(t,.Fast in flags && format==.BC1,"Fast, for the sync of a build to encode again") @@ -207,8 +252,10 @@ an_import_takes_a_mesh_that_another_import_built_from_the_mesh_cache :: proc(t:^ types:asset.Types;asset.types_init(&types);defer asset.types_destroy(&types) root,_:=test_project(t,"mesh-cache") if root=="" {return} + kept:=test_store(t,fmt.tprintf("%s/store",root)) + if kept==nil {return} source:=copy_in(root,DEMO,"gallery/vase.obj","gallery/vase.mtl") - options:=importer.Import_Options{world=prefab.import_world(t),mesh_cache=fmt.tprintf("%s/cache",root)} + options:=importer.Import_Options{world=prefab.import_world(t),store=kept} colliders :: proc(t:^testing.T,types:^asset.Types,path:string) -> int { a:asset.Asset if !testing.expect_value(t,asset.asset_load(&a,path,context.temp_allocator),asset.Error.None) {return -1} @@ -217,52 +264,54 @@ an_import_takes_a_mesh_that_another_import_built_from_the_mesh_cache :: proc(t:^ return len(view.colliders) } imported :: proc(t:^testing.T,source:string,options:importer.Import_Options) -> (mesh:string,ok:bool) { - report,error:=import_source(source,options) + report,error:=import_test(t,source,options) defer importer.report_destroy(&report) - mesh,_=written_of(&report) + mesh,_=written_of(options.store,&report) return mesh,testing.expectf(t,error.code==.None,"%v %s",error.code,error.message) } mesh_path,ok:=imported(t,source,options) if !ok {return} built,_:=os.read_entire_file(mesh_path,context.temp_allocator) - entries:=cached_files(options.mesh_cache,".mesh") + entries:=cached_refs(kept,importer.MESH_CACHE) if !testing.expect_value(t,len(entries),1) {return} - // A mesh whose file is gone is built from the cache, to the same bytes. - testing.expect(t,os.remove(mesh_path)==nil) + // A mesh whose build is gone is built from the cache, to the same bytes. + builds_forget(kept) if _,ok=imported(t,source,options);!ok {return} again,_:=os.read_entire_file(mesh_path,context.temp_allocator) testing.expect(t,len(built)>0 && string(again)==string(built),"what the cache gives is what building gives") // What the cache holds is what the import takes: a mesh put under its key, with a collider, is the one written. + key:=cached_key(entries[0]) cached:asset.Asset - if !testing.expect_value(t,asset.asset_load(&cached,entries[0].fullpath,context.temp_allocator),asset.Error.None) {return} + if !testing.expect(t,cache.load({store=kept},importer.MESH_CACHE,key,importer.MESH_CACHE_VERSION,&cached,context.temp_allocator)) {return} view,view_error:=asset.mesh_view(&cached,&types) if !testing.expect_value(t,view_error,asset.Error.None) {return} points:=[4][3]f32{{0,0,0},{1,0,0},{0,1,0},{0,0,1}} hull:=[1]asset.Collider{{kind=.Hull,rotation=1,point_count=4}} - desc:=asset.Mesh_Desc{id=cached.blob.header.id,geometry=view.geometry,slots=view.slots,colliders=hull[:],collision_points=points[:]} - testing.expect_value(t,asset.mesh_save(entries[0].fullpath,&types,desc),asset.Error.None) - testing.expect(t,os.remove(mesh_path)==nil) - if _,ok=imported(t,source,options);!ok {return} - testing.expect_value(t,colliders(t,&types,mesh_path),1) + doc:asset.Document + if !testing.expect_value(t,asset.mesh_document(&doc,&types,{id=cached.blob.header.id,geometry=view.geometry,slots=view.slots,colliders=hull[:],collision_points=points[:]},context.temp_allocator),asset.Error.None) {return} + hulled,bytes_error:=asset.document_bytes(&doc,context.temp_allocator) + testing.expect(t,bytes_error==.None && cache.put({store=kept},importer.MESH_CACHE,key,importer.MESH_CACHE_VERSION,hulled)) + builds_forget(kept) + hulled_path:string + if hulled_path,ok=imported(t,source,options);!ok {return} + testing.expect_value(t,colliders(t,&types,hulled_path),1) + + // A store that misses asks its source before building: what a peer built is taken as it is. + peer:=options + peer.store=test_store(t,fmt.tprintf("%s/peer",root)) + peer.cache_source=cache.store_source(kept) + peer_path:string + if peer_path,ok=imported(t,source,peer);!ok {return} + testing.expect_value(t,colliders(t,&types,peer_path),1) // Rebuilding does not read the cache, and builds it again. rebuilt:=options rebuilt.rebuild=true - if _,ok=imported(t,source,rebuilt);!ok {return} - testing.expect_value(t,colliders(t,&types,mesh_path),0) - testing.expect(t,os.remove(mesh_path)==nil) - if _,ok=imported(t,source,options);!ok {return} - last,_:=os.read_entire_file(mesh_path,context.temp_allocator) + rebuilt_path:string + if rebuilt_path,ok=imported(t,source,rebuilt);!ok {return} + testing.expect_value(t,colliders(t,&types,rebuilt_path),0) + last,_:=os.read_entire_file(rebuilt_path,context.temp_allocator) testing.expect(t,string(last)==string(built)) } - -// The files of a cache folder that hold what was built, of extension. -@(private="file") -cached_files :: proc(folder,extension:string) -> []os.File_Info { - entries,_:=os.read_all_directory_by_path(folder,context.temp_allocator) - files:=make([dynamic]os.File_Info,context.temp_allocator) - for entry in entries {if strings.has_suffix(entry.name,extension) {append(&files,entry)}} - return files[:] -} diff --git a/cook/import/sources/cli_test.odin b/cook/import/sources/cli_test.odin index 5778747c4..68bdc60cc 100644 --- a/cook/import/sources/cli_test.odin +++ b/cook/import/sources/cli_test.odin @@ -6,11 +6,13 @@ import "core:os" import "core:slice" import "core:strings" import "core:testing" +import "mirai:asset" import "mirai:cook/prefab" import importer "mirai:cook/import" import "mirai:project" import "mirai:std/base" import "mirai:std/subprocess" +import "mirai:store" // End to end: imports the fixtures with the command of `mirai import`, as the // import program of the engine runs it, and checks what it writes with @@ -22,10 +24,15 @@ import "mirai:std/subprocess" @(private="file") Result :: struct {code:int,out,err:string} +// mirai import with args, with --store store_root when it has any: the +// store of a test is never the machine's. @(private="file") -import_run :: proc(args:..string) -> Result { +import_run :: proc(store_root:string,args:..string) -> Result { output,errors:=strings.builder_make(context.temp_allocator),strings.builder_make(context.temp_allocator) - code:=command(args,strings.to_writer(&output),strings.to_writer(&errors),test_library) + all:=make([dynamic]string,context.temp_allocator) + append(&all,..args) + if len(args)>0 {append(&all,"--store",store_root)} + code:=command(all[:],strings.to_writer(&output),strings.to_writer(&errors),test_library) return {code,strings.to_string(output),strings.to_string(errors)} } @@ -53,6 +60,24 @@ reported_paths :: proc(out:string) -> []string { return paths[:] } +// Opens each object of the store at root a report names ("object
"), +// as the engine loads it: how many open. +objects_open :: proc(t:^testing.T,root:string,paths:[]string,loc:=#caller_location) -> (opened:int) { + s:=test_store(t,root,loc) + if s==nil {return} + defer store.close(s) + for path in paths { + if !strings.has_prefix(path,"object ") {continue} + address,ok:=store.parse_hex(path[len("object "):]) + a:asset.Asset + if testing.expectf(t,ok && asset.asset_open_object(&a,s,address,context.temp_allocator)==.None,"%s opens",path,loc=loc) { + asset.asset_close(&a) + opened+=1 + } + } + return +} + // The lines of a report that name an asset: id, kind, bytes and path. reported_assets :: proc(out:string) -> []string { lines:=make([dynamic]string,context.temp_allocator) @@ -83,118 +108,124 @@ mirai_import_writes_assets_that_mirai_asset_verifies :: proc(t:^testing.T) { if !testing.expectf(t,error==nil && state.exit_code==0,"the tool does not build: %v\n%s%s",error,string(stdout),string(stderr)) {return} } - // A model with a texture in the assets of a project: mesh, three materials, the image of the leaves and the prefab. + kept:=fmt.tprintf("%s/store",root_folder) + // A model with a texture in the assets of a project: ingested (mesh, three materials, the + // image of the leaves) and added to the project, with its documents and prefab beside it. plant:=copy_in(root_folder,DEMO,"gallery/plant.gltf","gallery/leaves_color.png") - first:=import_run(plant,"--uncompressed") + first:=import_run(kept,plant,"--uncompressed") testing.expectf(t,first.code==0 && first.err=="","import fails: %d\n%s%s",first.code,first.out,first.err) paths:=reported_paths(first.out) testing.expect_value(t,len(paths),6) - testing.expect(t,strings.contains(first.out,"6 assets\n") && !strings.contains(first.out,"warning"),first.out) + testing.expect(t,strings.contains(first.out,"6 assets\n") && strings.contains(first.out,"\npackage ") && !strings.contains(first.out,"warning"),first.out) for kind in ([]string{"mesh ","material ","texture ","scene "}) {testing.expectf(t,strings.contains(first.out,kind),"the report names a %s",kind)} + list,_,list_error:=asset.packages_load(root_folder,context.temp_allocator) + testing.expect(t,list_error==.None && len(list.packages)==1 && list.packages[0].folder=="gallery","the project pins the package under the folder of the source") // The scene is checked against the game of a project, which this folder is not. verify:=make([dynamic]string,context.temp_allocator) append(&verify,"asset","verify") - for path in paths {if !strings.has_suffix(path,".scene") {append(&verify,path)}} + for path in paths {if !strings.has_suffix(path,".scene") && !strings.has_prefix(path,"object ") {append(&verify,path)}} checked:=run(t,exe,..verify[:]) testing.expectf(t,checked.code==0,"asset verify fails: %s%s",checked.out,checked.err) - listing:=run(t,exe,"asset","ls",folder) - testing.expect(t,listing.code==0 && strings.contains(listing.out," gallery/plant/Leaves.mat") && strings.contains(listing.out," gallery/plant/Plant.mesh") && strings.contains(listing.out," gallery/plant/leaves_color.tex"),listing.out) + objects_open(t,kept,paths) // Importing again keeps every id, and the mesh and the texture are not built again. - again:=import_run(plant,"--uncompressed") + again:=import_run(kept,plant,"--uncompressed") testing.expect(t,again.code==0 && strings.contains(again.out,"6 assets, 2 were already up to date") && slice.equal(reported_paths(again.out),paths),again.out) - rebuilt:=import_run(plant,"--uncompressed","--rebuild") + rebuilt:=import_run(kept,plant,"--uncompressed","--rebuild") testing.expect(t,rebuilt.code==0 && slice.equal(reported_assets(rebuilt.out),reported_assets(first.out)),"--rebuild builds everything again, to the same bytes") // An image on its own. bricks:=copy_in(root_folder,DEMO,"surfaces/bricks/bricks_color.png") - texture:=import_run(bricks,"--srgb") - testing.expectf(t,texture.code==0 && strings.contains(texture.out,"texture") && strings.contains(texture.out,".tex"),"%d: %s%s",texture.code,texture.out,texture.err) - verify_texture:=make([dynamic]string,context.temp_allocator) - append(&verify_texture,"asset","verify") - append(&verify_texture,..reported_paths(texture.out)) - verified:=run(t,exe,..verify_texture[:]) - testing.expect_value(t,verified.code,0) - - // The sidecar of an import, next to its source, keeps its settings: the options change them, and importing again uses them. + texture:=import_run(kept,bricks,"--srgb") + testing.expectf(t,texture.code==0 && strings.contains(texture.out,"texture"),"%d: %s%s",texture.code,texture.out,texture.err) + testing.expect_value(t,objects_open(t,kept,reported_paths(texture.out)),1) + current:=import_run(kept,bricks) + testing.expectf(t,current.code==0 && strings.contains(current.out,"0 assets"),"a current source is not imported again: %s%s",current.out,current.err) + + // The package of a source keeps its settings: the options change them, and importing again uses them. copy:=copy_in(root_folder,DEMO,"gallery/vase.obj","gallery/vase.mtl","gallery/vase_color.png") - turned:=import_run(copy,"--front","x") - testing.expectf(t,turned.code==0 && strings.contains(turned.out," triangles") && !strings.contains(turned.out,"hull") && strings.contains(turned.out,"sidecar ") && strings.contains(turned.out,"vase.obj.import"),"%d: %s%s",turned.code,turned.out,turned.err) - sidecar:=fmt.tprintf("%s.import",copy) - stored,_:=os.read_entire_file(sidecar,context.temp_allocator) - testing.expect(t,strings.contains(string(stored),"\"front\": \"X\""),string(stored)) + turned:=import_run(kept,copy,"--front","x","--rebuild") + testing.expectf(t,turned.code==0 && strings.contains(turned.out," triangles") && !strings.contains(turned.out,"hull"),"%d: %s%s",turned.code,turned.out,turned.err) + stored,_,_:=importer.settings_load({store=test_store(t,kept)},copy) + testing.expect_value(t,stored.front,importer.Front.X) ids :: proc(out:string) -> []string { found:=make([dynamic]string,context.temp_allocator) for line in reported_assets(out) {append(&found,line[:32])} return found[:] } - mesh_path:string - for line in reported_assets(turned.out) {if strings.contains(line," mesh ") {mesh_path=reported_paths(line)[0]}} - with_front,_:=os.read_entire_file(mesh_path,context.temp_allocator) - repeated:=import_run(copy) - same,_:=os.read_entire_file(mesh_path,context.temp_allocator) - testing.expect(t,repeated.code==0 && slice.equal(reported_paths(repeated.out),reported_paths(turned.out)) && string(same)==string(with_front) && len(same)>0,"importing again without options gives the same assets") - bare:=import_run(copy,"--front","negative_z","--lods","none") + repeated:=import_run(kept,copy,"--rebuild") + testing.expect(t,repeated.code==0 && slice.equal(reported_paths(repeated.out),reported_paths(turned.out)),"importing again without options gives the same assets") + bare:=import_run(kept,copy,"--front","negative_z","--lods","none") testing.expect(t,bare.code==0 && slice.equal(ids(bare.out),ids(turned.out)) && strings.contains(bare.out," 3232 triangles"),bare.out) - _=os.write_entire_file(sidecar,"{\"lods\": \"Never\"}") - refused:=import_run(copy) - testing.expect(t,refused.code==1 && strings.contains(refused.err,"vase.obj.import"),refused.err) + + // A source outside any project: ingested into the library alone. + outside:=fmt.tprintf("%s/outside",root_folder) + os.make_directory_all(outside) + data,_:=os.read_entire_file(DEMO+"surfaces/bricks/bricks_color.png",context.temp_allocator) + loose:=fmt.tprintf("%s/loose.png",outside) + _=os.write_entire_file(loose,data) + alone:=import_run(kept,loose) + testing.expectf(t,alone.code==0 && strings.contains(alone.out,"texture") && strings.contains(alone.out,"package "),"%d: %s%s",alone.code,alone.out,alone.err) + list,_,_=asset.packages_load(root_folder,context.temp_allocator) + testing.expect_value(t,len(list.packages),3) + // Added to a project when it is named, under the folder given. + named:=import_run(kept,loose,"--project",root_folder,"--folder","loose") + testing.expectf(t,named.code==0,"%d: %s%s",named.code,named.out,named.err) + list,_,_=asset.packages_load(root_folder,context.temp_allocator) + testing.expect(t,len(list.packages)==4 && list.packages[3].folder=="loose") + + // A refresh of the project: everything is here, nothing is built. + refreshed:=import_run(kept,"--refresh",root_folder) + testing.expectf(t,refreshed.code==0 && strings.contains(refreshed.out,"4 packages, 0 ingested"),"%d: %s%s",refreshed.code,refreshed.out,refreshed.err) // Misuse and failures. - testing.expect_value(t,import_run(copy,"--lods","some").code,2) - testing.expect_value(t,import_run(copy,"--front","up").code,2) - testing.expect_value(t,import_run(bricks,"--front","x").code,2) - testing.expect_value(t,import_run().code,2) - testing.expect_value(t,import_run(copy,"--srgb").code,2) - testing.expect_value(t,import_run(copy,"--alpha-coverage","0.5").code,2) - testing.expect_value(t,import_run(bricks,"--alpha-coverage","1.5").code,2) - testing.expect_value(t,import_run(bricks,"--rdo-lambda","101").code,2) - testing.expect_value(t,import_run(bricks,"--rdo-lambda","-1").code,2) - testing.expect_value(t,import_run(copy,"--nope").code,2) - testing.expect_value(t,import_run(copy,"--out",folder).code,2) - missing:=import_run(fmt.tprintf("%s/missing.obj",folder)) + testing.expect_value(t,import_run(kept,copy,"--lods","some").code,2) + testing.expect_value(t,import_run(kept,copy,"--front","up").code,2) + testing.expect_value(t,import_run(kept,bricks,"--front","x").code,2) + testing.expect_value(t,import_run(kept).code,2) + testing.expect_value(t,import_run(kept,copy,"--srgb").code,2) + testing.expect_value(t,import_run(kept,copy,"--alpha-coverage","0.5").code,2) + testing.expect_value(t,import_run(kept,bricks,"--alpha-coverage","1.5").code,2) + testing.expect_value(t,import_run(kept,bricks,"--rdo-lambda","101").code,2) + testing.expect_value(t,import_run(kept,bricks,"--rdo-lambda","-1").code,2) + testing.expect_value(t,import_run(kept,copy,"--nope").code,2) + testing.expect_value(t,import_run(kept,copy,"--out",folder).code,2) + testing.expect_value(t,import_run(kept,loose,"--folder","loose").code,2) + testing.expect_value(t,import_run(kept,"--refresh",root_folder,copy).code,2) + missing:=import_run(kept,fmt.tprintf("%s/missing.obj",folder)) testing.expect(t,missing.code==1 && strings.contains(missing.err,"mirai import:") && strings.contains(missing.err,"missing.obj"),missing.err) - outside:=import_run(fmt.tprintf("%s/elsewhere.obj",root_folder)) - testing.expect(t,outside.code==1 && strings.contains(outside.err,"assets"),outside.err) write_text(folder,"broken.obj","v 0 0 0\nf 1 2 3\n") - broken:=import_run(fmt.tprintf("%s/broken.obj",folder)) + broken:=import_run(kept,fmt.tprintf("%s/broken.obj",folder)) testing.expect(t,broken.code==1 && strings.contains(broken.err,"outside"),broken.err) } -// mirai import does what the sync of the editor does with the same sources: -// a model and an image the sync imported again in a checkout without their -// outputs, their textures built Fast with no Final in the cache, are already -// up to date for the command, which builds nothing again. Final is what -// --final asks for, and builds them again. +// mirai import takes what the editor's import left of the same source: an +// image the editor ingested with Fast textures, with no Final in the store, +// is current for the command, which builds nothing again. Final +// is what --final asks for, and builds them again. @(test) -mirai_import_takes_what_the_sync_of_the_editor_left :: proc(t:^testing.T) { - root_folder,folder:=test_project(t,"cli-sync") +mirai_import_takes_what_the_import_of_the_editor_left :: proc(t:^testing.T) { + root_folder,folder:=test_project(t,"cli-fast") if folder=="" {return} context.user_ptr=prefab.import_world(t) - cache:=fmt.tprintf("%s/cache",root_folder) - plant:=copy_in(root_folder,DEMO,"gallery/plant.gltf","gallery/leaves_color.png") + kept:=fmt.tprintf("%s/store",root_folder) + s:=test_store(t,kept) + if s==nil {return} bricks:=copy_in(root_folder,DEMO,"surfaces/bricks/bricks_color.png") - // Their sidecars, which a checkout has, and none of their outputs. - for source in ([]string{plant,bricks}) { - first:=import_run(source,"--texture-cache",cache,"--mesh-cache",cache) - if !testing.expectf(t,first.code==0,"%s: %s",source,first.err) {return} - } - testing.expect(t,base.remove_all(project.imported_path(root_folder,context.temp_allocator))==nil) - testing.expect(t,base.remove_all(cache)==nil) - synced:=sync(root_folder,{texture_cache=cache,mesh_cache=cache,fast_textures=true,library=test_library}) - defer sync_report_destroy(&synced) - if !testing.expectf(t,synced.imported==2 && len(synced.failures)==0,"the sync imports both: %v",synced.failures) {return} - Case :: struct {source:string,args:[]string,kept:string} + // As the editor imports it: Fast, with the use it gives an image by its name. + settings:=importer.Import_Settings{image=image_use(bricks)} + report,_,error:=project_ingest(root_folder,bricks,"",{store=s,root=root_folder,plugins=test_library,fast_textures=true},&settings,context.temp_allocator) + if !testing.expectf(t,error.code==.None && !report.final,"%s",error.message) {return} + Case :: struct {source:string,args:[]string,built:bool} cases:=[]Case{ - {plant,{},"6 assets, 2 were already up to date"}, - {bricks,{},"1 assets, 1 were already up to date"}, - {bricks,{"--final"},"1 assets\n"}, + {bricks,{},false}, + {bricks,{"--final"},true}, } for c in cases { args:=make([dynamic]string,context.temp_allocator) - append(&args,c.source,"--texture-cache",cache,"--mesh-cache",cache) + append(&args,c.source) append(&args,..c.args) - done:=import_run(..args[:]) - testing.expectf(t,done.code==0 && strings.contains(done.out,c.kept),"%s %v: %s%s",c.source,c.args,done.out,done.err) + done:=import_run(kept,..args[:]) + testing.expectf(t,done.code==0 && (len(reported_paths(done.out))>0)==c.built,"%s %v: %s%s",c.source,c.args,done.out,done.err) } } diff --git a/cook/import/sources/cluster_test.odin b/cook/import/sources/cluster_test.odin index 7eb115793..5c369de63 100644 --- a/cook/import/sources/cluster_test.odin +++ b/cook/import/sources/cluster_test.odin @@ -19,7 +19,7 @@ demo_level0 :: proc(t:^testing.T,folder,relative:string,file:^asset.Asset,types: source,name:=filepath.dir(relative),filepath.base(relative) copied:=copy_in(folder,DEMO,source) if filepath.ext(source)==".obj" {copy_in(folder,DEMO,"gallery/vase.mtl","gallery/vase_color.png")} - report,import_error:=import_source(copied,{world=prefab.import_world(t),textures=.Uncompressed}) + report,import_error:=import_test(t,copied,{world=prefab.import_world(t),textures=.Uncompressed}) if !testing.expectf(t,import_error.code==.None,"%s: %s",source,import_error.message) {return} defer importer.report_destroy(&report) written,found:=written_of(&report,asset.KIND_MESH,name) diff --git a/cook/import/sources/command.odin b/cook/import/sources/command.odin index fc4091e25..935bd2036 100644 --- a/cook/import/sources/command.odin +++ b/cook/import/sources/command.odin @@ -3,45 +3,55 @@ package sources import "base:runtime" import "core:fmt" import "core:io" +import "core:os" import "core:path/filepath" import "core:strconv" import "core:strings" import "core:time" import "mirai:asset" +import "mirai:cook/cache" import importer "mirai:cook/import" import "mirai:cook/textures" import "mirai:project" import "mirai:std/base" import "mirai:std/jobs" +import "mirai:store" // The command mirai import, which the import program of the engine runs: it -// turns a model (OBJ, glTF, GLB, USD, FBX, or a format a plugin of the +// ingests a model (OBJ, glTF, GLB, USD, FBX, or a format a plugin of the // project adds), an image, a sound (WAV, Ogg Vorbis), a font (TrueType, // OpenType), a manifest of prompts (.prompts.json) or a source a plugin -// of the project imports whole in the assets folder of a project or a plugin -// into assets with import_source, as the editor does: a model also gets the -// prefab of its hierarchy. -// A source is imported with the settings its sidecar keeps (.import), -// changed by the options given; the result is written back there with the id -// of each asset, so importing again without options gives the same assets. +// of the project imports whole, from wherever it is, into the library of the +// machine store (ingest), and adds it to a project (project_add): the one of +// --project, else the one whose assets hold the file, else none. Its cache +// is listed under --folder of the project's assets (else the folder of the +// file under them, else the top), and its documents and the prefab of a +// model are written there. +// A source is ingested with the settings its package keeps, changed by the +// options given; ingesting again without options gives the same assets, and +// a source whose package is current is not imported again, unless --rebuild. // --rdo-lambda 0 encodes the blocks of textures for the least error and a -// higher lambda for smaller files. An asset whose file already is the one the -// import gives is kept as it is, unless --rebuild. A document the import -// writes over that changed since it last wrote it stops the import, which -// names it, unless --discard-changes; a sync leaves it as it is and warns. An -// image imported for the first time without --srgb, --data or --normal holds +// higher lambda for smaller files. A document the import writes over in the +// project that changed since it last wrote it stops the import, which names +// it, unless --discard-changes; a refresh leaves it as it is and warns. An +// image ingested for the first time without --srgb, --data or --normal holds // what its name says (image_use). Only models and images have settings. -// --sync , once per project or plugin, imports instead every source -// under it whose outputs in its imported folder are missing or stale (sync): -// what a checkout needs before it runs. Textures are built Fast, as the -// editor imports them, unless the texture cache keeps them Final, and outputs -// built either way are current: an import takes what the sync of the editor -// left. --final builds them Final: what a build that cooks and the package of -// the engine take. +// --refresh , once per project or plugin, instead brings every package +// its packages.mirai pins to a build this engine reads (project_refresh): what +// a checkout needs before it runs. It removes nothing and builds only from a +// source on this machine; a package nothing here gives is a failure that names +// it. Textures are built Fast, as the editor imports them, unless the texture +// cache keeps them Final, and a Fast build is taken; --final builds them Final +// and takes only a Final build: what a build that cooks and the package of the +// engine take. --from names a second store a build this store lacks +// is copied from before it is built (cache.store_source), as a peer gives it; +// an installed engine takes the store of the builds of its plugins +// (project.ENGINE_STORE) unless another is given. The store is that of the +// machine, or the one at --store. // The report goes to output, one asset per line: id, kind, bytes and path, and -// under it, indented, what the asset holds; then the settings file and the +// under it, indented, what the asset holds; then the package and the // warnings. How far a long import is goes to errors while it runs, and a -// failure as "mirai import: : ". A sync says each source it +// failure as "mirai import: : ". A refresh says each source it // imports, and how far it is, in output. // --profile adds the seconds each zone of the import took, summed over the // threads that cooked it and on the thread that imports, and the time no zone @@ -50,7 +60,7 @@ import "mirai:std/jobs" // Exit codes: 0 done, 1 failed, 2 misuse. @(private="file") -IMPORT_USAGE :: "Usage: mirai import [--uncompressed|--blocks] [--rdo-lambda L] [--srgb|--normal|--normal-green-down|--data] [--alpha-coverage C] [--lods auto|none] [--parts auto|all|none] [--front auto|negative_z|z|x|negative_x] [--rig source|humanoid|fox|horse|bird|dragon|kaiju|shark|snake|spider] [--rig-crotch M] [--rig-neck M] [--skeleton ] [--rebuild] [--discard-changes] [--final] [--texture-cache ] [--mesh-cache ] [--profile] [--release]" +IMPORT_USAGE :: "Usage: mirai import [--uncompressed|--blocks] [--rdo-lambda L] [--srgb|--normal|--normal-green-down|--data] [--alpha-coverage C] [--lods auto|none] [--parts auto|all|none] [--front auto|negative_z|z|x|negative_x] [--rig source|humanoid|fox|horse|bird|dragon|kaiju|shark|snake|spider] [--rig-crotch M] [--rig-neck M] [--skeleton ] [--project ] [--folder ] [--rebuild] [--discard-changes] [--final] [--store ] [--profile] [--release] | mirai import --refresh [--refresh ...] [--final] [--store ] [--from ] [--release]" // library reads the formats and the importers of whole sources the plugins of // a project add and the types of its game (in the import program of the @@ -58,8 +68,13 @@ IMPORT_USAGE :: "Usage: mirai import [--uncompressed|--blocks] [--rdo-lam command :: proc(args:[]string,output,errors:io.Writer,library:Library=nil) -> int { file:string options:=importer.Import_Options{fast_textures=true} - roots:=make([dynamic]string,context.temp_allocator) // of --sync + roots:=make([dynamic]string,context.temp_allocator) // of --refresh + root:string // of --project + folder:string // of --folder + foldered:bool profiled:bool + store_root:string // of the store whose caches keep what imports build; "": that of the machine + from_root:string // of --from // What the options change of the settings the source has. Change :: enum {Textures,Rdo_Lambda,Image,Alpha_Coverage,Lods,Parts,Front,Rig,Rig_Crotch,Rig_Neck,Skeleton} given:importer.Import_Settings @@ -73,9 +88,11 @@ command :: proc(args:[]string,output,errors:io.Writer,library:Library=nil) -> in for ;at in given.alpha_coverage=f32(cutoff) changed+={.Alpha_Coverage} } - case "--texture-cache": options.texture_cache,ok=value(args,&at) - case "--mesh-cache": options.mesh_cache,ok=value(args,&at) + case "--store": store_root,ok=value(args,&at) + case "--from": from_root,ok=value(args,&at) case "--profile": profiled=true case "--rebuild": options.rebuild=true case "--discard-changes": options.changed=.Discard @@ -163,6 +180,8 @@ command :: proc(args:[]string,output,errors:io.Writer,library:Library=nil) -> in } if !ok {fmt.wprintln(errors,IMPORT_USAGE);return 2} } + // Nothing to import is a misuse, whatever the store says. + if len(roots)==0 && file=="" {fmt.wprintln(errors,IMPORT_USAGE);return 2} js:jobs.Job_System if !jobs.system_init(&js,{threads=jobs.THREADS_AUTO,allocator=runtime.heap_allocator()}) { fmt.wprintln(errors,"mirai import: the threads to cook on could not be started") @@ -170,95 +189,170 @@ command :: proc(args:[]string,output,errors:io.Writer,library:Library=nil) -> in } defer jobs.system_destroy(&js) options.worker={system=&js,worker=jobs.OWNER} + if store_root=="" {store_root=store.default_root(context.temp_allocator)} + kept:store.Store + if store_root!="" { + opened:store.Error + if kept,opened=store.open(store_root,context.temp_allocator);opened!=.None { + fmt.wprintfln(errors,"mirai import: the store at %s could not be opened (%v): pass another with --store",store_root,opened) + return 1 + } + options.store=&kept + } + library_options:=Library_Options{store=options.store,fast_textures=options.fast_textures,rebuild=options.rebuild,changed=options.changed,plugins=library,worker=options.worker} + if options.store==nil {fmt.wprintln(errors,"mirai import: this machine has no folder of user data for its store: pass one with --store");return 1} if len(roots)>0 { - return sync_command(roots[:],options,library,output,errors) + if file!="" || root!="" || foldered || changed!={} || options.changed!=.Refuse {fmt.wprintln(errors,IMPORT_USAGE);return 2} + if from_root=="" { + if engine:=project.engine_root_find("",context.temp_allocator);project.engine_installed(engine) { + if bundled:=project.path(engine,project.ENGINE_STORE,allocator=context.temp_allocator);os.is_dir(bundled) {from_root=bundled} + } + } + from:store.Store + if from_root!="" { + opened:store.Error + if from,opened=store.open(from_root,context.temp_allocator);opened!=.None { + fmt.wprintfln(errors,"mirai import: the store at %s could not be opened (%v): pass another with --from",from_root,opened) + return 1 + } + library_options.source=cache.store_source(&from) + } + return refresh_command(roots[:],library_options,output,errors) } + if from_root!="" {fmt.wprintln(errors,IMPORT_USAGE);return 2} if file=="" {fmt.wprintln(errors,IMPORT_USAGE);return 2} - root,in_assets:=source_root(file,context.temp_allocator) - if !in_assets {fmt.wprintfln(errors,"mirai import: %s is in no assets folder of a project or a plugin: move it beside what it makes, under assets/",file);return 1} - options.root=root - if options.texture_cache=="" {options.texture_cache=project.content_cache_path(root,project.TEXTURE_CACHE,context.temp_allocator)} - if options.mesh_cache=="" {options.mesh_cache=project.content_cache_path(root,project.MESH_CACHE,context.temp_allocator)} + origin:=importer.path_key(file) + // The project it is added to, and where under its assets. + if root=="" { + if inside,in_assets:=importer.source_root(origin,context.temp_allocator);in_assets { + root=inside + if !foldered { + assets:=project.folder_path(root,.Assets,context.temp_allocator) + folder=strings.trim_prefix(filepath.dir(origin),assets) + folder=strings.trim_prefix(folder,"/") + } + } + } + if root!="" { + root=project.path_absolute(root,context.temp_allocator) + library_options.root=root + } else if foldered { + fmt.wprintln(errors,"mirai import: --folder is a folder of the assets of a project: give it with --project") + return 2 + } formats:=FORMATS[:] - kind:=source_kind(file) + kind:=source_kind(origin) plugins:Library_Imports defer if plugins.release!=nil {plugins.release(plugins.state)} - if (kind==.None || kind==.Model) && library!=nil { - plugins=library(root,context.temp_allocator) - options.world=plugins.world + if (kind==.None || kind==.Model) && library!=nil && root!="" { + plugins=library(root,options.store,context.temp_allocator) all:=make([dynamic]importer.Format,0,len(FORMATS)+len(plugins.formats),context.temp_allocator) append(&all,..FORMATS[:]) append(&all,..plugins.formats) formats=all[:] - kind=source_kind(file,formats) - if _,claimed:=whole_find(plugins.wholes,file);claimed {kind=.None} + kind=source_kind(origin,formats) + if _,claimed:=whole_find(plugins.wholes,origin);claimed {kind=.None} } if changed&{.Image,.Alpha_Coverage}!={} && kind!=.Image {fmt.wprintln(errors,"mirai import: --srgb, --normal, --normal-green-down, --data and --alpha-coverage apply only to images");return 2} if changed&{.Lods,.Parts,.Front,.Rig,.Rig_Crotch,.Rig_Neck,.Skeleton}!={} && kind!=.Model {fmt.wprintln(errors,"mirai import: --lods, --parts, --front, --skeleton and the --rig options apply only to models");return 2} + // The settings the package keeps, changed by the options: given only when they change it. + settings:^importer.Import_Settings if kind==.Model || kind==.Image { - stored,found,settings_error:=importer.settings_load(file) + stored,found,settings_error:=importer.settings_load(options,origin) if settings_error.code!=.None { fmt.wprintfln(errors,"mirai import: %s: %s",file,settings_error.message) return 1 } - options.settings,options.ignore_stored=stored,true + chosen:=stored // An image with no settings yet holds what its name says, as in the editor. - if kind==.Image && !found {options.image=image_use(file)} - if .Textures in changed {options.textures=given.textures} - if .Rdo_Lambda in changed {options.rdo_lambda=given.rdo_lambda} - if .Image in changed {options.image=given.image} - if .Alpha_Coverage in changed {options.alpha_coverage=given.alpha_coverage} - if .Lods in changed {options.lods=given.lods} - if .Parts in changed {options.parts=given.parts} - if .Front in changed {options.front=given.front} - if .Rig in changed {options.rig=given.rig} - if .Rig_Crotch in changed {options.rig_crotch=given.rig_crotch} - if .Rig_Neck in changed {options.rig_neck=given.rig_neck} - if .Skeleton in changed {options.skeleton=given.skeleton} + if kind==.Image && !found {chosen.image=image_use(origin)} + if .Textures in changed {chosen.textures=given.textures} + if .Rdo_Lambda in changed {chosen.rdo_lambda=given.rdo_lambda} + if .Image in changed {chosen.image=given.image} + if .Alpha_Coverage in changed {chosen.alpha_coverage=given.alpha_coverage} + if .Lods in changed {chosen.lods=given.lods} + if .Parts in changed {chosen.parts=given.parts} + if .Front in changed {chosen.front=given.front} + if .Rig in changed {chosen.rig=given.rig} + if .Rig_Crotch in changed {chosen.rig_crotch=given.rig_crotch} + if .Rig_Neck in changed {chosen.rig_neck=given.rig_neck} + if .Skeleton in changed {chosen.skeleton=given.skeleton} // The heights tell the fit of a person where the body is. - if options.rig!=.Humanoid { + if chosen.rig!=.Humanoid { if changed&{.Rig_Crotch,.Rig_Neck}!={} {fmt.wprintln(errors,"mirai import: --rig-crotch and --rig-neck go with --rig humanoid");return 2} - options.rig_crotch,options.rig_neck=0,0 + chosen.rig_crotch,chosen.rig_neck=0,0 + } + if chosen!=stored || !found { + settings=new(importer.Import_Settings,context.temp_allocator) + settings^=chosen } } profile:base.Profile started:=time.tick_now() if profiled && !importer.profile_begin(&profile,&options) {fmt.wprintln(errors,"mirai import: no memory for --profile");return 1} defer if profiled {profile_print(&profile,started,output);base.profile_destroy(&profile)} - progress:importer.Progress - importer.progress_begin(&progress,errors,filepath.base(file)) - options.progress=&progress - report,error:=import_source(file,options,formats,plugins.wholes) + library_options.worker=options.worker + library_options.progress=errors + if root=="" { + report,error:=ingest(origin,library_options,settings) + if error.code!=.None { + fmt.wprintfln(errors,"mirai import: %s: %s",file,error.message) + return 1 + } + defer importer.report_destroy(&report) + ingest_print(&report,output) + return 0 + } + report,added,error:=project_ingest(root,origin,folder,library_options,settings) + defer library_report_destroy(&added) if error.code!=.None { fmt.wprintfln(errors,"mirai import: %s: %s",file,error.message) return 1 } defer importer.report_destroy(&report) + ingest_print(&report,output) + return library_print(&added,root,output,errors) +} + +// The assets an ingest wrote, its warnings and its package. +@(private="file") +ingest_print :: proc(report:^importer.Import_Report,output:io.Writer) { written_print(output,report.assets[:]) - if report.saved!="" {fmt.wprintfln(output,"sidecar %s",report.saved)} for warning in report.warnings {fmt.wprintfln(output,"warning: %s",warning)} - return 0 + id:[base.ASSET_ID_TEXT]u8 + digits:[64]u8 + fmt.wprintfln(output,"package %s %s",base.asset_id_format(report.package_id,id[:]),store.hex(report.manifest,&digits)) } -// --sync: every root in turn, one line each with what its sync did. +// --refresh: every root in turn, one line each with what its refresh did. @(private="file") -sync_command :: proc(roots:[]string,options:importer.Import_Options,library:Library,output,errors:io.Writer) -> int { +refresh_command :: proc(roots:[]string,options:Library_Options,output,errors:io.Writer) -> int { code:=0 for root in roots { started:=time.tick_now() - report:=sync(root,{texture_cache=options.texture_cache,mesh_cache=options.mesh_cache,rebuild=options.rebuild,fast_textures=options.fast_textures,worker=options.worker,library=library,progress=output}) - defer sync_report_destroy(&report) - for warning in report.warnings {fmt.wprintfln(output,"warning: %s",warning)} - for failure in report.failures {fmt.wprintfln(errors,"mirai import: %s",failure)} - if len(report.failures)>0 {code=1} - if report.sources>0 || report.removed>0 { - fmt.wprintfln(output,"%s: %d of %d imported sources built again, %d stale files removed (%.1f s).",root,report.imported,report.sources,report.removed,time.duration_seconds(time.tick_since(started))) + options:=options + options.root=project.path_absolute(root,context.temp_allocator) + options.progress=output + report:=project_refresh(root,options) + defer library_report_destroy(&report) + if library_print(&report,root,output,errors)!=0 {code=1} + if report.packages>0 { + fmt.wprintfln(output,"%s: %d packages, %d ingested, %d builds fetched (%.1f s).",root,report.packages,report.ingested,report.fetched,time.duration_seconds(time.tick_since(started))) } } return code } +// The warnings, failures and missing packages of a report; 1 with any failure or package missing. +@(private="file") +library_print :: proc(report:^Library_Report,root:string,output,errors:io.Writer) -> int { + for warning in report.warnings {fmt.wprintfln(output,"warning: %s",warning)} + for failure in report.failures {fmt.wprintfln(errors,"mirai import: %s",failure)} + for missing in report.missing {fmt.wprintfln(errors,"mirai import: %s: %s",root,missing)} + return 1 if len(report.failures)>0 || len(report.missing)>0 else 0 +} + // The zones of a profiled import (importer.profile_totals), each with its // seconds summed over the threads and those on the thread that imports, and // the wall time no zone of that thread covers. @@ -273,7 +367,8 @@ profile_print :: proc(profile:^base.Profile,started:time.Tick,output:io.Writer) if lost>0 {fmt.wprintfln(output," %d zones were lost: the rings hold %d each",lost,importer.PROFILE_ZONES)} } -// One line per asset written (id, kind, bytes and path) with what it holds +// One line per asset written (id, kind, bytes, and its path or its object of +// the store) with what it holds // under it, and how many there are. @(private="file") written_print :: proc(output:io.Writer,assets:[]importer.Written) { @@ -281,7 +376,9 @@ written_print :: proc(output:io.Writer,assets:[]importer.Written) { kept:=0 for written in assets { info,_:=asset.kind_info(written.kind) - fmt.wprintfln(output,"%s %-8s %d %s",base.asset_id_format(written.id,buffer[:]),info.name,written.bytes,written.path) + digits:[64]u8 + at:=written.path if written.path!="" else fmt.tprintf("object %s",store.hex(written.object,&digits)) + fmt.wprintfln(output,"%s %-8s %d %s",base.asset_id_format(written.id,buffer[:]),info.name,written.bytes,at) if written.detail!="" {fmt.wprintfln(output," %s",written.detail)} if written.kept {kept+=1} } diff --git a/cook/import/sources/cook_test.odin b/cook/import/sources/cook_test.odin index 95cdb4d38..1e1612b2d 100644 --- a/cook/import/sources/cook_test.odin +++ b/cook/import/sources/cook_test.odin @@ -5,14 +5,19 @@ import "core:os" import "core:slice" import "core:strings" import "core:testing" +import "compressor:data" import "mirai:asset" +import "mirai:cook/cache" import importer "mirai:cook/import" import "mirai:cook/build" +import "mirai:cook/prefab" +import "mirai:project" import "mirai:std/base" +import "mirai:store" -// A texture the editor imported Fast ships Final: the sync of a build that +// A texture the editor imported Fast ships Final: the refresh of a build that // cooks builds it again from its source with the id it has, a cook without it -// says so, and the sync of a build that does not cook takes either as it is. +// says so, and the refresh of a build that does not cook takes either as it is. @(test) a_cook_ships_a_fast_texture_final_from_its_source :: proc(t:^testing.T) { types:asset.Types;asset.types_init(&types);defer asset.types_destroy(&types) @@ -31,12 +36,18 @@ a_cook_ships_a_fast_texture_final_from_its_source :: proc(t:^testing.T) { } source,final:=fmt.tprintf("%s/noise.tga",assets),fmt.tprintf("%s/final/noise.tga",assets) testing.expect(t,os.write_entire_file(source,image)==nil && os.write_entire_file(final,image)==nil) - imported :: proc(t:^testing.T,types:^asset.Types,source:string,options:importer.Import_Options) -> (id:base.Asset_Id,flags:asset.Texture_Flags,blocks:[]u8,path:string) { - report,error:=import_source(source,options) + s:=test_store(t,fmt.tprintf("%s/store",folder)) + if s==nil {return} + fast:=Library_Options{store=s,root=folder,fast_textures=true} + final_options:=fast + final_options.fast_textures=false + imported :: proc(t:^testing.T,types:^asset.Types,source:string,options:Library_Options) -> (id:base.Asset_Id,flags:asset.Texture_Flags,blocks:[]u8,manifest:store.Hash) { + report,error:=ingest(source,options) defer importer.report_destroy(&report) if !testing.expectf(t,error.code==.None && len(report.assets)==1,"%v %s",error.code,error.message) {return} + objects_written(options.store,&report) id,flags,blocks=level_of(t,types,report.assets[0].path) - return id,flags,blocks,strings.clone(report.assets[0].path,context.temp_allocator) + return id,flags,blocks,report.manifest } level_of :: proc(t:^testing.T,types:^asset.Types,path:string) -> (id:base.Asset_Id,flags:asset.Texture_Flags,blocks:[]u8) { a:asset.Asset @@ -45,31 +56,270 @@ a_cook_ships_a_fast_texture_final_from_its_source :: proc(t:^testing.T) { if !testing.expect_value(t,error,asset.Error.None) {return} return a.blob.header.id,view.info.flags,slice.clone(asset.texture_mip(view,0),context.temp_allocator) } - fast_id,fast_flags,fast_blocks,fast_path:=imported(t,&types,source,{fast_textures=true}) - _,final_flags,final_blocks,_:=imported(t,&types,final,{}) + fast_id,fast_flags,fast_blocks,manifest:=imported(t,&types,source,fast) + _,final_flags,final_blocks,_:=imported(t,&types,final,final_options) testing.expect(t,.Fast in fast_flags && .Fast not_in final_flags && !slice.equal(fast_blocks,final_blocks)) - manifest:=fmt.tprintf("%s/%s",folder,build.COOK_FILE) - testing.expect(t,os.write_entire_file(manifest,`{"roots":["noise.tex"]}`)==nil) + added:=project_add(folder,manifest,"",nil,fast,context.temp_allocator) + testing.expectf(t,len(added.failures)==0 && len(added.missing)==0,"%v",added) - testing.expect(t,strings.contains(fast_path,"/.mirai/imported/"),"the texture of a source of the project is of its imported folder, not of its assets") - - // The cook reads the folder as it is: it says who builds the texture Final. - report:=build.cook_assets({assets=assets,output=fmt.tprintf("%s/program",folder),manifest=manifest,root=folder,allocator=context.temp_allocator}) - testing.expectf(t,report.error==.Source_Failed && strings.contains(report.detail,"assets/noise.tga") && strings.contains(report.detail,"mirai build game"),"%v %s",report.error,report.detail) - - caches:=Sync_Options{texture_cache=fmt.tprintf("%s/cache/textures",folder),mesh_cache=fmt.tprintf("%s/cache/meshes",folder)} - fast:=caches - fast.fast_textures=true - kept:=sync(folder,fast,context.temp_allocator) - testing.expectf(t,kept.imported==0 && kept.removed==0 && len(kept.failures)==0,"Fast is current for a build that does not cook: %v",kept) - - // The sync of a build that cooks does, with the id the texture has. - synced:=sync(folder,caches,context.temp_allocator) - testing.expectf(t,synced.imported==1 && len(synced.failures)==0,"%v",synced) - kept=sync(folder,fast,context.temp_allocator) - testing.expectf(t,kept.imported==0 && kept.removed==0 && len(kept.failures)==0,"Final is current for a build that does not cook: %v",kept) - report=build.cook_assets({assets=assets,output=fmt.tprintf("%s/game",folder),manifest=manifest,root=folder,allocator=context.temp_allocator}) + // The refresh of a build that does not cook takes Fast; that of one that cooks builds Final, with the id it has. + kept:=project_refresh(folder,fast,context.temp_allocator) + testing.expectf(t,kept.ingested==0 && len(kept.failures)==0 && len(kept.missing)==0,"Fast is current for a build that does not cook: %v",kept) + cook_manifest:=fmt.tprintf("%s/%s",folder,build.COOK_FILE) + testing.expect(t,os.write_entire_file(cook_manifest,`{"roots":["noise.tex"]}`)==nil) + // The cook reads the project as it is: it says who builds the texture Final. + report:=build.cook_assets({assets=assets,output=fmt.tprintf("%s/program",folder),manifest=cook_manifest,root=folder,store=s,allocator=context.temp_allocator}) + testing.expectf(t,report.error==.Source_Failed && strings.contains(report.detail,"noise") && strings.contains(report.detail,"mirai build game"),"%v %s",report.error,report.detail) + refreshed:=project_refresh(folder,final_options,context.temp_allocator) + testing.expectf(t,refreshed.ingested==1 && len(refreshed.failures)==0 && len(refreshed.missing)==0,"%v",refreshed) + kept=project_refresh(folder,fast,context.temp_allocator) + testing.expectf(t,kept.ingested==0 && len(kept.failures)==0,"Final is current for a build that does not cook: %v",kept) + report=build.cook_assets({assets=assets,output=fmt.tprintf("%s/game",folder),manifest=cook_manifest,root=folder,store=s,allocator=context.temp_allocator}) testing.expectf(t,report.error==.None,"%v %s",report.error,report.detail) shipped_id,shipped_flags,shipped_blocks:=level_of(t,&types,fmt.tprintf("%s/game/noise.tex",folder)) testing.expect(t,shipped_id==fast_id && .Fast not_in shipped_flags && slice.equal(shipped_blocks,final_blocks),"Final, with the id it had") } + +// A game cooks what its project and its plugins pin from the store, and what a +// package was imported from inside a root stays behind. A peer with the +// project and none of its sources fails the cook by the name of each package +// it has no build of; once it fetched the builds alone, it ships the same +// files, which load as the runtime loads them, from the index of the cook. +@(test) +a_peer_with_only_fetched_builds_cooks_the_game_its_sources_cook :: proc(t:^testing.T) { + top:=base.test_folder(t,"cook-library") + if top=="" {return} + defer free_all(context.temp_allocator) + context.user_ptr=prefab.import_world(t) + game:=fmt.tprintf("%s/game",top) + plugin:=fmt.tprintf("%s/plugins/brushes",game) + os.make_directory_all(fmt.tprintf("%s/assets/props",game)) + os.make_directory_all(fmt.tprintf("%s/assets",plugin)) + box:=box_source(fmt.tprintf("%s/assets/props",game),"box.gltf","Wood","Paint") + brush:=fmt.tprintf("%s/assets/soft.png",plugin) + testing.expect(t,os.write_entire_file(brush,png_bytes(8,8,{200,40,40}))==nil) + here:=test_store(t,fmt.tprintf("%s/here",top)) + if here==nil {return} + names:=make([dynamic]string,context.temp_allocator) + for pinned in ([2][3]string{{game,box,"props"},{plugin,brush,""}}) { + options:=Library_Options{store=here,root=pinned[0],plugins=test_library} + report,error:=ingest(pinned[1],options,allocator=context.temp_allocator) + if !testing.expectf(t,error.code==.None,"%s: %s",pinned[1],error.message) {return} + p,_:=asset.package_get(here,report.manifest,context.temp_allocator) + append(&names,p.name) + added:=project_add(pinned[0],report.manifest,pinned[2],nil,options,context.temp_allocator) + testing.expectf(t,len(added.failures)==0 && len(added.missing)==0,"%v",added) + } + cook :: proc(root:string,s:^store.Store,output:string) -> build.Cook_Report { + plugins:=[1]project.Asset_Folder{{path=fmt.tprintf("%s/plugins/brushes/assets",root),prefix="plugins/brushes",root=fmt.tprintf("%s/plugins/brushes",root)}} + return build.cook_assets({assets=fmt.tprintf("%s/assets",root),output=output,root=root,store=s,plugins=plugins[:],allocator=context.temp_allocator}) + } + shipped:=cook(game,here,fmt.tprintf("%s/shipped",top)) + if !testing.expectf(t,shipped.error==.None,"%v %s",shipped.error,shipped.detail) {return} + paths:=make(map[string]bool,context.temp_allocator) + for file in shipped.files {paths[file.path]=true} + testing.expectf(t,paths["plugins/brushes/soft.tex"] && paths["props/Box.mesh"] && paths["props/wood.tex"],"the outputs of the packages ship: %v",paths) + testing.expectf(t,!paths["plugins/brushes/soft.png"] && !paths["props/box.gltf"] && !paths["props/wood.png"],"their sources stay behind: %v",paths) + + // A peer: the project and its plugin without their sources, and a store of its own. + peer:=fmt.tprintf("%s/peer",top) + folder_copy(game,peer) + for source in ([3]string{"assets/props/box.gltf","assets/props/wood.png","plugins/brushes/assets/soft.png"}) {testing.expect(t,os.remove(fmt.tprintf("%s/%s",peer,source))==nil)} + // Nor the files of their manifests: the source gives each by the hash its pin names. + for root in ([2]string{peer,fmt.tprintf("%s/plugins/brushes",peer)}) { + pins,_,_:=asset.packages_load(root,context.temp_allocator) + for entry in pins.packages {testing.expect(t,os.remove(asset.package_file(root,entry.id))==nil)} + } + there:=test_store(t,fmt.tprintf("%s/there",top)) + if there==nil {return} + lacking:=cook(peer,there,fmt.tprintf("%s/lacking",top)) + testing.expect_value(t,lacking.error,build.Cook_Error.Package_Missing) + testing.expectf(t,strings.contains(lacking.detail,"neither in the store"),"the package is named: %s",lacking.detail) + for root in ([2]string{peer,fmt.tprintf("%s/plugins/brushes",peer)}) { + fetched:=project_refresh(root,{store=there,root=root,source=cache.store_source(here)},context.temp_allocator) + testing.expectf(t,fetched.fetched==1 && fetched.ingested==0 && len(fetched.failures)==0 && len(fetched.missing)==0,"%v",fetched) + } + output:=fmt.tprintf("%s/again",top) + again:=cook(peer,there,output) + if !testing.expectf(t,again.error==.None,"%v %s",again.error,again.detail) {return} + testing.expect_value(t,len(again.files),len(shipped.files)) + indexed:asset.Registry + asset.registry_init(&indexed,output);defer asset.registry_destroy(&indexed) + testing.expect_value(t,asset.registry_load(&indexed,fmt.tprintf("%s/%s",output,asset.REGISTRY_INDEX)),asset.Error.None) + for file,at in again.files { + if !testing.expect_value(t,file.path,shipped.files[at].path) {continue} + first,_:=os.read_entire_file(fmt.tprintf("%s/shipped/%s",top,file.path),context.temp_allocator) + second,_:=os.read_entire_file(fmt.tprintf("%s/%s",output,file.path),context.temp_allocator) + testing.expectf(t,len(first)>0 && slice.equal(first,second),"%s ships as the machine with its source ships it",file.path) + if file.id=={} {continue} + listed,found:=asset.registry_find(&indexed,file.id) + testing.expectf(t,found && listed.path==file.path,"the index has %s",file.path) + a:asset.Asset + if testing.expectf(t,asset.asset_load(&a,fmt.tprintf("%s/%s",output,file.path),context.temp_allocator)==.None,"%s loads",file.path) {asset.asset_close(&a)} + } +} + +// A refresh takes the builds its store lacks from a second store, as the +// installed engine does from the store of the builds of its plugins: with +// the source gone, mirai import --refresh --from copies them and imports nothing. +@(test) +a_refresh_takes_the_builds_of_a_second_store_without_their_sources :: proc(t:^testing.T) { + top:=base.test_folder(t,"refresh-from") + if top=="" {return} + defer free_all(context.temp_allocator) + plugin:=fmt.tprintf("%s/plugin",top) + os.make_directory_all(fmt.tprintf("%s/assets",plugin)) + brush:=fmt.tprintf("%s/assets/soft.png",plugin) + testing.expect(t,os.write_entire_file(brush,png_bytes(8,8,{40,200,40}))==nil) + bundled_folder,machine_folder:=fmt.tprintf("%s/engine/builds",top),fmt.tprintf("%s/machine",top) + bundled:=test_store(t,bundled_folder) + if bundled==nil {return} + options:=Library_Options{store=bundled,root=plugin} + report,error:=ingest(brush,options,allocator=context.temp_allocator) + if !testing.expectf(t,error.code==.None,"%s",error.message) {return} + added:=project_add(plugin,report.manifest,"",nil,options,context.temp_allocator) + testing.expectf(t,len(added.failures)==0 && len(added.missing)==0,"%v",added) + testing.expect(t,os.remove(brush)==nil) + + out,errors:=strings.builder_make(context.temp_allocator),strings.builder_make(context.temp_allocator) + code:=command({"--refresh",plugin,"--store",machine_folder,"--from",bundled_folder},strings.to_writer(&out),strings.to_writer(&errors)) + testing.expectf(t,code==0 && strings.contains(strings.to_string(out),"1 builds fetched"),"%d\n%s%s",code,strings.to_string(out),strings.to_string(errors)) + machine:=test_store(t,machine_folder) + if machine==nil {return} + resolved,resolve_error:=asset.root_packages_in(machine,plugin,context.temp_allocator) + testing.expectf(t,resolve_error==.None && len(resolved.problems)==0 && len(resolved.listed)==1 && resolved.listed[0].state==.Built,"%v",resolved) + testing.expect(t,!os.exists(brush),"nothing is written where the source was") +} + +// The same source ingested where the store writes frames of this build's codec and where it +// writes those of a codec this build lacks (as a build with another compressor does) gives the +// same build, of the same objects: a peer of this build takes it from the other and reads it. +@(test) +a_build_is_the_same_objects_whatever_codec_its_store_writes :: proc(t:^testing.T) { + top:=base.test_folder(t,"refresh-codecs") + if top=="" {return} + defer free_all(context.temp_allocator) + plugin:=fmt.tprintf("%s/plugin",top) + os.make_directory_all(fmt.tprintf("%s/assets",plugin)) + brush:=fmt.tprintf("%s/assets/soft.png",plugin) + testing.expect(t,os.write_entire_file(brush,png_bytes(64,64,{40,200,40}))==nil) + host:=test_store(t,fmt.tprintf("%s/host",top)) + other:=test_store(t,fmt.tprintf("%s/other",top)) + if host==nil || other==nil {return} + // Zstandard under an id of its own: frames no build here decodes. + testing.expect_value(t,store.codec_hook(other,{codec=15,encode=proc(src,dst:[]u8) -> int { + e:data.Encoder + data.encoder_init(&e) + defer data.encoder_destroy(&e) + size,error:=data.encode(&e,.Fast,src,dst) + return size if error==.None else 0 + },decode=proc(src,dst:[]u8) -> bool { + d:data.Decoder + if data.decoder_init(&d)!=.None {return false} + defer data.decoder_destroy(&d) + return data.decode(&d,data.ZSTD,src,dst)==.None + }}),store.Error.None) + options:=Library_Options{store=host,root=plugin} + report,error:=ingest(brush,options,allocator=context.temp_allocator) + if !testing.expectf(t,error.code==.None,"%s",error.message) {return} + added:=project_add(plugin,report.manifest,"",nil,options,context.temp_allocator) + testing.expectf(t,len(added.failures)==0 && len(added.missing)==0,"%v",added) + // The other ingests it again from its source, as a machine of another compressor does. + again:=project_refresh(plugin,{store=other,root=plugin},context.temp_allocator) + testing.expectf(t,len(again.missing)==0 && len(again.failures)==0,"%v",again) + digits:[64]u8 + key:=store.hex(report.key,&digits) + namespace:=asset.REF_IMPORTS if report.final else asset.REF_IMPORTS_FAST + mine,_:=asset.build_get(host,report.build,context.temp_allocator) + theirs_ref,ref_error:=store.ref_get(other,namespace,key) + if !testing.expect_value(t,ref_error,store.Error.None) {return} + theirs,_:=asset.build_get(other,theirs_ref,context.temp_allocator) + testing.expect_value(t,theirs_ref,report.build) + if testing.expect_value(t,len(theirs.outputs),len(mine.outputs)) { + for o,at in mine.outputs {testing.expectf(t,theirs.outputs[at].object==o.object,"output %d is the same object",at)} + } + // A peer of this build takes the build from the other, whose frames it cannot decode. + testing.expect(t,os.remove(brush)==nil) + peer:=test_store(t,fmt.tprintf("%s/peer",top)) + if peer==nil {return} + fetched:=project_refresh(plugin,{store=peer,root=plugin,source=cache.store_source(other)},context.temp_allocator) + testing.expectf(t,fetched.fetched==1 && len(fetched.missing)==0 && len(fetched.failures)==0,"%v",fetched) + for o in mine.outputs { + a:asset.Asset + testing.expect_value(t,asset.asset_open_object(&a,peer,o.object,context.temp_allocator),asset.Error.None) + testing.expect_value(t,asset.asset_object_settle(&a),asset.Error.None) + asset.asset_close(&a) + } +} + +// A source of builds that holds only a Fast build of a package: the refresh +// of a cook never asks for it and fails by the package's name, the refresh of +// a build that does not cook takes it without importing, and a cook still +// ships nothing Fast. Once the source holds the Final build, the refresh of +// the cook takes it and the game ships it. +@(test) +a_cook_never_takes_a_fast_build_a_source_holds :: proc(t:^testing.T) { + types:asset.Types;asset.types_init(&types);defer asset.types_destroy(&types) + top:=base.test_folder(t,"cook-fast-source") + if top=="" {return} + defer free_all(context.temp_allocator) + game:=fmt.tprintf("%s/game",top) + os.make_directory_all(fmt.tprintf("%s/assets",game)) + poster:=fmt.tprintf("%s/assets/poster.png",game) + testing.expect(t,os.write_entire_file(poster,png_bytes(8,8,{30,60,220}))==nil) + host:=test_store(t,fmt.tprintf("%s/host",top)) + if host==nil {return} + fast_options:=Library_Options{store=host,root=game,fast_textures=true} + quick,quick_error:=ingest(poster,fast_options,allocator=context.temp_allocator) + if !testing.expectf(t,quick_error.code==.None,"%s",quick_error.message) {return} + added:=project_add(game,quick.manifest,"",nil,fast_options,context.temp_allocator) + if !testing.expectf(t,len(added.failures)==0 && len(added.missing)==0,"%v",added) {return} + testing.expect(t,!quick.final,"the host holds a Fast build alone") + p,_:=asset.package_get(host,quick.manifest,context.temp_allocator) + + peer:=fmt.tprintf("%s/peer",top) + folder_copy(game,peer) + testing.expect(t,os.remove(fmt.tprintf("%s/assets/poster.png",peer))==nil) + here:=test_store(t,fmt.tprintf("%s/here",top)) + if here==nil {return} + cook :: proc(root:string,s:^store.Store,output:string) -> build.Cook_Report { + manifest:=fmt.tprintf("%s/%s",root,build.COOK_FILE) + _=os.write_entire_file(manifest,`{"roots":["poster.tex"]}`) + return build.cook_assets({assets=fmt.tprintf("%s/assets",root),output=output,manifest=manifest,root=root,store=s,allocator=context.temp_allocator}) + } + digits:[64]u8 + key:=store.hex(quick.key,&digits) + from:=cache.store_source(host) + final:=project_refresh(peer,{store=here,root=peer,source=from},context.temp_allocator) + testing.expectf(t,final.fetched==0 && len(final.missing)==1 && strings.contains(final.missing[0],fmt.tprintf("package %s",p.name)),"%v",final) + _,asked:=store.ref_get(here,asset.REF_IMPORTS_FAST,key) + testing.expect_value(t,asked,store.Error.Missing) + lacking:=cook(peer,here,fmt.tprintf("%s/lacking",top)) + testing.expectf(t,lacking.error==.Package_Missing && strings.contains(lacking.detail,p.name),"%v %s",lacking.error,lacking.detail) + + taken:=project_refresh(peer,{store=here,root=peer,source=from,fast_textures=true},context.temp_allocator) + testing.expectf(t,taken.fetched==1 && taken.ingested==0 && len(taken.missing)==0 && len(taken.failures)==0,"%v",taken) + refused:=cook(peer,here,fmt.tprintf("%s/refused",top)) + testing.expectf(t,refused.error==.Source_Failed && strings.contains(refused.detail,p.name),"a Fast build taken is not shipped: %v %s",refused.error,refused.detail) + still:=project_refresh(peer,{store=here,root=peer,source=from},context.temp_allocator) + testing.expectf(t,len(still.missing)==1 && strings.contains(still.missing[0],"only a build with Fast textures is here"),"%v",still) + + shipped_report,shipped_error:=ingest(poster,{store=host,root=game},allocator=context.temp_allocator) + if !testing.expectf(t,shipped_error.code==.None,"%s",shipped_error.message) {return} + testing.expect(t,shipped_report.final && shipped_report.key==quick.key,"the host builds the Final of the same key") + fetched:=project_refresh(peer,{store=here,root=peer,source=from},context.temp_allocator) + testing.expectf(t,fetched.fetched==1 && fetched.ingested==0 && len(fetched.missing)==0 && len(fetched.failures)==0,"%v",fetched) + output:=fmt.tprintf("%s/game-out",top) + shipped:=cook(peer,here,output) + if !testing.expectf(t,shipped.error==.None,"%v %s",shipped.error,shipped.detail) {return} + textures_shipped:=0 + for file in shipped.files { + a:asset.Asset + if asset.asset_load(&a,fmt.tprintf("%s/%s",output,file.path),context.temp_allocator)!=.None {continue} + if view,error:=asset.texture_view(&a,&types);error==.None { + textures_shipped+=1 + testing.expectf(t,.Fast not_in view.info.flags,"%s ships Final",file.path) + } + } + testing.expect(t,textures_shipped>0,"the texture ships") +} diff --git a/cook/import/sources/fuzz_test.odin b/cook/import/sources/fuzz_test.odin index 14233c6e2..d360f5b4a 100644 --- a/cook/import/sources/fuzz_test.odin +++ b/cook/import/sources/fuzz_test.odin @@ -125,55 +125,25 @@ damaged_images_and_settings_are_imported_or_refused :: proc(t:^testing.T) { png:=png_bytes(8,8,{200,120,60},true) path:=fmt.tprintf("%s/subject.png",folder) imported,refused:=0,0 - attempt :: proc(path,root:string,data:[]u8,imported,refused:^int) { + options:=test_options(t,path,{root=root,image=.Color,ignore_stored=true}) + attempt :: proc(path:string,options:importer.Import_Options,data:[]u8,imported,refused:^int) { if os.write_entire_file(path,data)!=nil {return} - report,error:=importer.import_texture(path,{root=root,image=.Color,ignore_stored=true}) + report,error:=importer.import_texture(path,options) if error.code==.None { importer.report_destroy(&report) imported^+=1 } else {refused^+=1} } - attempt(path,root,png,&imported,&refused) + attempt(path,options,png,&imported,&refused) testing.expect_value(t,imported,1) - for length in 0..len(png)/2,"%d of %d damaged images are refused",refused,imported+refused) - - // The sidecar of a source: any text is the settings and the outputs it says or is refused, and never the defaults or other ids by accident. - wanted:=importer.Import_Settings{textures=.Uncompressed,rdo_lambda=30,image=.Color,alpha_coverage=0.25,lods=.None,front=.X} - listed:=make([dynamic]importer.Output,context.temp_allocator) - append(&listed,importer.Output{kind=asset.KIND_MESH,name="Crate \"big\"",index=2,id={3,4},part=true},importer.Output{kind=asset.KIND_MATERIAL,name="Wood",id={5,6}},importer.Output{kind=asset.KIND_TEXTURE,name="wood",id={7,8}}) - text:=importer.sidecar_text({settings=wanted,folder="crate",outputs=listed},context.temp_allocator) - parsed,valid:=importer.sidecar_parse(text,context.temp_allocator) - testing.expect(t,valid && parsed.settings==wanted && parsed.folder=="crate" && len(parsed.outputs)==3) - if valid && len(parsed.outputs)==3 { - for output,at in parsed.outputs {testing.expectf(t,output==listed[at],"output %d is read as written: %v",at,output)} - } - read,rejected:=0,0 - for length in 0..>33)%u64(len(work)))]=marks[int((state>>20)%u64(len(marks)))] - if changed,ok:=importer.sidecar_parse(string(work),context.temp_allocator);ok { - read+=1 - // What is read is settings an import can use, and outputs it can write. - changed_settings:=changed.settings - testing.expect(t,changed_settings.alpha_coverage>=0 && changed_settings.alpha_coverage<=1 && changed_settings.lods<=max(importer.Lods) && changed_settings.front<=max(importer.Front)) - for output in changed.outputs {testing.expect(t,output.id!={} && output.kind!=0,"an output has an id and a kind")} - } else {rejected+=1} - } - testing.expectf(t,rejected>len(text)/2 && read>0,"%d texts are sidecars and %d are refused",read,rejected) } // Fonts and manifests of prompts, cut at lengths spread over them or with @@ -191,26 +161,27 @@ damaged_fonts_and_prompts_are_imported_or_refused :: proc(t:^testing.T) { for subject in subjects { path:=fmt.tprintf("%s/%s",folder,subject.file) imported,refused:=0,0 - attempt :: proc(path,folder:string,data:[]u8,imported,refused:^int) { + options:=test_options(t,path) + attempt :: proc(path:string,options:importer.Import_Options,data:[]u8,imported,refused:^int) { if os.write_entire_file(path,data)!=nil {return} - report,error:=import_source(path,{}) + report,error:=import_source(path,options) if error.code==.None { importer.report_destroy(&report) imported^+=1 } else {refused^+=1} } - attempt(path,folder,subject.data,&imported,&refused) + attempt(path,options,subject.data,&imported,&refused) testing.expectf(t,imported==1,"%s as it is is imported",subject.file) step:=max(len(subject.data)/200,1) - for length:=0;length0 && imported>1,"%s: %d damaged files imported and %d refused",subject.file,imported-1,refused) } diff --git a/cook/import/sources/library.odin b/cook/import/sources/library.odin new file mode 100644 index 000000000..38615cd44 --- /dev/null +++ b/cook/import/sources/library.odin @@ -0,0 +1,845 @@ +package sources + +import "core:fmt" +import "core:io" +import "core:mem" +import "core:os" +import "core:path/filepath" +import "core:slice" +import "core:strings" +import "core:time" +import "mirai:asset" +import "mirai:core" +import "mirai:cook/cache" +import importer "mirai:cook/import" +import "mirai:cook/workspace" +import "mirai:project" +import "mirai:std/base" +import "mirai:std/jobs" +import "mirai:std/serial" +import "mirai:store" + +// The library of the machine: the packages its store keeps (asset/packages.odin) +// and what a project does with them. +// +// - ingest: a source anywhere on disk becomes a package of the store, where +// it is; nothing is copied into a project. Ingesting the same origin again +// finds its package (asset.REF_ORIGINS) and keeps its ids; one whose +// identity and build are current is not imported again. Two processes that +// ingest one origin at once take turns (a lock per origin), and the second +// finds the work done. +// - project_add: pins a package in the project's packages.mirai, with the +// text of its manifest beside it (project_pin and project_unpin, their only +// writers), and follows it into the project. +// - project_refresh: for each package a project pins, the build this engine +// reads, from the local refs, else from the caller's source, else by +// ingesting its origin when it is on this machine (also when the source +// changed); else the report says, by package name, that a machine with its +// source must ingest it. A manifest the store lacks comes from the +// project's file of it, else from the caller's source. What it asks the +// source for is asked together first (refresh_prefetch). It never removes a +// blob, a ref or a file, and never builds without the source. +// +// Following a package writes or follows its documents in the project's folder +// of it: from the build's baselines (importer.documents_follow), and for a +// model whose source is here, by importing it with the project's folder, +// which also writes its prefab in the game's types (prefab.model_prefab). The +// pin records what each file it writes over holds as written +// (asset.Package_Written): a refresh leaves one that changed since as it is +// and warns, and an import into the project asks first (Library_Options.changed). + +// What the library of the editor of the project at root gives its imports: +// the formats of models and the importers of whole sources its plugins add, +// with what they read prepared for its assets, and the types of its game, +// which the prefab of a model is written in, with the packages of root listed +// from the store s; release(state) lets them go. problem: why there are none. +// Only a program that links the editor reads it (editor.import_library). +Library :: #type proc(root:string,s:^store.Store,allocator:mem.Allocator) -> Library_Imports +Library_Imports :: struct { + formats:[]importer.Format, + wholes:[]Whole_Source, + world:^core.World_Info, + release:proc(state:rawptr), + state:rawptr, + problem:string, +} + +Library_Options :: struct { + store:^store.Store, + // Asked for what the store does not have before building: a manifest by + // (asset.REF_MANIFESTS, ), a build by (asset.REF_IMPORTS, ) and, + // with Fast textures allowed, then by (asset.REF_IMPORTS_FAST, ), with + // every object it lists, a result of cook/cache. + source:cache.Source, + // Fast textures allowed: imports build them Fast unless the texture cache + // has them Final, and a Fast build is taken, here or from the source; false: + // Final required, what a cook ships, and a Fast build is never asked for. + fast_textures:bool, + rebuild:bool, // ingest: imports the source again, also when it is current + // What an import into a project (project_ingest) does with its documents + // that changed since they were last written; a refresh and the follow of + // project_add keep them, unless Discard. + changed:importer.Changed, + // The project or plugin the library works for: its plugins read sources + // (plugins) and its game's types hold prefabs. "": the formats of the engine alone. + root:string, + plugins:Library, // nil: no formats of plugins, and no prefabs of models + // What the plugins give, already prepared by the caller, which releases + // it (the editor, from its session): taken in place of plugins. + imports:^Library_Imports, + worker:jobs.Worker_Ref, + progress:io.Writer, // where each source imported is said as it goes; nil: nowhere +} + +Library_Report :: struct { + packages:int, // pinned by the project + ingested:int, // imported now + fetched:int, // builds the source gave + missing:[dynamic]string, // a package with no build here and no source on this machine, by name, and what gives it + failures:[dynamic]string, // what could not be read or imported, with the reason + warnings:[dynamic]string, + allocator:mem.Allocator, +} + +library_report_destroy :: proc(report:^Library_Report) { + for text in report.missing {delete(text,report.allocator)} + for text in report.failures {delete(text,report.allocator)} + for text in report.warnings {delete(text,report.allocator)} + delete(report.missing) + delete(report.failures) + delete(report.warnings) + report^={} +} + +// Ingests the source at origin, a file or a folder anywhere on disk, as a +// package of options.store. settings, when given, replace those its package +// keeps. A source whose package is current (same identity, a build this +// engine reads) is not imported: the report names its package and lists no +// assets. In allocator. +ingest :: proc(origin:string,options:Library_Options,settings:^importer.Import_Settings=nil,allocator:=context.allocator) -> (report:importer.Import_Report,error:importer.Error) { + l:Library_State + library_begin(&l,options,allocator) + defer library_end(&l) + return library_ingest(&l,origin,{},settings,"",nil,.Keep,allocator) +} + +// Pins the package whose manifest the store keeps under manifest in the +// project at root, its cache listed under folder of its assets, using assets +// (none: all), and follows it there (project_refresh). In allocator. +project_add :: proc(root:string,manifest:store.Hash,folder:string,assets:[]base.Asset_Id,options:Library_Options,allocator:=context.allocator) -> (report:Library_Report) { + report_begin(&report,allocator) + l:Library_State + library_begin(&l,options,allocator) + defer library_end(&l) + context.temp_allocator=l.temp + top:=project.path_absolute(root,context.temp_allocator) + digits:[64]u8 + p,error:=asset.package_get(options.store,manifest,context.temp_allocator) + if error!=.None { + report_line(&report.failures,"%s: the manifest %s is not a package in the store at %s (%v): ingest its source, or fetch it from a peer that has it",top,store.hex(manifest,&digits),options.store.root,error) + return + } + entry:=asset.Project_Package{id=p.id,manifest=manifest,folder=folder} + entry.assets=make([dynamic]base.Asset_Id,context.temp_allocator) + for id in assets { + known:=false + for &a in p.assets {known||=a.id==id} + if !known { + report_line(&report.failures,"%s: package %s has no asset %v: use one its manifest lists",top,p.name,id) + return + } + append(&entry.assets,id) + } + if pin_error:=project_pin(top,options.store,entry);pin_error!=.None { + report_line(&report.failures,"%s: %s could not be written (%v): a folder is relative to the assets, with forward slashes, and never leaves them",top,asset.PACKAGES_FILE,pin_error) + return + } + report.packages=1 + project_follow_begin(&l,top) + package_refresh(&l,top,entry,&report,true) + return +} + +// Ingests the source at origin as ingest does and pins its package in the +// project at root under folder, with what the project used of it before: one +// import, which for a model writes its documents and its prefab there. The +// report of the import and that of the project, in allocator. +project_ingest :: proc(root,origin,folder:string,options:Library_Options,settings:^importer.Import_Settings=nil,allocator:=context.allocator) -> (report:importer.Import_Report,added:Library_Report,error:importer.Error) { + report_begin(&added,allocator) + l:Library_State + library_begin(&l,options,allocator) + defer library_end(&l) + caller:=context.temp_allocator + context.temp_allocator=l.temp + top:=project.path_absolute(root,context.temp_allocator) + project_follow_begin(&l,top) + entry:=asset.Project_Package{folder=folder} + documents:=l.assets if folder=="" else fmt.tprintf("%s/%s",l.assets,folder) + r,_:=reader_of(&l,importer.path_key(origin)) + model:=r.kind==.Model && plugins_world(&l)!=nil + // What the project pins of the package its source was ingested as before. + if p,_,found,_:=importer.package_previous(options.store,{},importer.path_key(origin),context.temp_allocator);found { + list,_,_:=asset.packages_load(top,context.temp_allocator) + for &other in list.packages { + if other.id==p.id {entry.assets,entry.written=other.assets,other.written} + } + } + report,error=library_ingest(&l,origin,{},settings,documents if model else "",entry.written[:],l.options.changed,allocator) + if error.code!=.None { + error.message=strings.clone(error.message,caller) + return + } + if entry.id!=report.package_id {entry.assets,entry.written={},{}} + entry.id,entry.manifest=report.package_id,report.manifest + if model {entry.written=written_list(report.written[:])} + if pin_error:=project_pin(top,options.store,entry);pin_error!=.None { + report_line(&added.failures,"%s: %s could not be written (%v): a folder is relative to the assets, with forward slashes, and never leaves them",top,asset.PACKAGES_FILE,pin_error) + return + } + added.packages=1 + if !model {package_refresh(&l,top,entry,&added,true)} + return +} + +// A list of what was written the pin keeps, by id, in the temp allocator. +@(private="file") +written_list :: proc(written:[]asset.Package_Written) -> [dynamic]asset.Package_Written { + list:=make([dynamic]asset.Package_Written,context.temp_allocator) + append(&list,..written) + slice.sort_by(list[:],proc(a,b:asset.Package_Written) -> bool {return a.id[0] (report:Library_Report) { + report_begin(&report,allocator) + l:Library_State + library_begin(&l,options,allocator) + defer library_end(&l) + context.temp_allocator=l.temp + top:=project.path_absolute(root,context.temp_allocator) + list,_,error:=asset.packages_load(top,context.temp_allocator) + if error!=.None { + report_line(&report.failures,"%s/%s does not read (%v): fix it from its history; nothing was refreshed",top,asset.PACKAGES_FILE,error) + return + } + report.packages=len(list.packages) + if len(list.packages)==0 {return} + project_follow_begin(&l,top) + refresh_prefetch(&l,top,list.packages[:]) + for &entry,at in list.packages { + if options.progress.procedure!=nil {fmt.wprintfln(options.progress,"package %d of %d",at+1,len(list.packages))} + package_refresh(&l,top,entry,&report,false) + } + return +} + +// Pins entry in the packages.mirai of the project at root, in place of the +// entry of the same package, and writes beside it the text of the manifest +// that s keeps under entry.manifest (asset.package_file). It and +// project_unpin are the only writers of those files. A pin lands whole or +// leaves the root as it was: a list that does not save puts the manifest's +// file back. +project_pin :: proc(root:string,s:^store.Store,entry:asset.Project_Package) -> asset.Error { + entry:=entry + list,_,error:=asset.packages_load(root,context.temp_allocator) + if error!=.None {return error} + replaced:=false + for &other in list.packages { + if other.id!=entry.id {continue} + other=entry + replaced=true + } + if !replaced { + if _,no_room:=append(&list.packages,entry);no_room!=nil {return .Out_Of_Memory} + } + // What refuses the list refuses it before a file changes. + asset.packages_text(&list,context.temp_allocator) or_return + path:=asset.package_file(root,entry.id) + folder:=os.dir(path) + had_folder:=os.is_dir(folder) + before,read_error:=serial.file_read(path,context.temp_allocator,limit=asset.RECORD_BYTES) + if read_error!=.None && read_error!=.Not_Found {return .IO} + asset.package_file_save(s,root,&entry) or_return + saved:=asset.packages_save(root,&list) + if saved==.None {return .None} + if read_error==.None { + serial.file_write_atomic(path,before) + } else { + os.remove(path) + if !had_folder {os.remove(folder)} + } + return saved +} + +// Removes the package id from the packages.mirai of the project at root, and +// the file of its manifest; what the store keeps of it stays. False with +// nothing to remove. +project_unpin :: proc(root:string,id:base.Asset_Id) -> (removed:bool,error:asset.Error) { + list,_:=asset.packages_load(root,context.temp_allocator) or_return + for &other,at in list.packages { + if other.id!=id {continue} + ordered_remove(&list.packages,at) + asset.packages_save(root,&list) or_return + removed=true + break + } + return removed,asset.package_file_remove(root,id) +} + +// What ingest, project_add and project_refresh share for one call. +@(private="file") +Library_State :: struct { + options:Library_Options, + scratch:mem.Dynamic_Arena, + temp:mem.Allocator, + plugins:Library_Imports, + plugins_read:bool, + formats:[]importer.Format, // FORMATS and those of the plugins, once read + assets:string, // of the project followed + registry:^asset.Registry, // its assets, scanned once + asked:map[string]bool, // "/" asked of the source, and whether it gave it +} + +@(private="file") +library_begin :: proc(l:^Library_State,options:Library_Options,allocator:mem.Allocator) { + l.options=options + mem.dynamic_arena_init(&l.scratch,allocator,allocator,block_size=1<<20,out_band_size=1<<20) + l.temp=base.dynamic_arena_allocator(&l.scratch) + l.formats=FORMATS[:] + l.asked=make(map[string]bool,l.temp) +} + +@(private="file") +library_end :: proc(l:^Library_State) { + if l.plugins.release!=nil {l.plugins.release(l.plugins.state)} + mem.dynamic_arena_destroy(&l.scratch) +} + +// The formats, the whole importers and the game types of the plugins of the +// root of the library, read the first time a source needs them. +@(private="file") +plugins_read :: proc(l:^Library_State) { + if l.plugins_read {return} + l.plugins_read=true + switch { + case l.options.imports!=nil: + l.plugins=l.options.imports^ + l.plugins.release=nil + case l.options.plugins==nil || l.options.root=="": return + case: l.plugins=l.options.plugins(l.options.root,l.options.store,l.temp) + } + all:=make([dynamic]importer.Format,0,len(FORMATS)+len(l.plugins.formats),l.temp) + append(&all,..FORMATS[:]) + append(&all,..l.plugins.formats) + l.formats=all[:] +} + +@(private="file") +project_follow_begin :: proc(l:^Library_State,top:string) { + if l.options.root=="" {l.options.root=top} + l.assets=project.folder_path(top,.Assets,l.temp) + l.registry=new(asset.Registry,l.temp) + asset.registry_init(l.registry,l.assets,"",l.temp) + asset.registry_scan(l.registry) +} + +// What reads the source at origin: kind (None for one a plugin imports +// whole), the formats it is read with and the whole importer that claims it. +@(private="file") +Source_Reader :: struct { + kind:Source_Kind, + whole:Whole_Source, + claimed:bool, +} + +@(private="file") +reader_of :: proc(l:^Library_State,origin:string) -> (r:Source_Reader,ok:bool) { + r.kind=source_kind(origin,l.formats) + if r.kind==.None || r.kind==.Model { + plugins_read(l) + if r.whole,r.claimed=whole_find(l.plugins.wholes,origin);r.claimed {return {kind=.None,whole=r.whole,claimed=true},true} + r.kind=source_kind(origin,l.formats) + } + return r,r.kind!=.None +} + +// The key of the build of p this engine reads (importer.build_key): the +// versions of the engine with that of what reads its origin, an engine +// importer, a format of the engine or of the plugins, or the importer of a +// plugin that imports it whole, with the code of that plugin. False when none +// of those is the one its manifest records. +@(private="file") +package_key :: proc(l:^Library_State,p:^asset.Package) -> (key:store.Hash,ok:bool) { + version:u32 + code:u64 + switch p.origin.importer { + case "image","sound","font","prompts": + version=importer.import_version(0) + case: + found:=false + for format in FORMATS { + if format.name==p.origin.importer {version,found=importer.import_version(format.version),true} + } + if !found { + plugins_read(l) + for format in l.formats { + if format.name==p.origin.importer {version,found=importer.import_version(format.version),true} + } + for whole in l.plugins.wholes { + if !found && whole.name!="" && whole.name==p.origin.importer {version,code,found=importer.import_version(whole.version),whole.code,true} + } + } + if !found {return} + } + return importer.build_key(p.identity,version,code),true +} + +@(private="file") +Found :: struct { + build:asset.Build, + hash:store.Hash, + final:bool, + fast_here:bool, // a Fast build is here, which a Final one was required over +} + +// The build of p under key this store has whole: Final, or Fast when Fast +// textures are allowed. A build is taken only under the namespace of what its +// manifest says it is. +@(private="file") +build_here :: proc(l:^Library_State,p:^asset.Package,key:store.Hash) -> (f:Found,ok:bool) { + s:=l.options.store + digits:[64]u8 + name:=store.hex(key,&digits) + for namespace,at in ([2]string{asset.REF_IMPORTS,asset.REF_IMPORTS_FAST}) { + hash,ref_error:=store.ref_get(s,namespace,name) + if ref_error!=.None {continue} + b,error:=asset.build_get(s,hash,l.temp) + if error!=.None || b.package_id!=p.id || b.key!=key || b.final!=(at==0) {continue} + whole:=true + for o in b.outputs {whole&&=asset.object_whole(s,o.object)} + if !whole {continue} + if at==1 && !l.options.fast_textures { + f.fast_here=true + continue + } + return {build=b,hash=hash,final=at==0},true + } + return +} + +// The build of p this engine reads here: by its key, or for a source whose +// importer this library lacks (a plugin that imports it whole and is not +// read), the one this machine chose for its manifest (asset.REF_BUILDS) when +// it is whole. +@(private="file") +build_found :: proc(l:^Library_State,p:^asset.Package,manifest,key:store.Hash,keyed:bool) -> (f:Found,ok:bool) { + if keyed {return build_here(l,p,key)} + s:=l.options.store + digits:[64]u8 + hash,ref_error:=store.ref_get(s,asset.REF_BUILDS,store.hex(asset.build_select(manifest,p),&digits)) + if ref_error!=.None {return} + b,error:=asset.build_get(s,hash,l.temp) + if error!=.None || b.package_id!=p.id {return} + for o in b.outputs { + if !asset.object_whole(s,o.object) {return} + } + if !b.final && !l.options.fast_textures {return {fast_here=true},false} + return {build=b,hash=hash,final=b.final},true +} + +// Asks the source of the library for the ref (namespace, name), once per call +// of the library. +@(private="file") +fetch :: proc(l:^Library_State,namespace,name:string) -> bool { + if given,asked:=l.asked[fmt.aprintf("%s/%s",namespace,name,allocator=l.temp)];asked {return given} + names:=[1]string{name} + fetch_many(l,namespace,names[:]) + return l.asked[fmt.aprintf("%s/%s",namespace,name,allocator=l.temp)] +} + +// Asks the source for the refs of namespace with names together, but those +// asked before. +@(private="file") +fetch_many :: proc(l:^Library_State,namespace:string,names:[]string) { + source:=l.options.source + if source.fetch==nil || len(names)==0 {return} + asking:=make([dynamic]string,l.temp) + keys:=make([dynamic]string,l.temp) + for name in names { + key:=fmt.aprintf("%s/%s",namespace,name,allocator=l.temp) + if key in l.asked {continue} + l.asked[key]=false + append(&asking,name) + append(&keys,key) + } + given:=make([]bool,len(asking),l.temp) + source.fetch(source.user,l.options.store,namespace,asking[:],given) + for key,k in keys {l.asked[key]=given[k]} +} + +// Asks the source for the builds of the keys (hex of import keys) together: +// Final, then, with Fast textures allowed, Fast for those it gave no Final of. +@(private="file") +fetch_builds :: proc(l:^Library_State,keys:[]string) { + fetch_many(l,asset.REF_IMPORTS,keys) + if !l.options.fast_textures {return} + lacking:=make([dynamic]string,l.temp) + for key in keys { + if !build_given(l,key) {append(&lacking,key)} + } + fetch_many(l,asset.REF_IMPORTS_FAST,lacking[:]) +} + +// Whether the source gave a build of the key (hex of an import key). +@(private="file") +build_given :: proc(l:^Library_State,key:string) -> bool { + return l.asked[fmt.aprintf("%s/%s",asset.REF_IMPORTS,key,allocator=l.temp)] || l.asked[fmt.aprintf("%s/%s",asset.REF_IMPORTS_FAST,key,allocator=l.temp)] +} + +// What a refresh will ask the source for, asked together before the packages +// are refreshed in order, which then find it here: the manifests the store +// does not keep, then the builds of the packages that have none here and no +// changed source here to ingest again. A refresh asks for nothing else. +@(private="file") +refresh_prefetch :: proc(l:^Library_State,top:string,entries:[]asset.Project_Package) { + if l.options.source.fetch==nil {return} + s:=l.options.store + lacking:=make([dynamic]string,l.temp) + for entry in entries { + entry:=entry + if _,error:=asset.package_load(s,top,&entry,l.temp);error==.Not_Found { + digits:[64]u8 + append(&lacking,strings.clone(store.hex(entry.manifest,&digits),l.temp)) + } + } + fetch_many(l,asset.REF_MANIFESTS,lacking[:]) + keys:=make([dynamic]string,l.temp) + for entry in entries { + entry:=entry + p,error:=asset.package_load(s,top,&entry,l.temp) + if error!=.None {continue} + source:=importer.origin_path(p.origin.path,top) + if source!="" && !importer.unsafe_path(source) && os.exists(source) && source_changed(l,&p,source) {continue} + key,keyed:=package_key(l,&p) + if !keyed {continue} + if _,found:=build_found(l,&p,entry.manifest,key,keyed);found {continue} + digits:[64]u8 + append(&keys,strings.clone(store.hex(key,&digits),l.temp)) + } + fetch_builds(l,keys[:]) +} + +// The import of the source at origin into the store, with its lock: what +// ingest does. manifest: the package it was ingested as (zero: the one this +// machine names for origin). documents: the folder of the project the +// documents and the prefab of the import are written or followed in ("": +// none), with the registry and the game types of the library; written and +// changed: what the project's pin records its files held when last written, +// and what becomes of one that changed since (Import_Options.changed). +@(private="file") +library_ingest :: proc(l:^Library_State,path:string,manifest:store.Hash,settings:^importer.Import_Settings,documents:string,written:[]asset.Package_Written,changed:importer.Changed,allocator:mem.Allocator) -> (report:importer.Import_Report,error:importer.Error) { + caller:=context.temp_allocator + context.temp_allocator=l.temp + s:=l.options.store + if s==nil {return {},importer.error_make(.Write,"%s: the library keeps packages in a store: give it one",path,allocator=caller)} + if importer.unsafe_path(path) || !os.exists(path) {return {},importer.error_make(.Not_Found,"%s does not exist",path,allocator=caller)} + origin:=importer.path_key(path) + r,known:=reader_of(l,origin) + if !known { + return {},importer.error_make(.Unsupported,"%s is not a source Mirai imports: a model (%s), an image, a sound (WAV, Ogg Vorbis), a font (TrueType, OpenType) or a manifest of prompts (%s); a format a plugin adds imports with the plugin in the project%s", + origin,importer.format_names(l.formats),importer.PROMPTS_MANIFEST,l.plugins.problem!=""?fmt.tprintf(" (%s)",l.plugins.problem):"",allocator=caller) + } + name:[64]u8 + lock,locked:=workspace.lock_acquire(fmt.tprintf("%s/locks/%s",s.root,asset.origin_name(origin,&name)),fmt.tprintf("mirai ingest %s",origin)) + defer if locked {workspace.lock_release(&lock)} + hashes:=importer.source_hashes_load(s,origin,l.temp) + defer importer.source_hashes_save(s,origin,&hashes) + code:=r.whole.code if r.claimed else 0 + // Current: the work is done, here or by another process that held the lock. + if !l.options.rebuild && settings==nil && documents=="" { + if p,hash,found,_:=importer.package_previous(s,manifest,origin,l.temp);found { + kept,valid:=importer.settings_parse(p.settings) + if valid && importer.identity_of(p.id,origin,p.inputs[:],kept,code,&hashes)==p.identity { + key,keyed:=package_key(l,&p) + if f,here:=build_found(l,&p,hash,key,keyed);here { + { + importer.report_init(&report,allocator) + report.package_id,report.manifest,report.build,report.key,report.final=p.id,hash,f.hash,key,f.final + report.settings=kept + return + } + } + } + } + } + options:=importer.Import_Options{ + store=s,cache_source=l.options.source,manifest=manifest,documents=documents,root=l.options.root, + rebuild=l.options.rebuild,fast_textures=l.options.fast_textures,worker=l.options.worker,hashes=&hashes, + } + if documents!="" {options.registry,options.world,options.written,options.changed=l.registry,l.plugins.world,written,changed} + if options.root=="" {options.root,_=importer.source_root(origin,l.temp)} + if settings!=nil {options.settings,options.ignore_stored=settings^,true} + progress:importer.Progress + if l.options.progress.procedure!=nil { + importer.progress_begin(&progress,l.options.progress,filepath.base(origin)) + options.progress=&progress + } + wholes:[]Whole_Source + if r.claimed {wholes=l.plugins.wholes} + report,error=import_source(origin,options,l.formats,wholes,allocator) + if error.code!=.None {error.message=strings.clone(error.message,caller)} + return +} + +// One package a project pins, brought to a build this engine reads and +// followed into the project: when its build changed, or always with follow. +@(private="file") +package_refresh :: proc(l:^Library_State,top:string,entry:asset.Project_Package,report:^Library_Report,follow:bool) { + s:=l.options.store + entry:=entry + digits,id_text:[64]u8 + id_name:=base.asset_id_format(entry.id,id_text[:32]) + p,error:=asset.package_load(s,top,&entry,l.temp) + // The manifest the pin names, by the hash of its text: the ref that names its object. + if error==.Not_Found && fetch(l,asset.REF_MANIFESTS,store.hex(entry.manifest,&digits)) {p,error=asset.package_load(s,top,&entry,l.temp)} + #partial switch error { + case .None: + case .Not_Found: + report_line(&report.missing,"package %s: neither the store at %s nor %s has its manifest %s, and no peer gave it: restore that file from version control",id_name,s.root,asset.package_file(top,entry.id),store.hex(entry.manifest,&digits)) + return + case .Hash: + report_line(&report.failures,"package %s: %s is not the manifest %s/%s pins: restore both from version control, or ingest its source again",id_name,asset.package_file(top,entry.id),top,asset.PACKAGES_FILE) + return + case: + report_line(&report.failures,"package %s: its manifest %s in the store at %s is not a package of this version (%v): ingest its source again",id_name,store.hex(entry.manifest,&digits),s.root,error) + return + } + source:=importer.origin_path(p.origin.path,top) + origin_here:=source!="" && !importer.unsafe_path(source) && os.exists(source) + documents:=l.assets if entry.folder=="" else fmt.tprintf("%s/%s",l.assets,entry.folder) + prefab:=model_prefab_of(l,&p) + // A source here that changed is ingested again, keeping its ids, and the project follows it. + if origin_here && source_changed(l,&p,source) { + reingest(l,top,&entry,&p,source,documents,prefab,"its source changed",report) + return + } + key,keyed:=package_key(l,&p) + f:Found + found:bool + f,found=build_found(l,&p,entry.manifest,key,keyed) + key_name:=strings.clone(store.hex(key,&digits),l.temp) + if found && keyed && build_given(l,key_name) { + // The source gave it before the refresh came to it. + report.fetched+=1 + } else if !found && keyed { + names:=[1]string{key_name} + fetch_builds(l,names[:]) + if build_given(l,key_name) { + if f,found=build_here(l,&p,key);found {report.fetched+=1} + } + } + if !found { + if origin_here { + reingest(l,top,&entry,&p,source,documents,prefab,"this engine reads no build of it here",report) + return + } + switch { + case !keyed: + report_line(&report.missing,"package %s: no plugin of the project imports it as %q, so its build cannot be found by key: activate the plugin that imports it, or ingest its source (%s) with this engine",p.name,p.origin.importer,p.origin.path) + case f.fast_here: + report_line(&report.missing,"package %s: only a build with Fast textures is here, and Final is required: a machine with its source (%s) must ingest it with this engine, or a peer that has its Final build (import key %s) must give it",p.name,p.origin.path,store.hex(key,&digits)) + case: + report_line(&report.missing,"package %s: no build of it for this engine (import key %s) is here or at a peer: a machine with its source (%s) must ingest it with this engine",p.name,store.hex(key,&digits),p.origin.path) + } + return + } + select:=store.hex(asset.build_select(entry.manifest,&p),&digits) + previous,_:=store.ref_get(s,asset.REF_BUILDS,select) + if previous!=f.hash { + if ref_error:=store.ref_set(s,asset.REF_BUILDS,select,f.hash);ref_error!=.None { + report_line(&report.failures,"package %s: the ref %s/%s could not be set in the store at %s: %v",p.name,asset.REF_BUILDS,select,s.root,ref_error) + return + } + } + if !follow && previous==f.hash {return} + // A model's prefab is written from its source in the game's types: the import does it, with the documents. + if prefab && origin_here && plugins_world(l)!=nil { + reingest(l,top,&entry,&p,source,documents,prefab,"it is followed into the project",report) + return + } + documents_write(l,top,&entry,&p,&f.build,documents,report) + if prefab && follow { + if _,has:=asset.registry_find(l.registry,prefab_id(&p));!has { + report_line(&report.warnings,"package %s: its prefab is written from its source (%s) in the types of the game: add it on a machine that has the source, with the project's editor built",p.name,p.origin.path) + } + } +} + +// The types of the game of the library's root, read once. +@(private="file") +plugins_world :: proc(l:^Library_State) -> ^core.World_Info { + plugins_read(l) + return l.plugins.world +} + +// Writes or follows the documents of the build b of p into documents and pins +// what their files hold as it leaves them in entry. +@(private="file") +documents_write :: proc(l:^Library_State,top:string,entry:^asset.Project_Package,p:^asset.Package,b:^asset.Build,documents:string,report:^Library_Report) { + changed:=importer.Changed.Discard if l.options.changed==.Discard else .Keep + written,error:=importer.documents_follow(l.options.store,p,b,documents,l.registry,entry.written[:],changed,plugin_kinds(l),l.temp) + if error.code!=.None { + report_line(&report.failures,"package %s: %s",p.name,error.message) + return + } + recorded:=make([dynamic]asset.Package_Written,l.temp) + for w in written { + if w.warning!="" {report_line(&report.warnings,"package %s: %s",p.name,w.warning)} + if w.hash!=0 {append(&recorded,asset.Package_Written{w.id,w.hash})} + } + written_pin(l,top,entry,recorded[:],p,report) +} + +// The extensions of the kinds of the active plugins of the root of the +// library, which a new file of a document of theirs takes. +@(private="file") +plugin_kinds :: proc(l:^Library_State) -> []importer.Kind_Extension { + if l.options.root=="" {return nil} + p,error:=project.open(l.options.root,project.engine_root_find("",context.temp_allocator),context.temp_allocator) + if error.code!=.None {return nil} + out:=make([dynamic]importer.Kind_Extension,l.temp) + for index in p.active { + for &k in p.plugins[index].manifest.asset_kinds { + extension,code:=base.text_get(k.extension[:]),base.text_get(k.kind[:]) + if extension!="" && code!="" {append(&out,importer.Kind_Extension{asset.fourcc(code),strings.clone(extension,l.temp)})} + } + } + return out[:] +} + +// Pins in entry what the files of its documents hold as written, when it +// differs from what it records. +@(private="file") +written_pin :: proc(l:^Library_State,top:string,entry:^asset.Project_Package,written:[]asset.Package_Written,p:^asset.Package,report:^Library_Report) { + // A document of the package this pass did not write holds what it held, so + // its record stays: a model's prefab, which only a library with the game's + // types writes, is not forgotten by a refresh without them. + merged:=make([dynamic]asset.Package_Written,context.temp_allocator) + append(&merged,..written) + records: for old in entry.written { + for w in written { + if w.id==old.id {continue records} + } + for &a in p.assets { + if a.id==old.id { + append(&merged,old) + continue records + } + } + } + written:=written_list(merged[:]) + same:=len(written)==len(entry.written) + for w,at in written { + if !same {break} + same=entry.written[at]==w + } + if same {return} + entry.written=written + if pin_error:=project_pin(top,l.options.store,entry^);pin_error!=.None { + report_line(&report.failures,"package %s: %s/%s could not be written: %v",p.name,top,asset.PACKAGES_FILE,pin_error) + } +} + +// Whether what the source of p, here at source, is made from now is not what +// p was ingested from: its bytes, those of its inputs, or the code of the +// plugin that imports it whole. +@(private="file") +source_changed :: proc(l:^Library_State,p:^asset.Package,source:string) -> bool { + settings,valid:=importer.settings_parse(p.settings) + if !valid {return false} + code:u64 + if r,known:=reader_of(l,source);known && r.claimed {code=r.whole.code} + hashes:=importer.source_hashes_load(l.options.store,source,l.temp) + defer importer.source_hashes_save(l.options.store,source,&hashes) + return importer.identity_of(p.id,source,p.inputs[:],settings,code,&hashes)!=p.identity +} + +// Ingests the source of p again, here at source, keeping its ids, and pins +// the manifest it gives; a model is imported with the project's folder, which follows its +// documents and writes its prefab, and any other follows from its build. +@(private="file") +reingest :: proc(l:^Library_State,top:string,entry:^asset.Project_Package,p:^asset.Package,source,documents:string,prefab:bool,why:string,report:^Library_Report) { + if l.options.progress.procedure!=nil {fmt.wprintfln(l.options.progress,"importing %s: %s",source,why)} + started:=time.tick_now() + model:=prefab && plugins_world(l)!=nil + changed:=importer.Changed.Discard if l.options.changed==.Discard else .Keep + written,error:=library_ingest(l,source,entry.manifest,nil,documents if model else "",entry.written[:],changed,l.temp) + if error.code!=.None { + report_line(&report.failures,"package %s: %s",p.name,error.message) + return + } + defer importer.report_destroy(&written) + report.ingested+=1 + for warning in written.warnings {report_line(&report.warnings,"package %s: %s",p.name,warning)} + if l.options.progress.procedure!=nil {fmt.wprintfln(l.options.progress," %s: %d assets, %.1f s",p.name,len(written.assets),time.duration_seconds(time.tick_since(started)))} + if written.manifest!=entry.manifest { + entry.manifest=written.manifest + if pin_error:=project_pin(top,l.options.store,entry^);pin_error!=.None { + report_line(&report.failures,"package %s: %s/%s could not be written: %v",p.name,top,asset.PACKAGES_FILE,pin_error) + return + } + } + if model { + written_pin(l,top,entry,written.written[:],p,report) + return + } + again,read_error:=asset.package_get(l.options.store,written.manifest,l.temp) + b,build_error:=asset.build_get(l.options.store,written.build,l.temp) + if read_error!=.None || build_error!=.None { + report_line(&report.failures,"package %s: the store at %s does not read what its import put (%v, %v)",p.name,l.options.store.root,read_error,build_error) + return + } + documents_write(l,top,entry,&again,&b,documents,report) + if prefab { + if _,has:=asset.registry_find(l.registry,prefab_id(&again));!has { + report_line(&report.warnings,"package %s: its prefab is written in the types of the game, which the library of the project's editor has: %s",p.name,"build the project's editor, then refresh") + } + } +} + +// Whether p is a model whose prefab its import writes from the source: one a +// format reads, with a scene its build has no baseline of. +@(private="file") +model_prefab_of :: proc(l:^Library_State,p:^asset.Package) -> bool { + if prefab_id(p)=={} {return false} + for format in FORMATS { + if format.name==p.origin.importer {return true} + } + plugins_read(l) + for format in l.formats { + if format.name==p.origin.importer {return true} + } + return false +} + +@(private="file") +prefab_id :: proc(p:^asset.Package) -> base.Asset_Id { + for &a in p.assets { + if a.kind==asset.KIND_SCENE {return a.id} + } + return {} +} + +@(private="file") +report_begin :: proc(report:^Library_Report,allocator:mem.Allocator) { + report.allocator=allocator + report.missing.allocator,report.failures.allocator,report.warnings.allocator=allocator,allocator,allocator +} + +@(private="file") +report_line :: proc(lines:^[dynamic]string,format:string,args:..any) { + append(lines,fmt.aprintf(format,..args,allocator=lines.allocator)) +} diff --git a/cook/import/sources/library_test.odin b/cook/import/sources/library_test.odin new file mode 100644 index 000000000..ec9ce07d6 --- /dev/null +++ b/cook/import/sources/library_test.odin @@ -0,0 +1,535 @@ +package sources + +import "core:fmt" +import "core:os" +import "core:path/filepath" +import "core:strings" +import "core:sync" +import "core:testing" +import "core:thread" +import "mirai:asset" +import "mirai:cook/cache" +import "mirai:cook/prefab" +import importer "mirai:cook/import" +import "mirai:std/base" +import "mirai:store" + +// The files under folder, by their paths under it, in the temp allocator. +files_of :: proc(folder:string) -> map[string]bool { + files:=make(map[string]bool,context.temp_allocator) + todo:=make([dynamic]string,context.temp_allocator) + append(&todo,folder) + for next:=0;next (listed:map[string]asset.Package_Listed) { + listed=make(map[string]asset.Package_Listed,context.temp_allocator) + list,_,error:=asset.packages_load(root,context.temp_allocator) + if !testing.expect_value(t,error,asset.Error.None,loc=loc) {return} + resolved,resolve_error:=asset.packages_resolve(s,root,&list,context.temp_allocator) + if !testing.expect_value(t,resolve_error,asset.Error.None,loc=loc) {return} + testing.expectf(t,len(resolved.problems)==0,"%v",resolved.problems[:],loc=loc) + for l in resolved.listed {listed[l.path]=l} + return +} + +// Every ref of the namespaces of builds of s, removed: what an engine whose +// versions changed finds of them. +@(private="file") +builds_forget :: proc(s:^store.Store) { + for namespace in ([3]string{asset.REF_IMPORTS,asset.REF_IMPORTS_FAST,asset.REF_BUILDS}) { + refs,_:=store.ref_list(s,namespace,context.temp_allocator) + for ref in refs {store.ref_remove(s,namespace,ref.name)} + } +} + +@(private="file") +report_clean :: proc(t:^testing.T,report:Library_Report,loc:=#caller_location) -> bool { + return testing.expectf(t,len(report.failures)==0 && len(report.missing)==0,"failures %v, missing %v",report.failures[:],report.missing[:],loc=loc) +} + +@(private="file") +ingested :: proc(t:^testing.T,source:string,options:Library_Options,loc:=#caller_location) -> (report:importer.Import_Report,ok:bool) { + error:importer.Error + report,error=ingest(source,options,allocator=context.temp_allocator) + ok=testing.expectf(t,error.code==.None,"%s: %s",source,error.message,loc=loc) + return +} + +// A project pins its packages by manifest: a peer with the project and none +// of its sources fetches their builds from a store that has them, lists and +// loads them, and imports nothing. Without such a store it says, for each +// package, what gives it, and neither removes a file nor builds anything. +@(test) +a_peer_without_sources_lists_and_loads_and_nothing_is_removed_or_built :: proc(t:^testing.T) { + top:=base.test_folder(t,"library-peer") + if top=="" {return} + context.user_ptr=prefab.import_world(t) + downloads,game:=fmt.tprintf("%s/downloads",top),fmt.tprintf("%s/game",top) + os.make_directory_all(downloads) + os.make_directory_all(fmt.tprintf("%s/assets",game)) + box:=box_source(downloads,"box.gltf","Wood","Paint") + sign:=fmt.tprintf("%s/sign.png",downloads) + _=os.write_entire_file(sign,png_bytes(8,8,{200,40,40})) + here:=test_store(t,fmt.tprintf("%s/here",top)) + if here==nil {return} + options:=Library_Options{store=here,root=game,plugins=test_library} + for source in ([2]string{box,sign}) { + report,ok:=ingested(t,source,options) + if !ok {return} + added:=project_add(game,report.manifest,"props",nil,options,context.temp_allocator) + if !report_clean(t,added) {return} + } + listed:=listed_of(t,here,game) + _,has_mesh:=listed["props/Box.mesh"] + _,has_sign:=listed["props/sign.tex"] + testing.expectf(t,has_mesh && has_sign,"the cache of both is listed under their folder: %v",listed) + + // A peer: the project, not its sources, and a store of its own. + testing.expect(t,os.rename(downloads,fmt.tprintf("%s/moved",top))==nil) + peer:=fmt.tprintf("%s/peer",top) + folder_copy(game,peer) + there:=test_store(t,fmt.tprintf("%s/there",top)) + if there==nil {return} + alone:=options + alone.store,alone.root=there,peer + before:=files_of(peer) + lacking:=project_refresh(peer,alone,context.temp_allocator) + testing.expectf(t,len(lacking.missing)==2 && len(lacking.failures)==0 && lacking.ingested==0 && lacking.fetched==0,"%v",lacking) + list,_,_:=asset.packages_load(peer,context.temp_allocator) + for &entry in list.packages { + // The project carries its manifests: each package is named by its name. + p,_:=asset.package_get(here,entry.manifest,context.temp_allocator) + named:=false + for line in lacking.missing {named||=strings.contains(line,fmt.tprintf("package %s:",p.name))} + testing.expectf(t,named,"the package %s is named: %v",p.name,lacking.missing[:]) + } + builds,_:=store.ref_list(there,asset.REF_BUILDS,context.temp_allocator) + testing.expect(t,len(builds)==0,"nothing is built") + after:=files_of(peer) + testing.expect_value(t,len(after),len(before)) + for file in before {testing.expectf(t,after[file],"%s stays",file)} + + // With a store that has them, the builds are fetched: listed where the project lists them, loaded. + // The builds of both packages are asked together, in one call of the source. + Counted :: struct {inner:cache.Source,calls,names:int} + counted:=Counted{inner=cache.store_source(here)} + fetching:=alone + fetching.source={user=&counted,fetch=proc(user:rawptr,into:^store.Store,namespace:string,names:[]string,given:[]bool) { + c:=(^Counted)(user) + if namespace==asset.REF_IMPORTS {c.calls,c.names=c.calls+1,c.names+len(names)} + c.inner.fetch(c.inner.user,into,namespace,names,given) + }} + fetched:=project_refresh(peer,fetching,context.temp_allocator) + testing.expectf(t,fetched.fetched==2 && fetched.ingested==0,"%v",fetched) + testing.expectf(t,counted.calls==1 && counted.names==2,"%d calls for %d builds",counted.calls,counted.names) + if !report_clean(t,fetched) {return} + seen:=listed_of(t,there,peer) + testing.expect_value(t,len(seen),len(listed)) + for path,l in listed { + other,found:=seen[path] + if !testing.expectf(t,found && other.id==l.id && other.state==.Built,"%s is listed with its id",path) {continue} + a:asset.Asset + if testing.expectf(t,asset.asset_open_object(&a,there,other.object,context.temp_allocator)==.None,"%s loads",path) { + testing.expect_value(t,a.blob.header.id,l.id) + asset.asset_close(&a) + } + } + for file in before {testing.expectf(t,os.exists(fmt.tprintf("%s%s",peer,file)),"%s stays",file)} + again:=project_refresh(peer,alone,context.temp_allocator) + testing.expectf(t,again.fetched==0 && again.ingested==0,"then nothing is asked for: %v",again) + report_clean(t,again) +} + +// A source outside any project is ingested where it is: nothing is written +// beside it, two projects that add it list the same blobs, ingesting it again +// imports nothing, and once it changed it is another manifest of the same +// package with the same ids, which a project follows on its refresh. +@(test) +a_source_outside_any_project_ingests_and_two_projects_use_the_same_blobs :: proc(t:^testing.T) { + top:=base.test_folder(t,"library-outside") + if top=="" {return} + context.user_ptr=prefab.import_world(t) + downloads:=fmt.tprintf("%s/downloads",top) + os.make_directory_all(downloads) + box:=box_source(downloads,"box.gltf","Wood","Paint") + s:=test_store(t,fmt.tprintf("%s/store",top)) + if s==nil {return} + options:=Library_Options{store=s,plugins=test_library} + first,ok:=ingested(t,box,options) + if !ok {return} + testing.expect(t,len(first.assets)>0 && first.prefab_path=="","its cache and documents are blobs; the prefab is a project's") + testing.expect_value(t,len(files_of(downloads)),2) + again,again_ok:=ingested(t,box,options) + testing.expect(t,again_ok && len(again.assets)==0 && again.manifest==first.manifest && again.package_id==first.package_id,"current: nothing is imported again") + + projects:=[2]string{fmt.tprintf("%s/one",top),fmt.tprintf("%s/two",top)} + lists:[2]map[string]asset.Package_Listed + for project,at in projects { + os.make_directory_all(fmt.tprintf("%s/assets",project)) + o:=options + o.root=project + if !report_clean(t,project_add(project,first.manifest,"props",nil,o,context.temp_allocator)) {return} + lists[at]=listed_of(t,s,project) + p,_:=asset.package_get(s,first.manifest,context.temp_allocator) + r:asset.Registry + asset.registry_init(&r,fmt.tprintf("%s/assets",project),"",context.temp_allocator) + asset.registry_scan(&r) + for a in p.assets { + if asset.import_cached(a.kind) {continue} + _,found:=asset.registry_find(&r,a.id) + testing.expectf(t,found,"%s: the document %s, the prefab among them, is a file of the project with its id",project,a.name) + } + } + testing.expect(t,len(lists[0])>0 && len(lists[0])==len(lists[1])) + for path,l in lists[0] {testing.expectf(t,lists[1][path].object==l.object,"%s is the same object for both",path)} + + box_source(downloads,"box.gltf","Wood","Paint","0.9") + changed,changed_ok:=ingested(t,box,options) + if !changed_ok {return} + testing.expect(t,changed.manifest!=first.manifest && changed.package_id==first.package_id,"a changed source is another manifest of its package") + was,_:=asset.package_get(s,first.manifest,context.temp_allocator) + now,_:=asset.package_get(s,changed.manifest,context.temp_allocator) + testing.expect_value(t,len(now.assets),len(was.assets)) + for a in was.assets { + kept:=false + for b in now.assets {kept||=b.id==a.id && b.kind==a.kind && b.name==a.name} + testing.expectf(t,kept,"%s keeps its id",a.name) + } + o:=options + o.root=projects[0] + followed:=project_refresh(projects[0],o,context.temp_allocator) + report_clean(t,followed) + list,_,_:=asset.packages_load(projects[0],context.temp_allocator) + testing.expect(t,len(list.packages)==1 && list.packages[0].manifest==changed.manifest,"the project pins what its source is now") +} + +// When the versions of the engine change, no build of a package is current: +// a machine with its source ingests it again, and one without it says which +// package needs it, by name. A build with Fast textures is not one a cook +// takes: where Final is required and only Fast is here, that is said too. +@(test) +an_engine_change_builds_again_where_the_source_is_and_is_reported_by_name_elsewhere :: proc(t:^testing.T) { + top:=base.test_folder(t,"library-engine") + if top=="" {return} + downloads,game:=fmt.tprintf("%s/downloads",top),fmt.tprintf("%s/game",top) + os.make_directory_all(downloads) + os.make_directory_all(fmt.tprintf("%s/assets",game)) + sign:=fmt.tprintf("%s/billboard.png",downloads) + _=os.write_entire_file(sign,png_bytes(8,8,{200,40,40})) + here:=test_store(t,fmt.tprintf("%s/here",top)) + if here==nil {return} + options:=Library_Options{store=here,root=game} + first,ok:=ingested(t,sign,options) + if !ok || !report_clean(t,project_add(game,first.manifest,"",nil,options,context.temp_allocator)) {return} + current:=project_refresh(game,options,context.temp_allocator) + testing.expectf(t,current.ingested==0 && current.fetched==0,"a current package is left alone: %v",current) + + builds_forget(here) + rebuilt:=project_refresh(game,options,context.temp_allocator) + testing.expectf(t,rebuilt.ingested==1,"%v",rebuilt) + report_clean(t,rebuilt) + _,has:=listed_of(t,here,game)["billboard.tex"] + testing.expect(t,has) + + testing.expect(t,os.rename(downloads,fmt.tprintf("%s/moved",top))==nil) + builds_forget(here) + lacking:=project_refresh(game,options,context.temp_allocator) + testing.expectf(t,lacking.ingested==0 && len(lacking.missing)==1 && strings.contains(lacking.missing[0],"package billboard") && strings.contains(lacking.missing[0],"must ingest it"),"%v",lacking.missing[:]) + _,listed:=listed_of(t,here,game)["billboard.tex"] + testing.expect(t,listed,"what is not built is still listed, and nothing is removed") + + // Fast only, where Final is required: an image no Final of which was ever built. + poster:=fmt.tprintf("%s/poster.png",top) + _=os.write_entire_file(poster,png_bytes(8,8,{30,60,220})) + fast:=options + fast.fast_textures=true + quick,quick_ok:=ingested(t,poster,fast) + if !quick_ok || !report_clean(t,project_add(game,quick.manifest,"",nil,fast,context.temp_allocator)) {return} + testing.expect(t,!quick.final,"a refresh that takes Fast builds Fast") + testing.expect(t,os.remove(poster)==nil) + final:=project_refresh(game,options,context.temp_allocator) + testing.expectf(t,len(final.missing)==2 && strings.contains(strings.join(final.missing[:],"\n",context.temp_allocator),"package poster: only a build with Fast textures is here, and Final is required"),"%v",final.missing[:]) + testing.expectf(t,len(project_refresh(game,fast,context.temp_allocator).missing)==1,"with Fast taken only the billboard, whose source is gone, is missing") +} + +// A root carries the manifests it pins: a copy of it elsewhere with its +// sources and an empty store refreshes into the same ids, manifests, build +// keys and blobs; a copy without its sources lists them not built and says so +// by package. Unpinning removes the manifest's file. +@(test) +a_copy_of_a_root_with_its_sources_and_an_empty_store_builds_the_same :: proc(t:^testing.T) { + top:=base.test_folder(t,"library-clone") + if top=="" {return} + context.user_ptr=prefab.import_world(t) + game:=fmt.tprintf("%s/game",top) + props:=fmt.tprintf("%s/assets/props",game) + os.make_directory_all(props) + box:=box_source(props,"box.gltf","Wood","Paint") + sign:=fmt.tprintf("%s/sign.png",props) + _=os.write_entire_file(sign,png_bytes(8,8,{200,40,40})) + here:=test_store(t,fmt.tprintf("%s/here",top)) + if here==nil {return} + options:=Library_Options{store=here,root=game,plugins=test_library} + keys:=make([dynamic]store.Hash,context.temp_allocator) + for source in ([2]string{box,sign}) { + report,added,error:=project_ingest(game,source,"props",options,allocator=context.temp_allocator) + if !testing.expectf(t,error.code==.None,"%s",error.message) || !report_clean(t,added) {return} + append(&keys,report.key) + } + pinned,_,_:=asset.packages_load(game,context.temp_allocator) + testing.expect_value(t,len(pinned.packages),2) + for &entry in pinned.packages { + text,read_error:=os.read_entire_file(asset.package_file(game,entry.id),context.temp_allocator) + testing.expectf(t,read_error==nil && store.address_of_bytes(text)==entry.manifest,"the root keeps the manifest it pins of %v",entry.id) + p,_:=asset.package_get(here,entry.manifest,context.temp_allocator) + testing.expectf(t,!strings.has_prefix(p.origin.path,"/") && strings.has_prefix(p.origin.path,"assets/props/"),"a source of the root is named from it: %s",p.origin.path) + } + blobs_of :: proc(t:^testing.T,s:^store.Store,root:string) -> map[string]store.Hash { + out:=make(map[string]store.Hash,context.temp_allocator) + for path,l in listed_of(t,s,root) { + out[path]=l.object + } + return out + } + before:=blobs_of(t,here,game) + + clone:=fmt.tprintf("%s/clone",top) + folder_copy(game,clone) + fresh:=test_store(t,fmt.tprintf("%s/fresh",top)) + if fresh==nil {return} + again:=options + again.store,again.root=fresh,clone + refreshed:=project_refresh(clone,again,context.temp_allocator) + testing.expectf(t,refreshed.ingested==2,"%v",refreshed) + if !report_clean(t,refreshed) {return} + cloned,_,_:=asset.packages_load(clone,context.temp_allocator) + testing.expect_value(t,len(cloned.packages),len(pinned.packages)) + for entry,at in cloned.packages { + testing.expectf(t,entry.id==pinned.packages[at].id && entry.manifest==pinned.packages[at].manifest,"the same package and manifest: %v",entry) + } + for key in keys { + digits:[64]u8 + _,ref_error:=store.ref_get(fresh,asset.REF_IMPORTS,store.hex(key,&digits)) + testing.expectf(t,ref_error==.None,"the build has the key it had where it was ingested") + } + after:=blobs_of(t,fresh,clone) + testing.expect_value(t,len(after),len(before)) + for path,hash in before {testing.expectf(t,after[path]==hash,"%s is the same blob",path)} + + // Without its sources nor a store that has them: listed, not built, said by package. + bare:=fmt.tprintf("%s/bare",top) + folder_copy(game,bare) + for file in ([]string{"box.gltf","sign.png","wood.png"}) {os.remove(fmt.tprintf("%s/assets/props/%s",bare,file))} + empty:=test_store(t,fmt.tprintf("%s/empty",top)) + if empty==nil {return} + alone:=options + alone.store,alone.root=empty,bare + lacking:=project_refresh(bare,alone,context.temp_allocator) + testing.expectf(t,len(lacking.missing)==2 && len(lacking.failures)==0 && lacking.ingested==0,"%v",lacking) + resolved,_:=asset.root_packages(bare,empty.root,context.temp_allocator) + testing.expectf(t,len(resolved.listed)==len(before) && len(resolved.problems)==2,"%v",resolved.problems) + for l in resolved.listed {testing.expectf(t,l.state==.Not_Built,"%s is not built",l.path)} + for entry in pinned.packages { + id:[base.ASSET_ID_TEXT]u8 + named:=false + for line in resolved.problems {named||=strings.contains(line,base.asset_id_format(entry.id,id[:]))} + testing.expectf(t,named,"%v is named: %v",entry.id,resolved.problems) + } + + unpinned,unpin_error:=project_unpin(bare,pinned.packages[0].id) + testing.expect(t,unpinned && unpin_error==.None,"unpinning removes the entry") + left,_,_:=asset.packages_load(bare,context.temp_allocator) + testing.expect(t,len(left.packages)==1 && left.packages[0].id==pinned.packages[1].id,"the other stays pinned") + testing.expect(t,!os.exists(asset.package_file(bare,pinned.packages[0].id)),"unpinning removes its manifest") + testing.expect(t,os.exists(asset.package_file(bare,pinned.packages[1].id))) +} + +// Following packages into a project writes the same bytes from any empty +// store, and a refresh that would write what is there writes nothing: no +// document, prefab or record of the root is written again. +@(test) +refreshing_from_two_empty_stores_writes_the_same_root_once :: proc(t:^testing.T) { + top:=base.test_folder(t,"library-idempotent") + if top=="" {return} + context.user_ptr=prefab.import_world(t) + game:=fmt.tprintf("%s/game",top) + props:=fmt.tprintf("%s/assets/props",game) + os.make_directory_all(props) + box:=box_source(props,"box.gltf","Wood","Paint") + sign:=fmt.tprintf("%s/sign.png",props) + _=os.write_entire_file(sign,png_bytes(8,8,{200,40,40})) + first:=test_store(t,fmt.tprintf("%s/first",top)) + if first==nil {return} + options:=Library_Options{store=first,root=game,plugins=test_library} + for source in ([2]string{box,sign}) { + _,added,error:=project_ingest(game,source,"props",options,allocator=context.temp_allocator) + if !testing.expectf(t,error.code==.None,"%s",error.message) || !report_clean(t,added) {return} + } + File :: struct {bytes:string,inode:u128} + state :: proc(root:string) -> map[string]File { + out:=make(map[string]File,context.temp_allocator) + for file in files_of(root) { + path:=fmt.tprintf("%s%s",root,file) + data,_:=os.read_entire_file(path,context.temp_allocator) + info,_:=os.stat(path,context.temp_allocator) + out[file]={string(data),info.inode} + } + return out + } + ingested:=state(game) + testing.expect(t,len(ingested)>4,"the sources, their documents, the prefab and the records") + for at in 0..<2 { + s:=test_store(t,fmt.tprintf("%s/empty%d",top,at)) + if s==nil {return} + again:=options + again.store=s + refreshed:=project_refresh(game,again,context.temp_allocator) + testing.expectf(t,refreshed.ingested==2,"%v",refreshed) + if !report_clean(t,refreshed) {return} + now:=state(game) + testing.expect_value(t,len(now),len(ingested)) + for file,was in ingested { + is:=now[file] + testing.expectf(t,is.bytes==was.bytes,"refresh %d: %s has the bytes it had",at,file) + testing.expectf(t,is.inode==was.inode,"refresh %d: %s is not written again",at,file) + } + } +} + +// What a project's packages.mirai or the store says that is not a record of +// this version is reported, never a crash nor a default. +@(test) +hostile_packages_and_manifests_are_reported_without_a_crash :: proc(t:^testing.T) { + top:=base.test_folder(t,"library-hostile") + if top=="" {return} + game:=fmt.tprintf("%s/game",top) + os.make_directory_all(fmt.tprintf("%s/assets",game)) + s:=test_store(t,fmt.tprintf("%s/store",top)) + if s==nil {return} + options:=Library_Options{store=s,root=game} + nonsense:=transmute([]u8)string(`{"version": 1, "id": "nonsense"}`) + garbage:=store.address_of_bytes(nonsense) + { + digits:[64]u8 + object,_:=asset.record_put(s,nonsense) + store.ref_set(s,asset.REF_MANIFESTS,store.hex(garbage,&digits),object) + } + other:=asset.Package{id={9,9},name="other",origin={path="/nowhere/other.png",importer="image"}} + other_manifest,_:=asset.package_put(s,&other) + lost:=asset.Package{id={7,7},name="lost",origin={path="/nowhere/lost.png",importer="image"}} + lost_manifest,_:=asset.package_put(s,&lost) + list:asset.Packages + list.packages=make([dynamic]asset.Project_Package,context.temp_allocator) + append(&list.packages,asset.Project_Package{id={1,1},manifest=garbage},asset.Project_Package{id={2,2},manifest=other_manifest},asset.Project_Package{id={7,7},manifest=lost_manifest,folder="lost"}) + testing.expect_value(t,asset.packages_save(game,&list),asset.Error.None) + report:=project_refresh(game,options,context.temp_allocator) + testing.expectf(t,report.packages==3 && len(report.failures)==2 && len(report.missing)==1 && strings.contains(report.missing[0],"package lost"),"%v",report) + // A file of the root that is not the manifest its pin names. + os.make_directory_all(fmt.tprintf("%s/%s",game,asset.PACKAGE_FILES)) + _=os.write_entire_file(asset.package_file(game,{1,1}),transmute([]u8)string("not the pinned manifest")) + mismatched:=project_refresh(game,options,context.temp_allocator) + testing.expectf(t,len(mismatched.failures)==2 && strings.contains(strings.join(mismatched.failures[:],"\n",context.temp_allocator),"is not the manifest"),"%v",mismatched.failures[:]) + os.remove(asset.package_file(game,{1,1})) + + text,_:=os.read_entire_file(fmt.tprintf("%s/%s",game,asset.PACKAGES_FILE),context.temp_allocator) + work:=make([]u8,len(text),context.temp_allocator) + path:=fmt.tprintf("%s/%s",game,asset.PACKAGES_FILE) + refused:=0 + for length:=0;length0) + } + marks:="{}[]\":,0-9ef \x00\xff" + state:=u64(0x9e3779b97f4a7c15) + for _ in 0..<300 { + copy(work,text) + state=state*6364136223846793005+1442695040888963407 + work[int((state>>33)%u64(len(work)))]=marks[int((state>>20)%u64(len(marks)))] + _=os.write_entire_file(path,work) + _=project_refresh(game,options,context.temp_allocator) + } + testing.expectf(t,refused>0,"cut texts are refused: %d",refused) + testing.expect(t,!os.exists(fmt.tprintf("%s/assets/lost",game)),"nothing is written for what is not here") +} + +// Two that ingest one source at once take turns, and the second finds the +// work done. +@(test) +two_ingests_of_one_source_at_once_import_it_once :: proc(t:^testing.T) { + top:=base.test_folder(t,"library-lock") + if top=="" {return} + sign:=fmt.tprintf("%s/sign.png",top) + _=os.write_entire_file(sign,png_bytes(64,64,{20,140,40})) + Run :: struct {root,source:string,assets:int,manifest:store.Hash,failed:bool,ready:^sync.Barrier} + ready:sync.Barrier + sync.barrier_init(&ready,2) + runs:[2]Run + for &run in runs {run={root=fmt.tprintf("%s/store",top),source=sign,ready=&ready}} + work :: proc(run:^Run) { + s,error:=store.open(run.root) + if error!=.None {run.failed=true;return} + defer store.close(&s) + sync.barrier_wait(run.ready) + report,ingest_error:=ingest(run.source,{store=&s}) + defer importer.report_destroy(&report) + run.failed=ingest_error.code!=.None + run.assets,run.manifest=len(report.assets),report.manifest + } + threads:[2]^thread.Thread + for &run,at in runs {threads[at]=thread.create_and_start_with_poly_data(&run,work)} + for th in threads { + thread.join(th) + thread.destroy(th) + } + testing.expect(t,!runs[0].failed && !runs[1].failed && runs[0].manifest==runs[1].manifest && runs[0].manifest!={}) + testing.expectf(t,(runs[0].assets==0)!=(runs[1].assets==0),"one imports and the other finds it done: %d and %d assets",runs[0].assets,runs[1].assets) +} + +// A pin that cannot be saved leaves the root as it was: the pin before it +// still reads its manifest, whose file is not replaced. +@(test) +a_failed_pin_leaves_the_root_as_it_was :: proc(t:^testing.T) { + top:=base.test_folder(t,"library-pin-whole") + if top=="" {return} + s:=test_store(t,fmt.tprintf("%s/store",top)) + if s==nil {return} + defer store.close(s) + root:=fmt.tprintf("%s/game",top) + os.make_directory_all(root) + id:=base.Asset_Id{7,9} + first:=asset.Package{id=id,name="first"} + second:=asset.Package{id=id,name="second"} + m1,e1:=asset.package_put(s,&first) + m2,e2:=asset.package_put(s,&second) + if !testing.expect(t,e1==.None && e2==.None) {return} + pinned:=asset.Project_Package{id=id,manifest=m1} + if !testing.expect_value(t,project_pin(root,s,pinned),asset.Error.None) {return} + // A folder that leaves the assets: the list refuses it. + refused:=asset.Project_Package{id=id,manifest=m2,folder="../out"} + testing.expect(t,project_pin(root,s,refused)!=.None,"the pin is refused") + list,_,_:=asset.packages_load(root,context.temp_allocator) + testing.expect(t,len(list.packages)==1 && list.packages[0].manifest==m1,"the list is the one before") + p,load_error:=asset.package_load(s,root,&list.packages[0],context.temp_allocator) + testing.expectf(t,load_error==.None && p.name=="first","the pin before reads its manifest: %v",load_error) +} diff --git a/cook/import/sources/reimport_test.odin b/cook/import/sources/reimport_test.odin index 02f587ada..a458be28a 100644 --- a/cook/import/sources/reimport_test.odin +++ b/cook/import/sources/reimport_test.odin @@ -14,14 +14,16 @@ import "core:time" import stbi "vendor:stb/image" import "mirai:asset" import "mirai:core" +import "mirai:cook/cache" import importer "mirai:cook/import" import "mirai:cook/prefab" import "mirai:cook/import/gltf" +import "mirai:cook/textures" import "mirai:project" import "mirai:std/base" +import "mirai:store" // A box with two materials, the first with the image wood.png next to it. -@(private="file") box_source :: proc(folder,file:string,first,second:string,roughness:="0.5") -> string { vertices:=[15]f32{0,0,0,1,0,0,0,1,0,0,0,1,0,0,1} encoded,_:=base64.encode(slice.to_bytes(vertices[:]),allocator=context.temp_allocator) @@ -38,7 +40,7 @@ box_source :: proc(folder,file:string,first,second:string,roughness:="0.5") -> s @(private="file") box_import :: proc(t:^testing.T,source,root:string) -> (report:importer.Import_Report,ok:bool) { error:importer.Error - report,error=import_source(source,{world=prefab.import_world(t),root=root,textures=.Uncompressed}) + report,error=import_test(t,source,{world=prefab.import_world(t),root=root,textures=.Uncompressed}) ok=testing.expectf(t,error.code==.None,"%s: %s",source,error.message) return } @@ -63,10 +65,9 @@ slot_materials :: proc(t:^testing.T,report:^importer.Import_Report,types:^asset. return {view.slots[0].material,view.slots[1].material} } -// What an import of a project builds that nobody edits is of its imported -// folder, named by its id; what the editor edits is of its assets, in the -// folder of the import beside the source, found by its id wherever it was -// moved. +// What an import builds that nobody edits is an object of the store; what the +// editor edits is also a file of the project, in the folder of the import, +// found by its id wherever it was moved; the package lists them all. @(test) the_parts_of_an_import_name_its_prefab_and_keep_their_ids_where_they_were_moved :: proc(t:^testing.T) { root,assets:=test_project(t,"reimport-moved") @@ -87,18 +88,19 @@ the_parts_of_an_import_name_its_prefab_and_keep_their_ids_where_they_were_moved testing.expect(t,strings.has_prefix(wood.detail,"of its own: an island"),wood.detail) for written in first.assets { cached:=asset.import_cached(written.kind) - testing.expectf(t,strings.contains(written.path,cached?"/.mirai/imported/":"/assets/models/box/"),"%s: of the %s",written.path,cached?"imported folder":"assets") + testing.expectf(t,strings.contains(written.path,cached?"/store-objects/":"/assets/models/box/"),"%s: of the %s",written.path,cached?"store":"assets") } - sidecar,found,_:=importer.sidecar_load(source,context.temp_allocator) - testing.expect(t,found && sidecar.folder=="box" && len(sidecar.outputs)==len(first.assets),"the sidecar next to the source lists what it gives and where") - for output in sidecar.outputs { - written,listed:=written_of(&first,output.kind,output.kind==asset.KIND_SCENE?"box":output.name) - testing.expectf(t,listed && written.id==output.id && output.part==(output.kind==asset.KIND_MESH),"%s is listed with its id: %v",output.name,output) + s:=test_store(t,fmt.tprintf("%s/store",root)) + p,read_error:=asset.package_get(s,first.manifest,context.temp_allocator) + testing.expect(t,read_error==.None && len(p.assets)==len(first.assets),"the package lists what the source gives") + for a in p.assets { + written,listed:=written_of(&first,a.kind,a.kind==asset.KIND_SCENE?"box":a.name) + testing.expectf(t,listed && written.id==a.id && a.part==(a.kind==asset.KIND_MESH),"%s is listed with its id: %v",a.name,a) } a,loaded:=read_asset(t,prefab,&types) if loaded { info,error:=asset.import_info(&a,&types) - testing.expect(t,error==.None && info!=nil && string_of(info.source[:])=="assets/models/box.gltf" && .Cooked in a.blob.header.flags,"the prefab records its import") + testing.expect(t,error==.None && info!=nil && string_of(info.source[:])=="box.gltf" && .Cooked in a.blob.header.flags,"the prefab records its import") asset.asset_close(&a) } @@ -119,25 +121,26 @@ the_parts_of_an_import_name_its_prefab_and_keep_their_ids_where_they_were_moved testing.expect_value(t,len(again.warnings),0) } -// The ids of an import are those its sidecar lists: a source moved, renamed -// or changed keeps them as long as its sidecar goes with it, and a copy of -// the source alone is another import. +// The ids of an import are those of its package: a source moved, renamed or +// changed keeps them with the manifest its project pins, and a copy of the +// source is another package. @(test) -a_source_keeps_the_ids_of_its_import_with_its_sidecar_and_a_copy_without_it_is_another_import :: proc(t:^testing.T) { +a_source_keeps_the_ids_of_its_package_and_a_copy_is_another_package :: proc(t:^testing.T) { root,assets:=test_project(t,"reimport-renamed") if root=="" {return} os.make_directory_all(fmt.tprintf("%s/props",assets)) source:=box_source(assets,"box.gltf","Wood","Paint") first,ok:=box_import(t,source,root);if !ok {return};defer importer.report_destroy(&first) - // Moved, renamed and changed, with its sidecar. + // Moved, renamed and changed, imported with the manifest the project pins. crate:=box_source(fmt.tprintf("%s/props",assets),"crate.gltf","Wood","Paint","0.8") - testing.expect(t,os.rename(fmt.tprintf("%s.import",source),fmt.tprintf("%s.import",crate))==nil && os.remove(source)==nil) - moved,moved_ok:=box_import(t,crate,root);if !moved_ok {return};defer importer.report_destroy(&moved) - testing.expect_value(t,moved.prefab,first.prefab) + testing.expect(t,os.remove(source)==nil) + moved,moved_error:=import_test(t,crate,{world=prefab.import_world(t),root=root,textures=.Uncompressed,manifest=first.manifest}) + if !testing.expectf(t,moved_error.code==.None,"%s",moved_error.message) {return};defer importer.report_destroy(&moved) + testing.expect(t,moved.prefab==first.prefab && moved.package_id==first.package_id) for written in first.assets { same,found:=written_of(&moved,written.kind,written.kind==asset.KIND_SCENE?"crate":written.name) - // A file of the imported folder is named by what it is built from, which changed. + // A document is followed in its file; a blob is another one, made from what changed. testing.expectf(t,found && same.id==written.id && (same.path==written.path || asset.import_cached(written.kind)),"%s keeps its id and its file",written.name) } testing.expect_value(t,len(moved.warnings),0) @@ -145,38 +148,56 @@ a_source_keeps_the_ids_of_its_import_with_its_sidecar_and_a_copy_without_it_is_a scene,_:=written_of(&moved,asset.KIND_SCENE,"crate") if a,loaded:=read_asset(t,scene,&types);loaded { info,_:=asset.import_info(&a,&types) - testing.expect(t,info!=nil && string_of(info.source[:])=="assets/props/crate.gltf","the import names the source where it is now") + testing.expect(t,info!=nil && string_of(info.source[:])=="crate.gltf","the import names the source by its name in its package") asset.asset_close(&a) } - // The same bytes without the sidecar: an import of its own, which takes nothing of the other. + // The same bytes elsewhere: a package of its own, which takes nothing of the other. copy:=box_source(assets,"copy.gltf","Wood","Paint","0.8") - second,second_error:=import_source(copy,{world=prefab.import_world(t),root=root,textures=.Uncompressed}) + second,second_error:=import_test(t,copy,{world=prefab.import_world(t),root=root,textures=.Uncompressed}) if !testing.expectf(t,second_error.code==.None,"%s",second_error.message) {return};defer importer.report_destroy(&second) - testing.expect(t,second.prefab!={} && second.prefab!=first.prefab,"the copy writes a prefab of its own") + testing.expect(t,second.prefab!={} && second.prefab!=first.prefab && second.package_id!=first.package_id,"the copy writes a prefab of its own") for written in second.assets { for other in first.assets {testing.expectf(t,other.id!=written.id,"%s has an id of its own",written.path)} } } -// A document an import writes over that changed since it wrote it (the -// prefab of a model with an entity added) stops an import that was not told -// what to do with it, which names it and writes nothing; a sync leaves it as -// it is, builds the rest and warns; discard_changes writes over it. A change -// of its owner alone is none (a material, which follows its source, records -// nothing: see the tests of write_test.odin). +// A document an import writes over in a project that changed since it wrote +// it (the prefab of a model with an entity added) stops an import into the +// project that was not told what to do with it, which names it and writes +// nothing; a refresh leaves it as it is, builds the rest and warns; +// discard_changes writes over it. The pin records what each such file held +// as written; a change of its owner alone is none, and a material, which +// follows its source, records nothing. @(test) an_import_does_not_write_over_a_document_changed_since_it_wrote_it :: proc(t:^testing.T) { root,assets:=test_project(t,"reimport-changed") if root=="" {return} source:=box_source(assets,"box.gltf","Wood","Paint") - first,ok:=box_import(t,source,root);if !ok {return};defer importer.report_destroy(&first) + s:=test_store(t,fmt.tprintf("%s/store",root)) + if s==nil {return} + context.user_ptr=prefab.import_world(t) + options:=Library_Options{store=s,root=root,plugins=test_library,fast_textures=true} + settings:=importer.Import_Settings{textures=.Uncompressed} + into :: proc(t:^testing.T,root,source:string,options:Library_Options,settings:^importer.Import_Settings) -> (report:importer.Import_Report,error:importer.Error) { + added:Library_Report + report,added,error=project_ingest(root,source,"",options,settings,context.temp_allocator) + testing.expectf(t,len(added.failures)==0,"%v",added.failures) + return + } + first,first_error:=into(t,root,source,options,&settings) + if !testing.expectf(t,first_error.code==.None,"%s",first_error.message) {return} scene,_:=written_of(&first,asset.KIND_SCENE,"box") wood,_:=written_of(&first,asset.KIND_MATERIAL,"Wood") + list,_,_:=asset.packages_load(root,context.temp_allocator) + if !testing.expect_value(t,len(list.packages),1) {return} + recorded:=list.packages[0].written[:] + testing.expectf(t,importer.written_of(recorded,scene.id)==header_of(scene.path).content_hash && importer.written_of(recorded,wood.id)==0,"the pin records what the prefab holds, and nothing of a material: %v",recorded) data,_:=os.read_entire_file(scene.path,context.temp_allocator) testing.expect_value(t,asset.header_owner_set(data,wood.id),asset.Error.None) _=os.write_entire_file(scene.path,data) - again,again_ok:=box_import(t,source,root);if !again_ok {return};importer.report_destroy(&again) + _,again_error:=into(t,root,source,options,nil) + if !testing.expectf(t,again_error.code==.None,"a change of owner alone is none: %s",again_error.message) {return} types:asset.Types;asset.types_init(&types);defer asset.types_destroy(&types) edited:asset.Scene @@ -185,34 +206,32 @@ an_import_does_not_write_over_a_document_changed_since_it_wrote_it :: proc(t:^te testing.expect_value(t,asset.scene_save(&edited,scene.path),asset.Error.None) asset.scene_destroy(&edited) changed,_:=os.read_entire_file(scene.path,context.temp_allocator) - sidecar,_:=os.read_entire_file(fmt.tprintf("%s.import",source),context.temp_allocator) + pins,_:=os.read_entire_file(fmt.tprintf("%s/%s",root,asset.PACKAGES_FILE),context.temp_allocator) held :: proc(t:^testing.T,path:string,bytes:[]u8,why:string) { now,_:=os.read_entire_file(path,context.temp_allocator) testing.expectf(t,string(now)==string(bytes),"%s: %s",path,why) } - refused,error:=import_source(source,{world=prefab.import_world(t),root=root,textures=.Uncompressed}) - importer.report_destroy(&refused) + _,error:=into(t,root,source,options,&settings) testing.expectf(t,error.code==.Changed && contains(error.message,"assets/box/box.scene") && contains(error.message,"discard_changes"),"the import names what changed and how to go on: %v %s",error.code,error.message) held(t,scene.path,changed,"the import wrote nothing") - held(t,fmt.tprintf("%s.import",source),sidecar,"nor its sidecar") - - testing.expect(t,base.remove_all(fmt.tprintf("%s/.mirai",root))==nil) - context.user_ptr=prefab.import_world(t) - synced:=sync(root,{library=test_library},context.temp_allocator) - testing.expectf(t,synced.imported==1 && len(synced.failures)==0 && len(synced.warnings)==1 && contains(synced.warnings[0],"assets/box/box.scene"),"sync builds the rest and says what it left: %v",synced) - held(t,scene.path,changed,"sync leaves it as it is") - mesh,_:=written_of(&first,asset.KIND_MESH,"Box") - testing.expect(t,os.exists(mesh.path),"and builds the cache") - entries,_:=os.read_all_directory_by_path(fmt.tprintf("%s/.mirai/imported",root),context.temp_allocator) - for entry in entries {testing.expectf(t,!strings.contains(entry.name,".kept"),"%s: what was written of it goes",entry.name)} - - discarded,discard_error:=import_source(source,{world=prefab.import_world(t),root=root,textures=.Uncompressed,changed=.Discard}) + held(t,fmt.tprintf("%s/%s",root,asset.PACKAGES_FILE),pins,"nor its pin") + + // The source changes: the refresh ingests it again and leaves the prefab. + box_source(assets,"box.gltf","Wood","Paint","0.8") + refreshed:=project_refresh(root,options,context.temp_allocator) + testing.expectf(t,refreshed.ingested==1 && len(refreshed.failures)==0 && len(refreshed.warnings)==1 && contains(refreshed.warnings[0],"assets/box/box.scene"),"the refresh builds the rest and says what it left: %v",refreshed) + held(t,scene.path,changed,"the refresh leaves it as it is") + _,still:=into(t,root,source,options,nil) + testing.expectf(t,still.code==.Changed,"what the pin records is still what was written: %v",still.code) + + options.changed=.Discard + _,discard_error:=into(t,root,source,options,nil) if !testing.expectf(t,discard_error.code==.None,"%s",discard_error.message) {return} - importer.report_destroy(&discarded) testing.expect(t,!scene_named(t,&types,scene.path,"Added"),"discard_changes writes over it") - last,last_ok:=box_import(t,source,root);if last_ok {importer.report_destroy(&last)} - testing.expect(t,last_ok,"then importing again asks nothing") + options.changed=.Refuse + _,last_error:=into(t,root,source,options,nil) + testing.expectf(t,last_error.code==.None,"then importing again asks nothing: %s",last_error.message) } // Whether the scene at path has an entity named name. @@ -249,21 +268,26 @@ a_material_renamed_in_the_source_is_a_new_one_and_a_part_taken_out_is_left_alone for warning in renamed.warnings {left||=strings.contains(warning,"Wood.mat is no longer in the source")} testing.expectf(t,left,"the import says Wood left: %v",renamed.warnings) - // Whether a mesh is a part is decided once and kept in the sidecar: taken out there, it stays out. + // Whether a mesh is a part is decided once and kept in the package: taken out there, it stays out. mesh,_:=written_of(&renamed,asset.KIND_MESH,"Box") testing.expect(t,header_of(mesh.path).owner==first.prefab) - file:=fmt.tprintf("%s.import",source) - listed,_:=os.read_entire_file(file,context.temp_allocator) - taken_out,replaced:=strings.replace(string(listed),", \"part\": true","",1,context.temp_allocator) - testing.expect(t,replaced && os.write_entire_file(file,taken_out)==nil) - after,after_ok:=box_import(t,source,root);if !after_ok {return};defer importer.report_destroy(&after) + s:=test_store(t,fmt.tprintf("%s/store",root)) + p,_:=asset.package_get(s,renamed.manifest,context.temp_allocator) + taken_out:=false + for &a in p.assets { + if a.kind==asset.KIND_MESH && a.part {a.part,taken_out=false,true} + } + manifest,put_error:=asset.package_put(s,&p) + testing.expect(t,taken_out && put_error==.None) + after,after_error:=import_test(t,source,{world=prefab.import_world(t),root=root,textures=.Uncompressed,manifest=manifest}) + if !testing.expectf(t,after_error.code==.None,"%s",after_error.message) {return};defer importer.report_destroy(&after) box,has_box:=written_of(&after,asset.KIND_MESH,"Box") testing.expect(t,has_box && box.id==mesh.id) header:=header_of(box.path) testing.expect(t,header.owner=={} && .Cooked in header.flags,"the part taken out keeps its id and its import, and stays out of its owner") // Forced, every material and texture is a part. - forced,forced_ok:=import_source(source,{world=prefab.import_world(t),root=root,textures=.Uncompressed,parts=.All,ignore_stored=true}) + forced,forced_ok:=import_test(t,source,{world=prefab.import_world(t),root=root,textures=.Uncompressed,parts=.All,ignore_stored=true}) if !testing.expectf(t,forced_ok.code==.None,"%s",forced_ok.message) {return};defer importer.report_destroy(&forced) for written in forced.assets { if written.kind==asset.KIND_MATERIAL || written.kind==asset.KIND_TEXTURE {testing.expectf(t,header_of(written.path).owner==first.prefab,"%s is forced into the prefab",written.name)} @@ -338,7 +362,7 @@ a_material_is_a_part_only_when_its_texture_is_an_atlas_of_one_model :: proc(t:^t if root=="" {return} atlas_images(assets) source:=atlas_source(assets,"props.gltf",{"atlas","waves","across","shared"}) - report,error:=import_source(source,{world=prefab.import_world(t),root=root,textures=.Uncompressed}) + report,error:=import_test(t,source,{world=prefab.import_world(t),root=root,textures=.Uncompressed}) if !testing.expectf(t,error.code==.None,"%s",error.message) {return};defer importer.report_destroy(&report) reason :: proc(material:string) -> string { switch material { @@ -361,33 +385,32 @@ a_material_is_a_part_only_when_its_texture_is_an_atlas_of_one_model :: proc(t:^t // Two models that embed the same image each get a texture and a material of // their own, encoded once in the content cache: neither import changes a -// file or a sidecar of the other, and each keeps its atlas as a part. +// file or the package of the other, and each keeps its atlas as a part. @(test) two_models_with_the_same_image_get_a_texture_each_and_neither_touches_the_other :: proc(t:^testing.T) { root,assets:=test_project(t,"parts-two-models") if root=="" {return} atlas_images(fmt.tprintf("%s/first",assets)) atlas_images(fmt.tprintf("%s/second",assets)) - cache:=fmt.tprintf("%s/cache/textures",root) + cache:=test_store(t,fmt.tprintf("%s/store",root)) + if cache==nil {return} first_source:=atlas_source(fmt.tprintf("%s/first",assets),"crate.gltf",{"atlas","waves","across","shared"}) - first,error:=import_source(first_source,{world=prefab.import_world(t),root=root,texture_cache=cache}) + first,error:=import_test(t,first_source,{world=prefab.import_world(t),root=root,store=cache}) + first_package,_:=asset.package_get(cache,first.manifest,context.temp_allocator) if !testing.expectf(t,error.code==.None,"%s",error.message) {return};defer importer.report_destroy(&first) atlas,_:=written_of(&first,asset.KIND_MATERIAL,"Atlas") texture,_:=written_of(&first,asset.KIND_TEXTURE,"atlas") testing.expect(t,header_of(atlas.path).owner==first.prefab && header_of(texture.path).owner==first.prefab) Kept :: struct {path:string,bytes:[]u8} kept:=make([dynamic]Kept,context.temp_allocator) - sidecar:=fmt.tprintf("%s.import",first_source) - sidecar_bytes,_:=os.read_entire_file(sidecar,context.temp_allocator) - append(&kept,Kept{sidecar,sidecar_bytes}) for written in first.assets { bytes,_:=os.read_entire_file(written.path,context.temp_allocator) append(&kept,Kept{written.path,bytes}) } - cached,_:=os.read_all_directory_by_path(cache,context.temp_allocator) + cached:=cached_refs(cache,textures.CACHE) second_source:=atlas_source(fmt.tprintf("%s/second",assets),"barrel.gltf",{"atlas","waves","across","shared"}) - second,second_error:=import_source(second_source,{world=prefab.import_world(t),root=root,texture_cache=cache}) + second,second_error:=import_test(t,second_source,{world=prefab.import_world(t),root=root,store=cache}) if !testing.expectf(t,second_error.code==.None,"%s",second_error.message) {return};defer importer.report_destroy(&second) second_atlas,wrote_material:=written_of(&second,asset.KIND_MATERIAL,"Atlas") second_texture,wrote_texture:=written_of(&second,asset.KIND_TEXTURE,"atlas") @@ -397,149 +420,12 @@ two_models_with_the_same_image_get_a_texture_each_and_neither_touches_the_other bytes,_:=os.read_entire_file(file.path,context.temp_allocator) testing.expectf(t,slice.equal(bytes,file.bytes),"%s is as the first import left it",file.path) } - again,_:=os.read_all_directory_by_path(cache,context.temp_allocator) + again_package,_:=asset.package_get(cache,first.manifest,context.temp_allocator) + testing.expect(t,len(again_package.assets)==len(first_package.assets),"the package of the first is as it was") + again:=cached_refs(cache,textures.CACHE) testing.expectf(t,len(again)==len(cached),"the images are encoded once: %d entries of the cache, then %d",len(cached),len(again)) } -// A checkout has the sources and their sidecars, and not what they build that -// nobody edits: sync builds that again, the same assets, whatever the order -// its sources come in, and builds nothing when nothing changed. -@(test) -a_checkout_without_its_imported_folder_gets_the_same_assets_from_its_sources :: proc(t:^testing.T) { - root,assets:=test_project(t,"sync") - if root=="" {return} - atlas_images(fmt.tprintf("%s/set/crate",assets)) - atlas_images(fmt.tprintf("%s/set/barrel",assets)) - crate:=atlas_source(fmt.tprintf("%s/set/crate",assets),"crate.gltf",{"atlas","waves","across","shared"}) - barrel:=atlas_source(fmt.tprintf("%s/set/barrel",assets),"barrel.gltf",{"atlas","waves","across","shared"}) - _=os.write_entire_file(fmt.tprintf("%s/sign.png",assets),png_bytes(8,8,{200,40,40})) - sources:=[3]string{crate,barrel,fmt.tprintf("%s/sign.png",assets)} - Built :: struct {path:string,id:base.Asset_Id,bytes:[]u8} - built:=make([dynamic]Built,context.temp_allocator) - caches:=Sync_Options{texture_cache=fmt.tprintf("%s/cache/textures",root),mesh_cache=fmt.tprintf("%s/cache/meshes",root),library=test_library} - context.user_ptr=prefab.import_world(t) - for source in sources { - report,error:=import_source(source,{world=prefab.import_world(t),root=root,textures=.Uncompressed,texture_cache=caches.texture_cache,mesh_cache=caches.mesh_cache}) - if !testing.expectf(t,error.code==.None,"%s: %s",source,error.message) {return} - for written in report.assets {append(&built,Built{path=strings.clone(written.path,context.temp_allocator),id=written.id})} - importer.report_destroy(&report) - } - for &file in built {file.bytes,_=os.read_entire_file(file.path,context.temp_allocator)} - sidecars:[3]string - for source,at in sources { - text,_:=os.read_entire_file(fmt.tprintf("%s.import",source),context.temp_allocator) - sidecars[at]=string(text) - } - imported:=fmt.tprintf("%s/.mirai/imported",root) - cached:=0 - for file in built {cached+=int(strings.has_prefix(file.path,imported))} - testing.expect(t,cached>0 && cached (listed:[dynamic]Listed,missing:int) { - r:asset.Registry - asset.registry_init(&r,fmt.tprintf("%s/assets",root),imported=fmt.tprintf("%s/.mirai/imported",root)) - defer asset.registry_destroy(&r) - testing.expect_value(t,asset.registry_scan(&r),asset.Error.None) - listed=make([dynamic]Listed,context.temp_allocator) - for file in r.files { - if file.imported!=0 {append(&listed,Listed{file.id,strings.clone(file.path,context.temp_allocator)})} - } - lost:=make([dynamic]asset.Registry_Missing,context.temp_allocator) - asset.registry_missing(&r,&lost) - return listed,len(lost) - } - path_of :: proc(listed:[]Listed,path:string) -> (id:base.Asset_Id,found:bool) { - for entry in listed { - if entry.path==path {return entry.id,true} - } - return - } - before,missing:=listing(t,root) - body,has_body:=path_of(before[:],"set/crate/crate/Body.mesh") - keg,has_keg:=path_of(before[:],"set/barrel/barrel/Body.mesh") - atlas,has_atlas:=path_of(before[:],"set/crate/crate/atlas.tex") - _,has_sign:=path_of(before[:],"sign.tex") - testing.expectf(t,has_body && has_keg && has_atlas && has_sign && missing==0 && len(before)==cached,"what an import gives with its documents, the texture of an image beside it: %v",before[:]) - - // A source that changed is imported again, and it alone; a file it reads counts as the source. - _=os.write_entire_file(fmt.tprintf("%s/sign.png",assets),png_bytes(8,8,{40,40,200})) - changed:=sync(root,caches,context.temp_allocator) - testing.expectf(t,changed.imported==1 && len(changed.failures)==0,"%v",changed) - _=os.write_entire_file(fmt.tprintf("%s/set/crate/across.png",assets),png_pixels(64,64,gradient_green)) - read:=sync(root,caches,context.temp_allocator) - testing.expectf(t,read.imported==1 && len(read.failures)==0,"an image a model reads changed: %v",read) - - // A folder with two sources moves, one of them is renamed and another source is deleted: at - // once, without an import, every asset is listed where its source is, with the id it had, - // what names them still finds them and nothing of the deleted source is listed. - testing.expect(t,os.rename(fmt.tprintf("%s/set",assets),fmt.tprintf("%s/props",assets))==nil) - for end in ([2]string{"",".import"}) {testing.expect(t,os.rename(fmt.tprintf("%s/props/barrel/barrel.gltf%s",assets,end),fmt.tprintf("%s/props/barrel/keg.gltf%s",assets,end))==nil)} - testing.expect(t,os.remove(fmt.tprintf("%s/sign.png",assets))==nil) - after,after_missing:=listing(t,root) - moved_body,_:=path_of(after[:],"props/crate/crate/Body.mesh") - moved_keg,_:=path_of(after[:],"props/barrel/barrel/Body.mesh") - moved_atlas,_:=path_of(after[:],"props/crate/crate/atlas.tex") - testing.expectf(t,moved_body==body && moved_keg==keg && moved_atlas==atlas && body!={} && keg!={} && after_missing==0 && len(after)==len(before)-1,"%v",after[:]) - for entry in after {testing.expectf(t,strings.has_prefix(entry.path,"props/"),"%s is listed where its source is",entry.path)} - // Nothing changed in what they are built from: the next build imports nothing, and removes what the deleted source gave. - again_synced:=sync(root,caches,context.temp_allocator) - testing.expectf(t,again_synced.sources==2 && again_synced.imported==0 && again_synced.removed==1 && len(again_synced.failures)==0,"%v",again_synced) -} - // A format of models a plugin adds, which no format of the engine reads: glTF after a magic of its own. @(private="file") PROBE_MAGIC :: "PROBE:" @(private="file") @@ -553,52 +439,57 @@ probe_format :: proc() -> importer.Format { return probe } -// Every import goes to the cache, whatever reads it: a format a plugin adds -// too, which sync reads again from the plugins of the project, and a source -// outside the assets is refused, saying where it goes. +// A format a plugin adds reads a source as the formats of the engine do: a +// refresh that must ingest it again reads it with the plugins of the project, +// and says so without them. @(test) -the_sources_of_a_plugin_format_build_in_the_cache_and_a_source_outside_the_assets_is_refused :: proc(t:^testing.T) { - root:=base.test_folder(t,"sync-plugin") +a_refresh_reads_a_source_again_with_the_formats_of_the_plugins :: proc(t:^testing.T) { + root:=base.test_folder(t,"refresh-plugin") if root=="" {return} - outside:=fmt.tprintf("%s/downloads",root) - project:=fmt.tprintf("%s/game",root) - os.make_directory_all(outside) - os.make_directory_all(fmt.tprintf("%s/assets",project)) - box:=box_source(outside,"box.gltf","Wood","Paint") - _,refused:=import_source(box,{world=prefab.import_world(t),root=project,textures=.Uncompressed}) - testing.expect(t,refused.code==.Not_Found && strings.contains(refused.message,"assets") && !os.exists(fmt.tprintf("%s.import",box)),refused.message) - - formats:=make([dynamic]importer.Format,context.temp_allocator) - append(&formats,..FORMATS[:]) - append(&formats,probe_format()) - crate:=box_source(fmt.tprintf("%s/assets",project),"crate.probe","Wood","Paint") + game:=fmt.tprintf("%s/game",root) + os.make_directory_all(fmt.tprintf("%s/assets",game)) + crate:=box_source(fmt.tprintf("%s/assets",game),"crate.probe","Wood","Paint") text,_:=os.read_entire_file(crate,context.temp_allocator) _=os.write_entire_file(crate,transmute([]u8)strings.concatenate({PROBE_MAGIC,string(text)},context.temp_allocator)) - first,error:=import_source(crate,{world=prefab.import_world(t),root=project,textures=.Uncompressed},formats[:]) - if !testing.expectf(t,error.code==.None,"%s",error.message) {return};defer importer.report_destroy(&first) - mesh,has_mesh:=written_of(&first,asset.KIND_MESH,"Box") - testing.expect(t,has_mesh && strings.contains(mesh.path,"/.mirai/imported/"),"what a plugin format gives that nobody edits is of the cache") - // A clone: sync builds it again with the formats of the plugins, and says so without them. - caches:=Sync_Options{texture_cache=fmt.tprintf("%s/cache/textures",root),mesh_cache=fmt.tprintf("%s/cache/meshes",root)} - testing.expect(t,base.remove_all(fmt.tprintf("%s/.mirai",project))==nil) - lacking:=sync(project,caches,context.temp_allocator) - testing.expectf(t,len(lacking.failures)==1 && strings.contains(lacking.failures[0],"no format"),"%v",lacking) + s:=test_store(t,fmt.tprintf("%s/store",root)) + if s==nil {return} context.user_ptr=prefab.import_world(t) - caches.library=proc(root:string,allocator:mem.Allocator) -> Library_Imports { + probing :: proc(root:string,s:^store.Store,allocator:mem.Allocator) -> Library_Imports { listed:=make([]importer.Format,1,allocator) listed[0]=probe_format() return {formats=listed,world=(^core.World_Info)(context.user_ptr)} } - synced:=sync(project,caches,context.temp_allocator) - testing.expectf(t,synced.sources==1 && synced.imported==1 && len(synced.failures)==0,"%v",synced) - testing.expect(t,os.exists(mesh.path),"the mesh is built again where it was") + options:=Library_Options{store=s,root=game,plugins=probing} + first,error:=ingest(crate,options,allocator=context.temp_allocator) + if !testing.expectf(t,error.code==.None,"%s",error.message) {return} + added:=project_add(game,first.manifest,"",nil,options,context.temp_allocator) + testing.expectf(t,len(added.failures)==0 && len(added.missing)==0,"%v",added) + mesh,has_mesh:=written_of(&first,asset.KIND_MESH,"Box") + testing.expect(t,has_mesh && mesh.path=="" && asset.object_whole(s,mesh.object),"what a plugin format gives that nobody edits is an object of the store") + + // An engine of other versions: without the plugins nothing reads it, with them it is ingested again. + forget :: proc(s:^store.Store) { + for namespace in ([2]string{asset.REF_IMPORTS,asset.REF_BUILDS}) { + refs,_:=store.ref_list(s,namespace,context.temp_allocator) + for ref in refs {store.ref_remove(s,namespace,ref.name)} + } + } + forget(s) + lacking:=options + lacking.plugins=nil + without:=project_refresh(game,lacking,context.temp_allocator) + testing.expectf(t,len(without.failures)==1 && strings.contains(without.failures[0],"not a source Mirai imports"),"%v",without) + with:=project_refresh(game,options,context.temp_allocator) + testing.expectf(t,with.ingested==1 && len(with.failures)==0 && len(with.missing)==0,"%v",with) + testing.expect(t,asset.object_whole(s,mesh.object),"the mesh is the object it was") } // A source a plugin imports whole: "LEVEL ", a level of the glTF at // that file beside it, which it reads and writes as a model of its import, // and with kit a document of the kind KIT of that version. code: the -// identity of the code of its plugin. -@(private="file") Level_Importer :: struct {runs:int,code:u64,kit:u16} +// identity of the code of its plugin; version: that of its format. +@(private="file") Level_Importer :: struct {runs:int,code:u64,kit:u16,version:u16} +@(private="file") LEVEL_IMPORTER :: "Levels" @(private="file") KIT :: asset.Fourcc(u32('K')|u32('I')<<8|u32('T')<<16|u32('S')<<24) @(private="file") level_claims :: proc(path:string,head:[]u8) -> bool {return strings.has_prefix(string(head),"LEVEL ")} @@ -608,7 +499,7 @@ level_run :: proc(state:rawptr,source:string,options:importer.Import_Options,all bytes,_:=os.read_entire_file(source,context.temp_allocator) model_path:=fmt.tprintf("%s/%s",filepath.dir(source),string(bytes[len("LEVEL "):])) o:importer.Outputs - if error=importer.outputs_begin(&o,source,"",u64(len(bytes)),base.hash64(bytes),importer.import_version(0),options,allocator);error.code!=.None {return} + if error=importer.outputs_begin(&o,source,"",u64(len(bytes)),base.hash64(bytes),importer.import_version((^Level_Importer)(state).version),options,LEVEL_IMPORTER,allocator);error.code!=.None {return} model:importer.Imported_Model if error=importer.output_input(&o,model_path);error.code==.None {model,error=importer.import_model(model_path,FORMATS[:],options)} defer importer.model_destroy(&model) @@ -619,87 +510,135 @@ level_run :: proc(state:rawptr,source:string,options:importer.Import_Options,all asset.document_init(&kit,KIT,id,context.temp_allocator) kit.kind_version=version if asset.document_save(&kit,path)!=.None {error=importer.error_make(.Write,"%s could not be written",path)} - importer.output_written(&o,id,KIT,"hall",path,false) + if error.code==.None {error=importer.output_written(&o,id,"hall")} } - return importer.outputs_end(&o,error.code!=.None),"",error + if error.code!=.None { + _,_=importer.outputs_end(&o,true) + return {},"",error + } + report,error=importer.outputs_end(&o,false,allocator) + return report,"",error } -// Sync runs the importer of a plugin that claims a whole source when what it -// is built from changes: its bytes or those of a file it reads, the code of -// the plugin, or the version of a kind of document it wrote, of the engine or -// of a plugin; it leaves a current one alone, and a document it wrote that -// was changed since as it is. +// A refresh ingests again a whole source a plugin claims when what it is +// built from changes: its bytes or those of a file it reads, or the code of +// the plugin; it leaves a current one alone, the ids stay, and a document it +// wrote into the project that was changed since stays as it is. @(test) -sync_imports_again_a_whole_source_when_what_it_is_built_from_changes :: proc(t:^testing.T) { - root,assets:=test_project(t,"sync-whole") +a_refresh_ingests_again_a_whole_source_when_what_it_is_built_from_changes :: proc(t:^testing.T) { + root,assets:=test_project(t,"refresh-whole") if root=="" {return} - write_text(root,project.PROJECT_FILE,`{"name": "Hall", "version": "0.1.0"}`) - os.make_directory_all(fmt.tprintf("%s/plugins/kit",root)) - kit_manifest :: proc(root:string,version:int) {write_text(fmt.tprintf("%s/plugins/kit",root),project.PLUGIN_FILE,strings.concatenate({`{"name": "kit", "version": "1.0.0", "asset_kinds": [{"extension": ".kit", "kind": "KITS", "version": `,fmt.tprint(version),`}]}`},context.temp_allocator))} - kit_manifest(root,2) props:=fmt.tprintf("%s/props",assets) os.make_directory_all(props) box_source(props,"crate.gltf","Wood","Paint") level:=write_text(assets,"hall.level","LEVEL props/crate.gltf") state:=Level_Importer{kit=2} - wholes:=[1]Whole_Source{{claims=level_claims,run=level_run,state=&state}} - caches:=Sync_Options{texture_cache=fmt.tprintf("%s/cache/textures",root),mesh_cache=fmt.tprintf("%s/cache/meshes",root)} - first,error:=import_source(level,{root=root,textures=.Uncompressed,texture_cache=caches.texture_cache,mesh_cache=caches.mesh_cache},wholes=wholes[:]) - if !testing.expectf(t,error.code==.None,"%s",error.message) {return} - defer importer.report_destroy(&first) - mesh,has_mesh:=written_of(&first,asset.KIND_MESH,"Box") - testing.expect(t,has_mesh && strings.contains(mesh.path,"/.mirai/imported/"),"the mesh of the level is of the cache") - sidecar,_,_:=importer.sidecar_load(level,context.temp_allocator) - testing.expectf(t,slice.contains(sidecar.inputs[:],"props/crate.gltf") && slice.contains(sidecar.inputs[:],"props/wood.png"),"the files it reads are its inputs: %v",sidecar.inputs) - - lacking:=sync(root,caches,context.temp_allocator) - testing.expectf(t,len(lacking.failures)==1 && strings.contains(lacking.failures[0],"no format or importer"),"%v",lacking) + s:=test_store(t,fmt.tprintf("%s/store",root)) + if s==nil {return} context.user_ptr=&state - caches.library=proc(root:string,allocator:mem.Allocator) -> Library_Imports { + leveling :: proc(root:string,s:^store.Store,allocator:mem.Allocator) -> Library_Imports { wholes:=make([]Whole_Source,1,allocator) - wholes[0]={claims=level_claims,run=level_run,state=context.user_ptr,code=(^Level_Importer)(context.user_ptr).code} + level:=(^Level_Importer)(context.user_ptr) + wholes[0]={name=LEVEL_IMPORTER,version=level.version,claims=level_claims,run=level_run,state=level,code=level.code} return {wholes=wholes} } - current:=sync(root,caches,context.temp_allocator) - testing.expectf(t,current.sources==1 && current.imported==0 && len(current.failures)==0 && state.runs==1,"a current source is left alone: %v",current) - Change :: enum {Input,Code,Engine_Kind,Plugin_Kind,Clone,Edited} - material,has_material:=written_of(&first,asset.KIND_MATERIAL,"Wood") - kit,has_kit:=written_of(&first,KIT,"hall") - if !testing.expect(t,has_material && has_kit,"the level wrote documents") {return} + options:=Library_Options{store=s,root=root,plugins=leveling,fast_textures=true} + first,error:=ingest(level,options,allocator=context.temp_allocator) + if !testing.expectf(t,error.code==.None,"%s",error.message) {return} + mesh,has_mesh:=written_of(&first,asset.KIND_MESH,"Box") + testing.expect(t,has_mesh && mesh.path=="" && asset.object_whole(s,mesh.object),"the mesh of the level is an object of the store") + p,_:=asset.package_get(s,first.manifest,context.temp_allocator) + testing.expectf(t,slice.contains(p.inputs[:],"props/crate.gltf") && slice.contains(p.inputs[:],"props/wood.png"),"the files it reads are its inputs: %v",p.inputs) + added:=project_add(root,first.manifest,"",nil,options,context.temp_allocator) + testing.expectf(t,len(added.failures)==0 && len(added.missing)==0,"%v",added) + _,has_kit:=written_of(&first,KIT,"hall") + testing.expect(t,has_kit,"the level wrote a document") + + runs:=state.runs + current:=project_refresh(root,options,context.temp_allocator) + testing.expectf(t,current.ingested==0 && len(current.failures)==0 && len(current.missing)==0 && state.runs==runs,"a current source is left alone: %v",current) + Change :: enum {Input,Code,Edited} + kit_id,_:=written_of(&first,KIT,"hall") + r:asset.Registry + asset.registry_init(&r,assets,"",context.temp_allocator) + testing.expect_value(t,asset.registry_scan(&r),asset.Error.None) + kit_file,kit_found:=asset.registry_find(&r,kit_id.id) + if !testing.expect(t,kit_found,"the project has its file of the document") {return} + kit:=asset.registry_file_path(&r,kit_file.path,context.temp_allocator) for change in Change { switch change { case .Input: _=os.write_entire_file(fmt.tprintf("%s/wood.png",props),png_bytes(4,4,{20,90,200})) case .Code: state.code=7 - case .Engine_Kind: - data,_:=os.read_entire_file(material.path,context.temp_allocator) - old:=asset.MATERIAL_VERSION-1 - data[10],data[11]=u8(old),u8(old>>8) - testing.expect(t,os.write_entire_file(material.path,data)==nil) - case .Plugin_Kind: - kit_manifest(root,3) - state.kit=3 - case .Clone: testing.expect(t,base.remove_all(fmt.tprintf("%s/.mirai",root))==nil) case .Edited: doc:asset.Document - testing.expect_value(t,asset.document_load(&doc,kit.path,context.temp_allocator),asset.Error.None) + testing.expect_value(t,asset.document_load(&doc,kit,context.temp_allocator),asset.Error.None) _,_=asset.document_table_add(&doc,"notes") - testing.expect_value(t,asset.document_save(&doc,kit.path),asset.Error.None) + testing.expect_value(t,asset.document_save(&doc,kit),asset.Error.None) state.code=9 } - edited,_:=os.read_entire_file(kit.path,context.temp_allocator) - runs:=state.runs - changed:=sync(root,caches,context.temp_allocator) - testing.expectf(t,changed.imported==1 && len(changed.failures)==0 && state.runs==runs+1,"%v: %v",change,changed) + edited,_:=os.read_entire_file(kit,context.temp_allocator) + runs=state.runs + changed:=project_refresh(root,options,context.temp_allocator) + testing.expectf(t,changed.ingested==1 && len(changed.failures)==0 && state.runs==runs+1,"%v: %v",change,changed) if change==.Edited { - now,_:=os.read_entire_file(kit.path,context.temp_allocator) - testing.expectf(t,string(now)==string(edited) && len(changed.warnings)==1 && contains(changed.warnings[0],"hall.kit"),"what was changed of it stays, and sync says so: %v",changed.warnings) + now,_:=os.read_entire_file(kit,context.temp_allocator) + testing.expectf(t,string(now)==string(edited) && len(changed.warnings)==1 && contains(changed.warnings[0],"assets/hall"),"what was changed of it stays, and the refresh says so: %v",changed.warnings) + } + testing.expectf(t,project_refresh(root,options,context.temp_allocator).ingested==0,"%v: then nothing",change) + } + list,_,_:=asset.packages_load(root,context.temp_allocator) + again,_:=asset.package_get(s,list.packages[0].manifest,context.temp_allocator) + testing.expect(t,list.packages[0].manifest!=first.manifest && len(again.assets)==len(p.assets) && again.assets[0].id==p.assets[0].id,"the level keeps its assets, pinned anew") +} + +// A store that holds the build of a source a plugin imports whole, and not +// the source, takes it by key from another store, lists and loads it, and +// imports nothing; another version or code of the importer misses that build. +@(test) +a_whole_source_build_is_found_by_key_without_its_source :: proc(t:^testing.T) { + root,assets:=test_project(t,"whole-by-key") + if root=="" {return} + props:=fmt.tprintf("%s/props",assets) + os.make_directory_all(props) + box_source(props,"crate.gltf","Wood","Paint") + level:=write_text(assets,"hall.level","LEVEL props/crate.gltf") + state:=Level_Importer{version=1} + context.user_ptr=&state + leveling :: proc(root:string,s:^store.Store,allocator:mem.Allocator) -> Library_Imports { + wholes:=make([]Whole_Source,1,allocator) + level:=(^Level_Importer)(context.user_ptr) + wholes[0]={name=LEVEL_IMPORTER,version=level.version,claims=level_claims,run=level_run,state=level,code=level.code} + return {wholes=wholes} + } + s:=test_store(t,fmt.tprintf("%s-host",root)) + if s==nil {return} + first,error:=ingest(level,{store=s,root=root,plugins=leveling},allocator=context.temp_allocator) + if !testing.expectf(t,error.code==.None,"%s",error.message) {return} + added:=project_add(root,first.manifest,"",nil,{store=s,root=root,plugins=leveling},context.temp_allocator) + if !testing.expectf(t,len(added.failures)==0 && len(added.missing)==0,"%v",added) {return} + peer:=fmt.tprintf("%s-peer",root) + folder_copy(root,peer) + testing.expect(t,os.remove(fmt.tprintf("%s/assets/hall.level",peer))==nil) + runs:=state.runs + Case :: struct {version:u16,code:u64,found:bool} + for c,k in ([]Case{{1,0,true},{2,0,false},{1,5,false}}) { + state.version,state.code=c.version,c.code + here:=test_store(t,fmt.tprintf("%s-store-%d",root,k)) + if here==nil {return} + refreshed:=project_refresh(peer,{store=here,root=peer,plugins=leveling,source=cache.store_source(s)},context.temp_allocator) + testing.expect(t,state.runs==runs && refreshed.ingested==0,"nothing is imported") + if !c.found { + testing.expectf(t,refreshed.fetched==0 && len(refreshed.missing)==1,"%v: another importer misses the build: %v",c,refreshed) + continue + } + testing.expectf(t,refreshed.fetched==1 && len(refreshed.missing)==0 && len(refreshed.failures)==0,"%v",refreshed) + listed,list_error:=asset.root_packages_in(here,peer,context.temp_allocator) + testing.expectf(t,list_error==.None && len(listed.problems)==0 && len(listed.listed)>0,"%v",listed) + for l in listed.listed { + a:asset.Asset + if testing.expectf(t,asset.asset_open_object(&a,here,l.object,context.temp_allocator)==.None,"%s loads",l.path) {asset.asset_close(&a)} } - testing.expectf(t,sync(root,caches,context.temp_allocator).imported==0,"%v: then nothing",change) } - testing.expect(t,header_of(material.path).kind_version==asset.MATERIAL_VERSION && header_of(material.path).id==material.id,"the material is of this version again, with its id") - testing.expect(t,header_of(kit.path).kind_version==3 && header_of(kit.path).id==kit.id,"the document of the plugin is of its version again, with its id") - again,_,_:=importer.sidecar_load(level,context.temp_allocator) - testing.expect(t,len(again.outputs)==len(sidecar.outputs) && again.outputs[0].id==sidecar.outputs[0].id,"the level keeps its assets") } // The code of a plugin is its Odin sources and those of the plugins it holds, @@ -724,34 +663,35 @@ the_code_of_a_plugin_changes_with_its_sources_alone :: proc(t:^testing.T) { // given once stays in its settings. @(test) an_image_on_its_own_holds_what_its_name_says_until_its_settings_say_otherwise :: proc(t:^testing.T) { - _,assets:=test_project(t,"image-use") + root,assets:=test_project(t,"image-use") if assets=="" {return} + kept:=fmt.tprintf("%s/store",root) run :: proc(args:..string) -> (code:int,err:string) { output,errors:=strings.builder_make(context.temp_allocator),strings.builder_make(context.temp_allocator) code=command(args,strings.to_writer(&output),strings.to_writer(&errors)) return code,strings.to_string(errors) } - stored :: proc(source:string) -> importer.Image_Use { - settings,_,_:=importer.settings_load(source) + stored :: proc(t:^testing.T,kept,source:string) -> importer.Image_Use { + settings,_,_:=importer.settings_load({store=test_store(t,kept)},source) return settings.image } masks:=fmt.tprintf("%s/T_Katana_SpecularMasks.png",assets) diffuse:=fmt.tprintf("%s/T_Katana_Diffuse.png",assets) for path in ([?]string{masks,diffuse}) {_=os.write_entire_file(path,png_bytes(4,4,{10,200,30}))} - code,err:=run(masks,"--uncompressed") + code,err:=run(masks,"--uncompressed","--store",kept) testing.expectf(t,code==0,"%s",err) - testing.expect_value(t,stored(masks),importer.Image_Use.Data) - code,err=run(diffuse,"--uncompressed") + testing.expect_value(t,stored(t,kept,masks),importer.Image_Use.Data) + code,err=run(diffuse,"--uncompressed","--store",kept) testing.expectf(t,code==0,"%s",err) - testing.expect_value(t,stored(diffuse),importer.Image_Use.Color) + testing.expect_value(t,stored(t,kept,diffuse),importer.Image_Use.Color) // Given once, the use stays: the name no longer decides. - code,err=run(masks,"--uncompressed","--srgb") + code,err=run(masks,"--uncompressed","--srgb","--store",kept) testing.expectf(t,code==0,"%s",err) - code,err=run(masks,"--uncompressed") + code,err=run(masks,"--uncompressed","--store",kept) testing.expectf(t,code==0,"%s",err) - testing.expect_value(t,stored(masks),importer.Image_Use.Color) + testing.expect_value(t,stored(t,kept,masks),importer.Image_Use.Color) Named :: struct {name:string,use:importer.Image_Use} for named in ([?]Named{{"data",.Data},{"Color",.Color},{"normal",.Normal},{"NORMAL_GREEN_DOWN",.Normal_Green_Down}}) { @@ -768,7 +708,7 @@ importing_over_a_skeleton_of_an_older_version_says_what_it_did_not_keep :: proc( root,folder:=test_project(t,"old-skeleton") if folder=="" {return} source:=copy_in(root,FIXTURES,"skeletal-analytic.glb") - first,error:=import_source(source,{world=prefab.import_world(t),textures=.Uncompressed}) + first,error:=import_test(t,source,{world=prefab.import_world(t),textures=.Uncompressed}) if !testing.expectf(t,error.code==.None,"%s",error.message) {return} defer importer.report_destroy(&first) skeleton:importer.Written @@ -778,7 +718,7 @@ importing_over_a_skeleton_of_an_older_version_says_what_it_did_not_keep :: proc( old:=asset.SKELETON_VERSION-1 data[10],data[11]=u8(old),u8(old>>8) testing.expect(t,os.write_entire_file(skeleton.path,data)==nil) - again,failed:=import_source(source,{world=prefab.import_world(t),textures=.Uncompressed}) + again,failed:=import_test(t,source,{world=prefab.import_world(t),textures=.Uncompressed}) if !testing.expectf(t,failed.code==.None,"%s",failed.message) {return} defer importer.report_destroy(&again) told:=false diff --git a/cook/import/sources/settings_test.odin b/cook/import/sources/settings_test.odin index cd1b76606..a0ac8da03 100644 --- a/cook/import/sources/settings_test.odin +++ b/cook/import/sources/settings_test.odin @@ -8,6 +8,8 @@ import "mirai:asset" import "mirai:std/base" import "mirai:std/geom" import importer "mirai:cook/import" +import "mirai:cook/prefab" +import "mirai:store" // An OBJ of a sphere, smooth, with texture coordinates. @(private="file") @@ -36,7 +38,7 @@ a_source_without_colliders_gives_a_mesh_of_levels_without_any :: proc(t:^testing if !ok {return} defer importer.model_destroy(&model) testing.expect(t,model.settings.front==.X && !model.stored) - report,error:=write_into(&model,fmt.tprintf("%s/out",folder),{front=.X}) + report,error:=write_into(t,&model,fmt.tprintf("%s/out",folder),{front=.X}) defer importer.report_destroy(&report) testing.expect_value(t,error.code,importer.Error_Code.None) if error.code!=.None || len(report.assets)==0 {return} @@ -58,69 +60,59 @@ a_source_without_colliders_gives_a_mesh_of_levels_without_any :: proc(t:^testing testing.expect(t,int(view.info.vertex_count)==len(view.vertices) && view.levels[0].triangle_count string {return fmt.tprintf("{{\"version\": 4, \"settings\": %s, \"outputs\": [%s]}}",settings,outputs)} - for broken in ([]string{ - whole("{\"lods\": \"Sometimes\"}",""),whole("{\"scale\": 2}",""),"{",whole("{\"rdo_lambda\": 1000}",""),whole("{\"alpha_coverage\": 1.5}",""),whole("{\"alpha_coverage\": \"nan\"}",""),whole("{\"rig\": \"Fox\", \"rig_crotch\": 0.9}",""), - "{}","{\"lods\": \"None\"}","{\"version\": 3, \"settings\": {}, \"outputs\": []}", - "{\"version\": 4, \"settings\": {}, \"folder\": \"../up\", \"outputs\": []}","{\"version\": 4, \"settings\": {}, \"folder\": \"/root\", \"outputs\": []}", - whole("{}","{\"kind\": \"MESH\", \"name\": \"Ball\", \"index\": 0}"), - whole("{}",fmt.tprintf("{{\"kind\": \"MESH\", \"name\": \"Ball\", \"index\": 0, %s}}, {{\"kind\": \"MAT\", \"name\": \"Skin\", \"index\": 0, %s}}",ID,ID)), - whole("{}",fmt.tprintf("{{\"kind\": \"MAT\", \"name\": \"Skin\", \"index\": 0, %s, \"part\": true}}",ID)), - whole("{}",fmt.tprintf("{{\"kind\": \"MESH\", \"name\": \"Ball\", \"index\": 0, %s, \"written\": \"00000000000000ff\"}}",ID)), - whole("{}",fmt.tprintf("{{\"kind\": \"MESH\", \"name\": \"Ball\", \"index\": 0, %s, \"path\": \"Ball.mesh\"}}",ID)), - }) { - write_text(folder,"ball.obj.import",broken) - _,_,broken_error:=importer.settings_load(source) - testing.expectf(t,broken_error.code==.Malformed,"%s is refused",broken) - _,import_error:=importer.import_model(source,FORMATS[:]) - testing.expect_value(t,import_error.code,importer.Error_Code.Malformed) + // Settings nobody wrote are refused, not the defaults: a value an import does not know, + // out of its range, a field it does not have, heights without the rig they go with. + for broken in ([]string{"{\"lods\": \"Sometimes\"}","{\"scale\": 2}","{","{\"rdo_lambda\": 1000}","{\"alpha_coverage\": 1.5}","{\"alpha_coverage\": \"nan\"}","{\"rig\": \"Fox\", \"rig_crotch\": 0.9}"}) { + _,valid:=importer.settings_parse(broken) + testing.expectf(t,!valid,"%s is refused",broken) } - write_text(folder,"ball.obj.import",whole("{}","")) - stored,found,load_error=importer.settings_load(source) - testing.expect(t,found && load_error.code==.None && stored=={},"a setting that is not named has its default") + // A package that keeps them refuses its import. + kept:=p + kept.settings="{\"lods\": \"Sometimes\"}" + manifest,put_error:=asset.package_put(options.store,&kept) + testing.expect_value(t,put_error,asset.Error.None) + pinned:=options + pinned.manifest=manifest + _,import_error:=importer.import_model(source,FORMATS[:],pinned) + testing.expect_value(t,import_error.code,importer.Error_Code.Malformed) } @(test) @@ -139,26 +131,29 @@ a_standalone_image_takes_its_use_from_its_settings :: proc(t:^testing.T) { if folder=="" {return} path:=fmt.tprintf("%s/normal.png",folder) _=os.write_entire_file(path,png_bytes(8,8,{128,128,255})) - testing.expect(t,importer.sidecar_save(path,{settings={image=.Normal,textures=.Uncompressed}})) - report,error:=importer.import_texture(path,{root=root,image=.Color}) + first,first_error:=importer.import_texture(path,test_options(t,path,{root=root,image=.Normal,textures=.Uncompressed})) + if !testing.expectf(t,first_error.code==.None,"%s",first_error.message) {return} + importer.report_destroy(&first) + options:=test_options(t,path,{root=root,image=.Color}) + report,error:=importer.import_texture(path,options) defer importer.report_destroy(&report) testing.expect_value(t,error.code,importer.Error_Code.None) if error.code!=.None {return} testing.expect(t,report.assets[0].id!={}) testing.expect(t,strings.contains(report.assets[0].detail,"8x8"),report.assets[0].detail) testing.expect_value(t,report.settings,importer.Import_Settings{image=.Normal,textures=.Uncompressed}) - text,_:=os.read_entire_file(fmt.tprintf("%s%s",path,importer.SIDECAR_EXTENSION),context.temp_allocator) - testing.expect(t,strings.contains(string(text),`"image": "Normal"`) && strings.contains(string(text),`"textures": "Uncompressed"`) && !strings.contains(string(text),"lods"),string(text)) + p,_:=asset.package_get(options.store,report.manifest,context.temp_allocator) + testing.expect(t,strings.contains(p.settings,`"image": "Normal"`) && strings.contains(p.settings,`"textures": "Uncompressed"`) && !strings.contains(p.settings,"lods"),p.settings) } @(test) -the_coverage_of_a_standalone_image_is_a_setting_its_sidecar_keeps :: proc(t:^testing.T) { +the_coverage_of_a_standalone_image_is_a_setting_its_package_keeps :: proc(t:^testing.T) { _,folder:=test_project(t,"image-coverage") if folder=="" {return} path:=fmt.tprintf("%s/leaves.png",folder) _=os.write_entire_file(path,png_bytes(16,16,{40,120,30},masked=true)) import_image :: proc(t:^testing.T,path:string,options:importer.Import_Options,loc:=#caller_location) -> (written:importer.Written,coverage:f32,ok:bool) { - report,error:=import_source(path,options) + report,error:=import_test(t,path,options) defer importer.report_destroy(&report) if !testing.expectf(t,error.code==.None && len(report.assets)==1,"%v %s",error.code,error.message,loc=loc) {return} written=report.assets[0] @@ -173,7 +168,7 @@ the_coverage_of_a_standalone_image_is_a_setting_its_sidecar_keeps :: proc(t:^tes first,coverage,ok:=import_image(t,path,{image=.Color,alpha_coverage=0.5}) if !ok {return} testing.expect_value(t,coverage,f32(0.5)) - stored,found,_:=importer.settings_load(path) + stored,found,_:=importer.settings_load(test_options(t,path),path) testing.expect(t,found && stored==importer.Import_Settings{image=.Color,alpha_coverage=0.5},"the coverage is saved with the use of the image") // Whoever imports it again, without knowing the material that shows it, gets the same texture. again,kept_coverage,kept_ok:=import_image(t,path,{}) @@ -189,10 +184,11 @@ a_vertex_that_is_not_a_number_fails_its_model_with_a_message :: proc(t:^testing. defer importer.model_destroy(&model) model.meshes[0].vertices[5].position.y=transmute(f32)u32(0x7fc00000) out:=fmt.tprintf("%s/out",folder) - report,error:=write_into(&model,out) + report,error:=write_into(t,&model,out) defer importer.report_destroy(&report) testing.expect_value(t,error.code,importer.Error_Code.Malformed) testing.expectf(t,strings.contains(error.message,"not a number") && strings.contains(error.message,model.meshes[0].name),"%q",error.message) - entries,_:=os.read_all_directory_by_path(fmt.tprintf("%s/.mirai/imported",out),context.temp_allocator) - for entry in entries {testing.expectf(t,!strings.has_suffix(entry.name,".mesh"),"%s stays after the failure",entry.name)} + s:=test_store(t,fmt.tprintf("%s/store",out)) + refs,_:=store.ref_list(s,asset.REF_PACKAGES,context.temp_allocator) + testing.expect(t,len(refs)==0,"a failed import names no package") } diff --git a/cook/import/sources/sources.odin b/cook/import/sources/sources.odin index 04a703529..5cb9ef96b 100644 --- a/cook/import/sources/sources.odin +++ b/cook/import/sources/sources.odin @@ -104,26 +104,14 @@ source_kind :: proc(path:string,formats:[]importer.Format=nil) -> Source_Kind { return .None } -// The project or the plugin whose assets folder has the file at path: the -// folder that holds the nearest assets folder above it. False for a file in -// no assets folder. -source_root :: proc(path:string,allocator:=context.allocator) -> (root:string,ok:bool) { - names:=project.FOLDER_NAMES - folder:=project.path_absolute(path,context.temp_allocator) - for { - parent:=filepath.dir(folder) - if parent==folder {return} - if filepath.base(parent)==names[.Assets] {return strings.clone(filepath.dir(parent),allocator),true} - folder=parent - } -} - // A source a plugin imports whole (editor.Source_Importer), prepared for the // assets of a root: claims says whether it is one, by its path and up to // importer.SNIFF_BYTES of its head, and run imports it as import_source does, // with the state its plugin prepared. scene: the scene it wrote, in // allocator, "" for none. Whole_Source :: struct { + name:string, // what imports it, as its package records it (asset.Package_Origin.importer) + version:u16, // the format version its run gives importer.import_version claims:proc(path:string,head:[]u8) -> bool, run:proc(state:rawptr,source:string,options:importer.Import_Options,allocator:mem.Allocator) -> (report:importer.Import_Report,scene:string,error:importer.Error), state:rawptr, @@ -168,19 +156,18 @@ whole_find :: proc(wholes:[]Whole_Source,path:string) -> (whole:Whole_Source,cla } // Imports a source file in the assets folder of a project or a plugin -// (options.root; "": source_root): one of wholes claims is imported whole by +// (options.root; "": importer.source_root): one of wholes claims is imported whole by // it, before the formats; a model writes its assets and, when it // has nodes, the prefab of its hierarchy, which the report lists last; an -// image or a sound writes one asset. The sidecar of the source -// (.import, Import_Report.saved) keeps the settings it was imported -// with, the folder of its assets and the id of each of them: importing the -// source again gives the same assets. What the import builds that nobody -// edits goes to the imported folder of the root (importer.Sidecar), whatever -// the format that reads it. +// image or a sound writes one asset. The package of the source in +// options.store keeps the settings it was imported with and the id of each +// asset: importing the source again gives the same assets. What the import +// builds that nobody edits is a blob of its build, whatever the format that +// reads it, and its documents go to options.documents. import_source :: proc(path:string,options:importer.Import_Options={},formats:[]importer.Format=nil,wholes:[]Whole_Source=nil,allocator:=context.allocator) -> (report:importer.Import_Report,error:importer.Error) { formats:=formats if formats!=nil else FORMATS[:] options:=options - if options.root=="" {options.root,_=source_root(path,context.temp_allocator)} + if options.root=="" {options.root,_=importer.source_root(path,context.temp_allocator)} if whole,claimed:=whole_find(wholes,path);claimed { options.importer_code=whole.code scene:string diff --git a/cook/import/sources/support_test.odin b/cook/import/sources/support_test.odin index 41de9397d..d0b3f96ac 100644 --- a/cook/import/sources/support_test.odin +++ b/cook/import/sources/support_test.odin @@ -5,6 +5,7 @@ import "core:math/linalg" import "core:mem" import "core:os" import "core:path/filepath" +import "core:strconv" import "core:strings" import "core:testing" import "mirai:asset" @@ -15,6 +16,7 @@ import "mirai:cook/import/obj" import "mirai:cook/import/usd" import "mirai:std/base" import mm "mirai:std/math" +import "mirai:store" FIXTURES :: #directory+"/../../../assets/fixtures/" DEMO :: #directory+"/../../../demo/assets/" @@ -35,7 +37,7 @@ write_text :: proc(folder,name,text:string) -> string { // A Library that gives no formats and the types of the game context.user_ptr // points to: what sync and mirai import write the prefab of a model in. -test_library :: proc(root:string,allocator:mem.Allocator) -> Library_Imports { +test_library :: proc(root:string,s:^store.Store,allocator:mem.Allocator) -> Library_Imports { return {world=(^core.World_Info)(context.user_ptr)} } @@ -68,9 +70,9 @@ string_of :: proc(field:[]u8) -> string { } // Writes model as the import of a copy of its source in the assets of the -// test project at root, as import_write takes it: a source in no assets -// folder imports nowhere. -write_into :: proc(model:^importer.Imported_Model,root:string,options:importer.Import_Options={}) -> (report:importer.Import_Report,error:importer.Error) { +// test project at root, as import_write takes it, into the store of the test +// (test_options). +write_into :: proc(t:^testing.T,model:^importer.Imported_Model,root:string,options:importer.Import_Options={}) -> (report:importer.Import_Report,error:importer.Error) { assets:=fmt.tprintf("%s/assets",root) copied:=fmt.tprintf("%s/%s",assets,filepath.base(model.source)) if model.source!=copied { @@ -81,7 +83,10 @@ write_into :: proc(model:^importer.Imported_Model,root:string,options:importer.I } options:=options options.root=root - return importer.import_write(model,options) + opened:=test_options(t,model.source,options) + report,error=importer.import_write(model,opened) + objects_written(opened.store,&report) + return } // Copies files of a folder (FIXTURES, DEMO), by their paths under it, into @@ -99,6 +104,93 @@ copy_in :: proc(root,from:string,files:..string) -> string { return first } +// options of an import of the source at path as a project that adds it +// imports it: in the store of the test, /store, with its documents in +// the folder of the source (root: options.root, else the project whose assets +// hold the source, else its folder), unless options give them. +test_options :: proc(t:^testing.T,path:string,options:importer.Import_Options={},loc:=#caller_location) -> importer.Import_Options { + options:=options + root:=options.root + if root=="" {root,_=importer.source_root(path,context.temp_allocator)} + if root=="" {root=filepath.dir(path)} + os.make_directory_all(root) + // A root that cannot hold a store leaves none: the import says so. + if options.store==nil { + if s,error:=store.open(fmt.tprintf("%s/store",root),context.temp_allocator);error==.None { + options.store=new(store.Store,context.temp_allocator) + options.store^=s + } + } + if options.documents=="" {options.documents=filepath.dir(importer.path_key(path))} + return options +} + +// import_source with test_options. +import_test :: proc(t:^testing.T,path:string,options:importer.Import_Options={},formats:[]importer.Format=nil,wholes:[]Whole_Source=nil,allocator:=context.allocator,loc:=#caller_location) -> (report:importer.Import_Report,error:importer.Error) { + opened:=test_options(t,path,options,loc) + report,error=import_source(path,opened,formats,wholes,allocator) + objects_written(opened.store,&report,allocator) + return +} + +// Gives each output of the cache a report lists, an object of the store, a +// file of its bytes beside the store named by its address (objects_folder), +// as its path: what tests read and compare. +objects_written :: proc(s:^store.Store,report:^importer.Import_Report,allocator:=context.allocator) { + if s==nil {return} + for &written in report.assets { + if written.path!="" || written.object=={} {continue} + digits:[64]u8 + folder:=objects_folder(s) + os.make_directory_all(folder) + path:=fmt.tprintf("%s/%s",folder,store.hex(written.object,&digits)) + if !os.exists(path) {asset.object_file_write(s,written.object,path)} + delete(written.path,allocator) + written.path=strings.clone(path,allocator) + } +} + +// Where objects_written writes the files of objects of s. +objects_folder :: proc(s:^store.Store) -> string {return fmt.tprintf("%s-objects",s.root)} + +// A store at folder, in the temp allocator; nil when it cannot be opened. +test_store :: proc(t:^testing.T,folder:string,loc:=#caller_location) -> ^store.Store { + s,error:=store.open(folder,context.temp_allocator) + if !testing.expect_value(t,error,store.Error.None,loc=loc) {return nil} + kept:=new(store.Store,context.temp_allocator) + kept^=s + return kept +} + +// The registry of the assets of the project at root with the cache of the +// package the import of report wrote listed at their top, as a project that +// pins it lists them. +registry_with :: proc(t:^testing.T,root:string,report:^importer.Import_Report,r:^asset.Registry,loc:=#caller_location) -> bool { + s:=test_store(t,fmt.tprintf("%s/store",root),loc) + if s==nil {return false} + list:asset.Packages + list.packages=make([dynamic]asset.Project_Package,context.temp_allocator) + append(&list.packages,asset.Project_Package{id=report.package_id,manifest=report.manifest}) + resolved,error:=asset.packages_resolve(s,"",&list,context.temp_allocator) + if !testing.expect_value(t,error,asset.Error.None,loc=loc) {return false} + asset.registry_init(r,fmt.tprintf("%s/assets",root),"",context.allocator,resolved.listed[:]) + r.store=s + return testing.expect_value(t,asset.registry_scan(r),asset.Error.None,loc=loc) +} + +// The results a cache of s keeps (cook/cache), in the temp allocator. +cached_refs :: proc(s:^store.Store,name:string) -> []store.Ref { + buffer:[store.NAME_MAX]u8 + refs,_:=store.ref_list(s,asset.cache_namespace(name,&buffer),context.temp_allocator) + return refs +} + +// The key of a result of a cache by the name of its ref, -. +cached_key :: proc(ref:store.Ref) -> u64 { + key,_:=strconv.parse_u64_of_base(ref.name[:min(16,len(ref.name))],16) + return key +} + // A folder of a test as the root of a project, with its assets folder, where // the sources of its imports are. "" when there is none. test_project :: proc(t:^testing.T,name:string) -> (root,assets:string) { diff --git a/cook/import/sources/sync.odin b/cook/import/sources/sync.odin deleted file mode 100644 index 52423dc82..000000000 --- a/cook/import/sources/sync.odin +++ /dev/null @@ -1,293 +0,0 @@ -package sources - -import "core:fmt" -import "core:io" -import "core:mem" -import "core:os" -import "core:path/filepath" -import "core:strings" -import "core:time" -import "mirai:asset" -import "mirai:core" -import importer "mirai:cook/import" -import "mirai:project" -import "mirai:std/base" -import "mirai:std/jobs" -import "mirai:std/serial" - -// Builds again what the imports of a project or a plugin keep in its imported -// folder, which is not versioned, from the sources in its assets folder and -// their sidecars: what every build of a project runs. An output of the cache -// is current when its file is named by the key of what its import builds from -// now (importer.import_key): the bytes of the source and of the files it -// read, its settings and the importer (for a source a plugin imports whole, -// the code of that plugin), with textures built Final, or Fast under -// Sync_Options.fast_textures, which also takes them Final; a file whose size -// and modification time the cache of the root knows is not read -// (importer.Source_Hashes). An import of documents alone is current by its -// stamp, named the same way. A source with an output that is not current, or -// with a document of a version of its kind this build does not read, is -// imported, with the formats of the engine or, for one none of them reads, -// the importer of a plugin of the project that claims it whole or else the -// formats the plugins add; a file of the folder that no sidecar with its -// source lists is removed. A document an import writes over that changed -// since it last wrote it is left as it is, with a warning (Changed.Keep). - -// What the library of the editor of the project at root gives its imports: -// the formats of models and the importers of whole sources its plugins add, -// with what they read prepared for its assets, and the types of its game, -// which the prefab of a model is written in; release(state) lets them go. -// problem: why there are none. Only a program that links the editor reads it -// (editor.import_library). -Library :: #type proc(root:string,allocator:mem.Allocator) -> Library_Imports -Library_Imports :: struct { - formats:[]importer.Format, - wholes:[]Whole_Source, - world:^core.World_Info, - release:proc(state:rawptr), - state:rawptr, - problem:string, -} - -Sync_Options :: struct { - texture_cache,mesh_cache:string, // "": those of the root (project.content_cache_path) - rebuild:bool, // imports every source again - // Textures built Fast, as the editor imports them, unless the texture - // cache keeps them Final: what a build that does not cook needs. Outputs - // built Final are current too. False: Final, what a cooked game ships. - fast_textures:bool, - worker:jobs.Worker_Ref, - library:Library, // nil: the formats of the engine alone, and no model imports - // Where it says, as it goes, each source it imports, why and how far it - // is (importer.Progress); nil: nowhere. - progress:io.Writer, -} - -Sync_Report :: struct { - sources:int, // with outputs - imported:int, // imported now - removed:int, // files of the imported folder that no sidecar lists, removed - failures:[dynamic]string, // a source that did not import, with the reason - warnings:[dynamic]string, - allocator:mem.Allocator, -} - -sync_report_destroy :: proc(report:^Sync_Report) { - for text in report.failures {delete(text,report.allocator)} - for text in report.warnings {delete(text,report.allocator)} - delete(report.failures) - delete(report.warnings) - report^={} -} - -// Brings the imported folder of the project or the plugin at root up to date -// with the sources of its assets folder. The report, in allocator, counts -// what was imported and lists the sources that did not import: with one, the -// folder is as up to date as the others allow, and nothing is removed from it. -sync :: proc(root:string,options:Sync_Options={},allocator:=context.allocator) -> (report:Sync_Report) { - report.allocator=allocator - report.failures.allocator,report.warnings.allocator=allocator,allocator - scratch:mem.Dynamic_Arena - mem.dynamic_arena_init(&scratch,allocator,allocator,block_size=1<<20,out_band_size=1<<20) - defer mem.dynamic_arena_destroy(&scratch) - context.temp_allocator=base.dynamic_arena_allocator(&scratch) - - top:=project.path_absolute(root,context.temp_allocator) - imported:=project.imported_path(top,context.temp_allocator) - assets:=project.folder_path(top,.Assets,context.temp_allocator) - registry:asset.Registry - asset.registry_init(®istry,assets,"",context.temp_allocator,imported) - asset.registry_scan(®istry) - import_options:=importer.Import_Options{ - root=top,registry=®istry,rebuild=options.rebuild,changed=.Keep,worker=options.worker,fast_textures=options.fast_textures, - texture_cache=options.texture_cache if options.texture_cache!="" else project.content_cache_path(top,project.TEXTURE_CACHE,context.temp_allocator), - mesh_cache=options.mesh_cache if options.mesh_cache!="" else project.content_cache_path(top,project.MESH_CACHE,context.temp_allocator), - } - // The library, read once a source needs a format of its plugins or a model imports. - library:Library_Read - defer if library.release!=nil {library.release(library.state)} - hashes_path:=project.cache_path(top,importer.SOURCE_HASHES,allocator=context.temp_allocator) - hashes:=importer.source_hashes_load(hashes_path,context.temp_allocator) - defer importer.source_hashes_save(&hashes,hashes_path) - listed:=make(map[string]struct{},context.temp_allocator) // the files of the imported folder the sidecars list - versions:=kind_versions(top,context.temp_allocator) - complete:=true - // The sources to import, found first so that each line says how many there are. - Pending :: struct {source:string,formats:[]importer.Format,wholes:[]Whole_Source,kind:Source_Kind,version:u32,code:u64,why:string} - pending:=make([dynamic]Pending,context.temp_allocator) - todo:=make([dynamic]string,context.temp_allocator) - append(&todo,assets) - for next:=0;next []string { - files:=make([dynamic]string,context.temp_allocator) - for output in sidecar.outputs { - known,_:=asset.kind_info(output.kind) - if asset.import_cached(output.kind) {append(&files,asset.imported_file(imported,output.id,key,known.extension,context.temp_allocator))} - } - if len(files)==0 && len(sidecar.outputs)>0 {append(&files,asset.imported_file(imported,sidecar.outputs[0].id,key,importer.STAMP_EXTENSION,context.temp_allocator))} - return files[:] -} - -// Whether the imported folder has every file of an import built from key. -@(private="file") -outputs_held :: proc(sidecar:importer.Sidecar,imported:string,key:u64) -> bool { - for file in outputs_files(sidecar,imported,key) { - if !os.exists(file) {return false} - } - return true -} - -// Whether this build reads every document the sidecar lists that the assets -// have: one of another version of its kind is made again by importing its -// source. versions: of each kind this build knows (kind_versions). -@(private="file") -documents_read :: proc(sidecar:importer.Sidecar,registry:^asset.Registry,versions:map[asset.Fourcc]u16) -> bool { - for output in sidecar.outputs { - if asset.import_cached(output.kind) {continue} - file,found:=asset.registry_find(registry,output.id) - if !found {continue} - if version,known:=versions[file.kind];known && file.kind_version!=version {return false} - } - return true -} - -// The version of each kind of asset this build reads: those of the engine -// and those the plugins of the engine and of the project at or above root -// declare. -@(private="file") -kind_versions :: proc(root:string,allocator:=context.allocator) -> map[asset.Fourcc]u16 { - versions:=make(map[asset.Fourcc]u16,allocator) - for info in asset.KINDS {versions[info.kind]=info.version} - names:=project.FOLDER_NAMES - plugins:=make([dynamic]project.Plugin,context.temp_allocator) - if engine:=project.engine_root_find("",context.temp_allocator);engine!="" { - _=project.plugins_discover(project.path(engine,names[.Plugins],allocator=context.temp_allocator),.Engine,&plugins,context.temp_allocator) - } - for folder:=root;folder!="" && folder!=".";folder=filepath.dir(folder) { - if os.is_file(project.path(folder,project.PROJECT_FILE,allocator=context.temp_allocator)) { - _=project.plugins_discover(project.path(folder,names[.Plugins],allocator=context.temp_allocator),.Project,&plugins,context.temp_allocator) - break - } - if filepath.dir(folder)==folder {break} - } - for &plugin in plugins { - for &kind in plugin.manifest.asset_kinds { - if letters:=base.text_get(kind.kind[:]);len(letters)==4 {versions[asset.fourcc(letters)]=kind.version} - } - } - return versions -} - -@(private="file") -Library_Read :: struct {using imports:Library_Imports,read:bool} - -@(private="file") -library_read :: proc(library:^Library_Read,options:Sync_Options,top:string,import_options:^importer.Import_Options) { - if library.read || options.library==nil {return} - library.imports=options.library(top,context.temp_allocator) - import_options.world=library.world - library.read=true -} diff --git a/cook/import/sources/write_test.odin b/cook/import/sources/write_test.odin index 1d886ba6e..e97beb39f 100644 --- a/cook/import/sources/write_test.odin +++ b/cook/import/sources/write_test.odin @@ -14,6 +14,7 @@ import stbi "vendor:stb/image" import "mirai:asset" import anim "mirai:std/anim" import "mirai:std/base" +import "mirai:store" import mm "mirai:std/math" import "mirai:std/serial" import importer "mirai:cook/import" @@ -46,7 +47,7 @@ png_bytes :: proc(width,height:int,rgb:[3]u8,masked:=false) -> []u8 { } read_asset :: proc(t:^testing.T,written:importer.Written,types:^asset.Types) -> (a:asset.Asset,ok:bool) { - error:=asset.asset_load(&a,written.path) + error:=asset.asset_load(&a,written.path) if written.path!="" else .Not_Found ok=testing.expectf(t,error==.None,"%s: %v",written.path,error) if ok { testing.expectf(t,asset.blob_verify_hash(&a.blob),"%s: the hash matches",written.path) @@ -69,14 +70,14 @@ a_written_obj_is_read_back_with_its_material_and_its_dependencies :: proc(t:^tes model,ok:=import_ok(t,FIXTURES+"concave.obj") if !ok {return} defer importer.model_destroy(&model) - report,error:=write_into(&model,fmt.tprintf("%s/out",folder)) + report,error:=write_into(t,&model,fmt.tprintf("%s/out",folder)) defer importer.report_destroy(&report) testing.expect_value(t,error.code,importer.Error_Code.None) testing.expect_value(t,len(report.assets),2) types:asset.Types;asset.types_init(&types);defer asset.types_destroy(&types) mesh_written,has_mesh:=written_of(&report,asset.KIND_MESH,"ConcaveL") material_written,has_material:=written_of(&report,asset.KIND_MATERIAL,"red") - testing.expect(t,has_mesh && has_material && strings.contains(mesh_written.path,"/.mirai/imported/") && strings.has_suffix(material_written.path,"/assets/concave/red.mat") && mesh_written.bytes>0) + testing.expect(t,has_mesh && has_material && strings.contains(mesh_written.path,"/store-objects/") && strings.has_suffix(material_written.path,"/assets/concave/red.mat") && mesh_written.bytes>0) if !has_mesh || !has_material {return} a,loaded:=read_asset(t,mesh_written,&types) @@ -108,7 +109,7 @@ a_written_model_with_textures_links_materials_to_them :: proc(t:^testing.T) { model,ok:=import_ok(t,DEMO+"gallery/plant.gltf") if !ok {return} defer importer.model_destroy(&model) - report,error:=write_into(&model,folder) + report,error:=write_into(t,&model,folder) defer importer.report_destroy(&report) testing.expect_value(t,error.code,importer.Error_Code.None) testing.expect_value(t,len(report.warnings),0) @@ -129,7 +130,7 @@ a_written_model_with_textures_links_materials_to_them :: proc(t:^testing.T) { plain_folder:=base.test_folder(t,"write-plant-rgba8") if plain_folder=="" {return} - plain_report,plain_error:=write_into(&model,plain_folder,{textures=.Uncompressed,ignore_stored=true}) + plain_report,plain_error:=write_into(t,&model,plain_folder,{textures=.Uncompressed,ignore_stored=true}) defer importer.report_destroy(&plain_report) testing.expect_value(t,plain_error.code,importer.Error_Code.None) if plain_written,found:=written_of(&plain_report,asset.KIND_TEXTURE,"leaves_color");testing.expect(t,found) { @@ -162,7 +163,7 @@ a_skinned_mesh_is_written_with_its_rig :: proc(t:^testing.T) { tracks:=[1]asset.Animation_Track_Desc{{target=base.text_get(model.skeletons[0].bones[len(model.skeletons[0].bones)-1].path[:]),property=.Rotation,keys=keys[:]}} bend:=[1]asset.Animation_Clip_Desc{{name="bend",tracks=tracks[:]}} brought:=model.animations;model.animations=bend[:] - report,error:=write_into(&model,folder) + report,error:=write_into(t,&model,folder) model.animations=brought defer importer.report_destroy(&report) testing.expect_value(t,error.code,importer.Error_Code.None) @@ -215,11 +216,11 @@ the_prefab_of_an_import_is_written_in_the_layout_of_the_game :: proc(t:^testing. if root=="" {return} world:=prefab.import_world(t,c.animated) if world==nil {return} - report,error:=import_source(copy_in(root,DEMO,..c.files),{world=world,root=root,textures=.Uncompressed}) + report,error:=import_test(t,copy_in(root,DEMO,..c.files),{world=world,root=root,textures=.Uncompressed}) if !testing.expectf(t,error.code==.None,"%s: %s",c.files[0],error.message) {return} defer importer.report_destroy(&report) p:=author.Project{name="layout",assets_root=assets,info=world} - asset.registry_init(&p.registry,assets,imported=fmt.tprintf("%s/.mirai/imported",root)) + asset.registry_init(&p.registry,assets) asset.types_init(&p.types) defer {asset.registry_destroy(&p.registry);asset.types_destroy(&p.types);author.scratch_world_free(&p.scratch)} p.assets={&p.registry,&p.types} @@ -261,7 +262,7 @@ clips_onto_a_skeleton_write_only_the_anim :: proc(t:^testing.T) { bend:=[2]asset.Animation_Clip_Desc{{name="bend",tracks=tracks[:]},{name="cue",markers=cue[:]}} brought:=model.animations;model.animations=bend[:] player:=base.Asset_Id{0x1234,0x5678} - report,error:=write_into(&model,folder,{settings={skeleton=player}}) + report,error:=write_into(t,&model,folder,{settings={skeleton=player}}) model.animations=brought defer importer.report_destroy(&report) testing.expect_value(t,error.code,importer.Error_Code.None) @@ -288,7 +289,7 @@ a_humanoid_skeleton_is_written_with_its_body_and_keeps_what_a_person_set :: proc model,ok:=import_ok(t,DEMO+"characters/mirai/mirai.glb",options) if !ok {return} defer importer.model_destroy(&model) - report,error:=write_into(&model,folder,options) + report,error:=write_into(t,&model,folder,options) defer importer.report_destroy(&report) testing.expect_value(t,error.code,importer.Error_Code.None) written,found:=written_of(&report,asset.KIND_SKELETON,model.skeletons[0].name) @@ -373,7 +374,7 @@ imported_clips_that_move_the_bone_above_the_pelvis_carry_it_as_their_root :: pro {name="mixamo",tracks={{target=hips,property=.Position,keys=bob[:]}}}, } brought:=model.animations;model.animations=clips[:] - report,error:=write_into(&model,folder,options) + report,error:=write_into(t,&model,folder,options) model.animations=brought defer importer.report_destroy(&report) testing.expect_value(t,error.code,importer.Error_Code.None) @@ -432,7 +433,7 @@ importing_again_keeps_what_was_changed_in_a_material_and_takes_the_rest_from_the import_once :: proc(t:^testing.T,source,folder:string,options:importer.Import_Options,types:^asset.Types) -> string { model,ok:=import_ok(t,source,options);if !ok {return ""} defer importer.model_destroy(&model) - report,error:=write_into(&model,folder,options);defer importer.report_destroy(&report) + report,error:=write_into(t,&model,folder,options);defer importer.report_destroy(&report) if !testing.expect_value(t,error.code,importer.Error_Code.None) {return ""} written,found:=written_of(&report,asset.KIND_MATERIAL,"Terracotta") if !testing.expect(t,found) {return ""} @@ -469,14 +470,14 @@ importing_again_keeps_the_identity_of_every_asset :: proc(t:^testing.T) { model,ok:=import_ok(t,DEMO+"gallery/plant.gltf") if !ok {return} defer importer.model_destroy(&model) - first,first_error:=write_into(&model,fmt.tprintf("%s/one",folder)) + first,first_error:=write_into(t,&model,fmt.tprintf("%s/one",folder)) defer importer.report_destroy(&first) testing.expect_value(t,first_error.code,importer.Error_Code.None) again,again_ok:=import_ok(t,DEMO+"gallery/plant.gltf") if !again_ok {return} defer importer.model_destroy(&again) - second,second_error:=write_into(&again,fmt.tprintf("%s/one",folder)) + second,second_error:=write_into(t,&again,fmt.tprintf("%s/one",folder)) defer importer.report_destroy(&second) testing.expect_value(t,second_error.code,importer.Error_Code.None) testing.expect_value(t,len(first.assets),len(second.assets)) @@ -484,23 +485,31 @@ importing_again_keeps_the_identity_of_every_asset :: proc(t:^testing.T) { testing.expectf(t,item.id==second.assets[i].id && item.path==second.assets[i].path,"%s keeps its id and its file",item.name) testing.expect(t,item.id!=base.Asset_Id{}) } + testing.expect(t,first.manifest==second.manifest,"and its package is the same manifest") ids:=make(map[base.Asset_Id]bool,context.temp_allocator) for item in first.assets { testing.expect(t,!ids[item.id],"ids are distinct") ids[item.id]=true } - // The documents beside the source and what nobody edits in the imported folder, with the sidecar that lists them. + // The documents in the folder of the import, and every asset a blob its package lists, with + // the prefab import_write leaves to its caller. documents,_:=os.read_all_directory_by_path(fmt.tprintf("%s/one/assets/plant",folder),context.temp_allocator) - cached,_:=os.read_all_directory_by_path(fmt.tprintf("%s/one/.mirai/imported",folder),context.temp_allocator) - testing.expect(t,len(documents)+len(cached)==len(first.assets) && os.exists(fmt.tprintf("%s/one/assets/plant.gltf.import",folder))) - - // A copy of a project, its sources and their sidecars with its documents, imports again with the ids of its assets. + s:=test_store(t,fmt.tprintf("%s/one/store",folder)) + p,_:=asset.package_get(s,first.manifest,context.temp_allocator) + cached:=0 + for a in p.assets {cached+=int(asset.import_cached(a.kind))} + testing.expectf(t,len(documents)+cached==len(first.assets) && len(p.assets)==len(first.assets)+int(first.prefab!={}),"%d documents, %d of the cache, %d listed, %d written",len(documents),cached,len(p.assets),len(first.assets)) + + // A copy of a project, its sources and documents, imported with the manifest the project pins, keeps the ids of its assets. + shared:=test_store(t,fmt.tprintf("%s/shared",folder)) copy_in(fmt.tprintf("%s/a",folder),DEMO,"gallery/vase.obj","gallery/vase.mtl","gallery/vase_color.png") ids_by_root:[2][dynamic]base.Asset_Id + pinned:store.Hash for root,i in ([]string{"a","b"}) { if i==1 {testing.expect(t,base.copy_directory_all(fmt.tprintf("%s/b/assets",folder),fmt.tprintf("%s/a/assets",folder))==nil)} - report,vase_error:=import_source(fmt.tprintf("%s/%s/assets/gallery/vase.obj",folder,root),{world=prefab.import_world(t)}) + report,vase_error:=import_test(t,fmt.tprintf("%s/%s/assets/gallery/vase.obj",folder,root),{world=prefab.import_world(t),store=shared,manifest=pinned}) testing.expect_value(t,vase_error.code,importer.Error_Code.None) + pinned=report.manifest ids_by_root[i].allocator=context.temp_allocator for item in report.assets {append(&ids_by_root[i],item.id)} importer.report_destroy(&report) @@ -520,7 +529,7 @@ images_that_are_missing_or_not_images_leave_a_warning_and_no_reference :: proc(t if !ok {return} defer importer.model_destroy(&model) testing.expect_value(t,len(model.warnings),0) - report,error:=write_into(&model,fmt.tprintf("%s/out",folder)) + report,error:=write_into(t,&model,fmt.tprintf("%s/out",folder)) defer importer.report_destroy(&report) testing.expect_value(t,error.code,importer.Error_Code.None) testing.expect_value(t,len(report.warnings),2) @@ -558,7 +567,7 @@ an_embedded_image_is_a_texture_of_its_whole_size :: proc(t:^testing.T) { defer importer.model_destroy(&model) testing.expect_value(t,len(model.images),1) testing.expect(t,model.images[0].path=="" && len(model.images[0].data)>0 && model.images[0].name=="wood") - report,error:=write_into(&model,fmt.tprintf("%s/out",folder),{textures=.Uncompressed}) + report,error:=write_into(t,&model,fmt.tprintf("%s/out",folder),{textures=.Uncompressed}) defer importer.report_destroy(&report) testing.expect_value(t,error.code,importer.Error_Code.None) testing.expect_value(t,len(report.warnings),0) @@ -641,7 +650,7 @@ gltf_and_glb_write_pbr_maps_with_color_spaces_channels_and_independent_uvs :: pr testing.expect(t,(image.path=="")==binary && (len(image.data)>0)==binary) } options:=importer.Import_Options{root=folder,textures=.Uncompressed} - report,error:=write_into(&model,fmt.tprintf("%s/%s",folder,"glb-out" if binary else "gltf-out"),options) + report,error:=write_into(t,&model,fmt.tprintf("%s/%s",folder,"glb-out" if binary else "gltf-out"),options) if !testing.expect_value(t,error.code,importer.Error_Code.None) {continue} defer importer.report_destroy(&report) testing.expect_value(t,len(report.warnings),0) @@ -699,7 +708,7 @@ gltf_and_glb_write_pbr_maps_with_color_spaces_channels_and_independent_uvs :: pr if testing.expect(t,asset.mesh_level_decode(mesh.geometry,0,&vertices,&indices)) {testing.expect_value(t,vertices[0].uv1,[2]f32{0.25,0.75})} } } - blocks,blocks_error:=write_into(&model,fmt.tprintf("%s/%s",folder,"glb-out" if binary else "gltf-out"),{textures=.Blocks,ignore_stored=true}) + blocks,blocks_error:=write_into(t,&model,fmt.tprintf("%s/%s",folder,"glb-out" if binary else "gltf-out"),{textures=.Blocks,ignore_stored=true}) if !testing.expect_value(t,blocks_error.code,importer.Error_Code.None) {continue} defer importer.report_destroy(&blocks) testing.expect_value(t,len(blocks.warnings),0) @@ -723,24 +732,33 @@ a_standalone_image_becomes_a_texture :: proc(t:^testing.T) { if folder=="" {return} path:=fmt.tprintf("%s/Bricks Color.png",folder) _=os.write_entire_file(path,png_bytes(16,16,{150,60,40})) - report,error:=importer.import_texture(path,{root=root,image=.Color}) + options:=test_options(t,path,{root=root,image=.Color}) + report,error:=importer.import_texture(path,options) defer importer.report_destroy(&report) + objects_written(options.store,&report) testing.expect_value(t,error.code,importer.Error_Code.None) - testing.expect_value(t,len(report.assets),1) - testing.expect(t,report.assets[0].id!={} && strings.contains(report.assets[0].path,"/.mirai/imported/")) - again,again_error:=importer.import_texture(path,{root=root,image=.Normal,ignore_stored=true}) + if !testing.expect_value(t,len(report.assets),1) {return} + testing.expect(t,report.assets[0].id!={} && strings.contains(report.assets[0].path,"/store-objects/")) + normal:=options + normal.image,normal.ignore_stored=.Normal,true + again,again_error:=importer.import_texture(path,normal) defer importer.report_destroy(&again) + objects_written(options.store,&again) testing.expect_value(t,again_error.code,importer.Error_Code.None) - testing.expect(t,again.assets[0].id==report.assets[0].id && again.assets[0].path!=report.assets[0].path && !os.exists(report.assets[0].path),"imported again as a normal map, the image is the same texture, and its file of before goes") + if !testing.expect_value(t,len(again.assets),1) {return} + testing.expect(t,again.assets[0].id==report.assets[0].id && again.assets[0].path!=report.assets[0].path && os.exists(report.assets[0].path),"imported again as a normal map, the image is the same texture, another object, and the store keeps the one of before") _=os.write_entire_file(path,[]u8{1,2,3}) - _,bad:=importer.import_texture(path,{root=root,image=.Color}) + _,bad:=importer.import_texture(path,options) testing.expect_value(t,bad.code,importer.Error_Code.Unsupported) - testing.expect(t,os.exists(again.assets[0].path),"a failed import leaves the files it had") - _,missing:=importer.import_texture(fmt.tprintf("%s/none.png",folder),{root=root,image=.Color}) + testing.expect(t,asset.object_whole(options.store,again.assets[0].object),"a failed import leaves the objects it had") + _,missing:=importer.import_texture(fmt.tprintf("%s/none.png",folder),options) testing.expect_value(t,missing.code,importer.Error_Code.Not_Found) - _,outside:=importer.import_texture(fmt.tprintf("%s/outside.png",root),{root=root,image=.Color}) - testing.expect(t,outside.code==.Not_Found && strings.contains(outside.message,"is not in the assets folder"),outside.message) + outside_path:=fmt.tprintf("%s/outside.png",root) + _=os.write_entire_file(outside_path,png_bytes(4,4,{1,2,3})) + outside,outside_error:=importer.import_texture(outside_path,options) + defer importer.report_destroy(&outside) + testing.expectf(t,outside_error.code==.None && len(outside.assets)==1,"an image outside the assets imports too: %s",outside_error.message) } // A Radiance image is an image whose light above 1 the texture keeps, in BC6H. @@ -754,7 +772,7 @@ a_radiance_image_becomes_an_hdr_texture :: proc(t:^testing.T) { for _ in 0..<8 {append(&hdr,200,200,200,135)} // 100 _=os.write_entire_file(path,hdr[:]) testing.expect_value(t,source_kind(path),Source_Kind.Image) - report,error:=import_source(path,{world=prefab.import_world(t)}) + report,error:=import_test(t,path,{world=prefab.import_world(t)}) defer importer.report_destroy(&report) if !testing.expectf(t,error.code==.None,"%s",error.message) {return} a:asset.Asset @@ -785,7 +803,7 @@ embedded_images_with_the_same_name_keep_distinct_texture_ids :: proc(t:^testing. "accessors":[{"bufferView":0,"componentType":5126,"count":3,"type":"VEC3","min":[0,0,0],"max":[1,1,0]},{"bufferView":1,"componentType":5126,"count":3,"type":"VEC2"}]}`},context.temp_allocator) path:=write_text(folder,"names.gltf",source) model,ok:=import_ok(t,path);if !ok {return};defer importer.model_destroy(&model) - report,error:=write_into(&model,fmt.tprintf("%s/out",folder),{textures=.Uncompressed}) + report,error:=write_into(t,&model,fmt.tprintf("%s/out",folder),{textures=.Uncompressed}) if !testing.expect_value(t,error.code,importer.Error_Code.None) {return};defer importer.report_destroy(&report) testing.expect(t,len(report.image_ids)==2 && report.image_ids[0]!=report.image_ids[1]) testing.expect_value(t,len(report.warnings),0) @@ -817,7 +835,7 @@ names_are_made_safe_and_unique_as_file_names :: proc(t:^testing.T) { defer importer.model_destroy(&model) testing.expect_value(t,len(model.meshes),3) testing.expect(t,model.meshes[0].name!=model.meshes[2].name,"a repeated name is made unique in the model") - report,error:=write_into(&model,fmt.tprintf("%s/out",folder)) + report,error:=write_into(t,&model,fmt.tprintf("%s/out",folder)) defer importer.report_destroy(&report) testing.expect_value(t,error.code,importer.Error_Code.None) seen:=make(map[string]bool,context.temp_allocator) @@ -836,9 +854,9 @@ a_destination_that_cannot_be_a_folder_fails_without_leaving_files :: proc(t:^tes if !ok {return} defer importer.model_destroy(&model) blocker:=write_text(folder,"blocker","this is a file") - _,error:=write_into(&model,blocker) + _,error:=write_into(t,&model,blocker) testing.expect_value(t,error.code,importer.Error_Code.Write) - _,error=write_into(&model,fmt.tprintf("%s/inside",blocker)) + _,error=write_into(t,&model,fmt.tprintf("%s/inside",blocker)) testing.expect_value(t,error.code,importer.Error_Code.Write) } @@ -858,7 +876,7 @@ every_fixture_imports_writes_and_reads_back_with_its_dependencies_resolved :: pr model,ok:=import_ok(t,source,formats=text[:]) if !ok {continue} defer importer.model_destroy(&model) - report,error:=write_into(&model,out) + report,error:=write_into(t,&model,out) defer importer.report_destroy(&report) testing.expectf(t,error.code==.None && len(report.warnings)==0,"%s: %v %s",source,error.code,error.message) counts:[3]int @@ -883,8 +901,7 @@ every_fixture_imports_writes_and_reads_back_with_its_dependencies_resolved :: pr } testing.expectf(t,counts[0]==len(model.meshes) && counts[1]==len(model.materials),"%s: every mesh and material is an asset",source) registry:asset.Registry - asset.registry_init(®istry,fmt.tprintf("%s/assets",out),imported=fmt.tprintf("%s/.mirai/imported",out)) - testing.expect_value(t,asset.registry_scan(®istry),asset.Error.None) + if !registry_with(t,out,&report,®istry) {continue} missing:[dynamic]asset.Registry_Missing missing.allocator=context.temp_allocator asset.registry_missing(®istry,&missing) @@ -900,7 +917,7 @@ the_report_names_the_asset_of_every_row_of_the_model :: proc(t:^testing.T) { model,ok:=import_ok(t,DEMO+"gallery/plant.gltf") if !ok {return} defer importer.model_destroy(&model) - report,error:=write_into(&model,folder) + report,error:=write_into(t,&model,folder) defer importer.report_destroy(&report) testing.expect_value(t,error.code,importer.Error_Code.None) testing.expect_value(t,len(report.mesh_ids),1) @@ -953,7 +970,7 @@ gltf_anisotropy_with_a_roughness_texture_is_converted_texel_by_texel :: proc(t:^ base:=f32(128)/255 testing.expectf(t,abs(along-(base*base+(1-base*base)*0.36))<0.01 && abs(across-base*base)<0.01,"alphas %v and %v",along,across) testing.expect(t,model.images[rough].made.texels[0]==255 && model.images[rough].made.texels[2]==64,"occlusion and metallic stay") - report,error:=write_into(&model,fmt.tprintf("%s/out",folder),{}) + report,error:=write_into(t,&model,fmt.tprintf("%s/out",folder),{}) defer importer.report_destroy(&report) testing.expect_value(t,error.code,importer.Error_Code.None) testing.expect_value(t,len(report.warnings),0) diff --git a/cook/import/space_test.odin b/cook/import/space_test.odin index 26a760946..ef8458e6a 100644 --- a/cook/import/space_test.odin +++ b/cook/import/space_test.odin @@ -222,15 +222,15 @@ a_model_is_turned_to_minus_z_in_place_by_its_rig_or_by_the_front_its_settings_st the_front_of_a_source_is_a_setting_kept_with_it_that_makes_another_mesh :: proc(t:^testing.T) { for front in Front { stated:=Import_Settings{front=front} - parsed,ok:=sidecar_parse(sidecar_text({settings=stated},context.temp_allocator),context.temp_allocator) - testing.expectf(t,ok && parsed.settings==stated,"%v is read as written",front) + parsed,ok:=settings_parse(settings_text(stated,context.temp_allocator)) + testing.expectf(t,ok && parsed==stated,"%v is read as written",front) named,known:=front_from_name(strings.to_lower(fmt.tprint(front),context.temp_allocator)) testing.expectf(t,known && named==front,"%v is named in lowercase",front) for other in Front { if other!=front {testing.expectf(t,mesh_settings_hash(stated)!=mesh_settings_hash({front=other}),"%v and %v make the same mesh",front,other)} } } - _,ok:=sidecar_parse(`{"version": 4, "settings": {"front": "Up"}, "outputs": []}`,context.temp_allocator) + _,ok:=settings_parse(`{"front": "Up"}`) testing.expect(t,!ok,"a front the import does not know is not the settings of a source") } diff --git a/cook/import/write.odin b/cook/import/write.odin index d056ab6ea..e1acc8b24 100644 --- a/cook/import/write.odin +++ b/cook/import/write.odin @@ -17,16 +17,25 @@ import "mirai:std/base" import "mirai:std/jobs" import mm "mirai:std/math" import "mirai:std/serial" +import "mirai:store" Written :: struct { id:base.Asset_Id, kind:asset.Fourcc, name:string, - path:string, + path:string, // for a document, its file in the folder of the documents; "" for an object of the store alone + object:store.Hash, source:string, // the source as imports name it bytes:int, detail:string, // what it holds, for a report: levels and colliders of a mesh, format and size of a texture - kept:bool, // its file already was the one this import gives: nothing was built + kept:bool, // an object the store had was taken: nothing was built +} + +// Opens what an import wrote: a document from its file, an output of the +// cache from its object of s. +written_open :: proc(s:^store.Store,w:Written,a:^asset.Asset,allocator:=context.allocator) -> asset.Error { + if w.path!="" {return asset.asset_load(a,w.path,allocator)} + return asset.asset_open_object(a,s,w.object,allocator) } Import_Report :: struct { @@ -36,15 +45,23 @@ Import_Report :: struct { // The asset of each mesh, material and image of the model, by row; zero for an image that was not written. mesh_ids,material_ids,image_ids,skeleton_ids:[]base.Asset_Id, animation_id:base.Asset_Id, - // The prefab of a model, the owner its parts name: the id the sidecar of - // the source lists and the file that has it, else new ones. Zero for a - // model without nodes that had no prefab. import_source writes it, with - // prefab_info. + // The prefab of a model, the owner its parts name: the id the previous + // manifest lists, else a new one, and the file it is written to: in the + // folder of the documents (none without one), or for Outputs the file + // output_written takes. Zero for a model without nodes that had no + // prefab. import_source writes it, with prefab_info. prefab:base.Asset_Id, prefab_path:string, prefab_info:asset.Import_Info, - settings:Import_Settings, // what the source was imported with: what its sidecar keeps - saved:string, // the sidecar of the source, which the import wrote or found saying the same + settings:Import_Settings, // what the source was imported with: what its package keeps + // What the project's file of each document the import writes over in + // Import_Options.documents holds as it leaves it: the pin records it. + written:[dynamic]asset.Package_Written, + // The package the import wrote: its id, the hash of its manifest, its + // build, the key of the build and whether its textures are Final. + package_id:base.Asset_Id, + manifest,build,key:store.Hash, + final:bool, allocator:mem.Allocator, } @@ -60,11 +77,11 @@ report_destroy :: proc(report:^Import_Report) { delete(report.assets) delete(report.warnings) delete(report.programs) + delete(report.written) delete(report.mesh_ids,report.allocator) delete(report.material_ids,report.allocator) delete(report.image_ids,report.allocator) delete(report.skeleton_ids,report.allocator) - delete(report.saved,report.allocator) delete(report.prefab_path,report.allocator) report^={} } @@ -75,35 +92,24 @@ report_init :: proc(report:^Import_Report,allocator:=context.allocator) { report.assets.allocator=allocator report.warnings.allocator=allocator report.programs.allocator=allocator + report.written.allocator=allocator } -// Moves what src lists (assets, warnings and programs) to the end of dst and +// Moves what src lists (assets, warnings, programs and what was written) to the end of dst and // destroys src; both live in the same allocator. report_take :: proc(dst,src:^Import_Report) { append(&dst.assets,..src.assets[:]) append(&dst.warnings,..src.warnings[:]) append(&dst.programs,..src.programs[:]) - clear(&src.assets);clear(&src.warnings);clear(&src.programs) + append(&dst.written,..src.written[:]) + clear(&src.assets);clear(&src.warnings);clear(&src.programs);clear(&src.written) report_destroy(src) } -// A source path as imports name it: relative to options.root when it is inside, with forward slashes. -source_path :: proc(options:Import_Options,path:string,allocator:=context.temp_allocator) -> string { - absolute,error:=filepath.abs(path,context.temp_allocator) - if error!=nil {absolute=path} - cleaned,_:=filepath.clean(absolute,context.temp_allocator) - slashed,_:=strings.replace_all(cleaned,"\\","/",context.temp_allocator) - if options.root!="" { - root_absolute,root_error:=filepath.abs(options.root,context.temp_allocator) - if root_error!=nil {root_absolute=options.root} - root_clean,_:=filepath.clean(root_absolute,context.temp_allocator) - root,_:=strings.replace_all(root_clean,"\\","/",context.temp_allocator) - prefix:=strings.concatenate({strings.trim_right(root,"/"),"/"},context.temp_allocator) - if len(slashed)>len(prefix) && strings.equal_fold(slashed[:len(prefix)],prefix) { - return strings.clone(slashed[len(prefix):],allocator) - } - } - return strings.clone(slashed,allocator) +// A source path as imports name it, relative to its package: the name of its +// file, the same on every machine. +source_path :: proc(path:string,allocator:=context.temp_allocator) -> string { + return strings.clone(filepath.base(path_key(path)),allocator) } @(private) @@ -115,35 +121,50 @@ read_whole :: proc(path:string) -> ([]u8,serial.File_Error) { @(private) Writer :: struct { options:Import_Options, - folder:string, // where new documents go: the folder of the source, or the one of it its sidecar names - registry:^asset.Registry, // the assets of the root, where documents are found by id + folder:string, // where new documents go: under the folder of the documents; "": none are written there + registry:^asset.Registry, // the assets of the root, where documents are found by id; nil without a folder of the documents types:asset.Types, report:^Import_Report, - created:[dynamic]string, + created:[dynamic]string, // the documents the import wrote that were not there names:map[string]bool, // files of the folder new parts took, in lower case - source:string, // the source as imports name it (source_path) - sidecar:Sidecar, // of the source, as the import found it - taken:[]bool, // the outputs of sidecar the import gives again - outputs:[dynamic]Output, // what the import gives: the outputs of the sidecar it writes - paths:[dynamic]string, // the file of each of outputs - imported:string, // the imported folder of options.root path:string, // the source + origin:string, // the source as an absolute path with forward slashes + source:string, // the source as imports name it (source_path) + previous:asset.Package, // the manifest the source was ingested as, when stored + previous_manifest:store.Hash, + stored:bool, + package_id:base.Asset_Id, + taken:[]bool, // the assets of previous the import gives again + outputs:[dynamic]asset.Package_Asset, // what the import gives: the assets of the manifest it writes + paths:[dynamic]string, // the scratch file of each of outputs + targets:[dynamic]string, // the file of each document of outputs in the folder of the documents; "": none + objects:[dynamic]store.Hash, // the object of each of outputs once the store took it; zero: none + followed:[dynamic]Followed_Skeleton, // the baselines of the skeletons taken, which clips follow onto + scratch:string, // the folder parts are written in before the store takes them version:u32, // import_version of what reads the source - inputs:[dynamic]string, // the files read besides the source, as the sidecar lists them (writer_input) - key:u64, // import_key of the import, once a file of the cache is named; inputs are whole by then - previous:map[string]string, // the files of the imported folder under another key, by their name without it (writer_reuse) - previous_read:bool, + inputs:[dynamic]string, // the files read besides the source, as the manifest lists them (writer_input) + fixed:bool, // identity and key are fixed (writer_identity): inputs are whole + identity,key:store.Hash, + candidates:[dynamic]Candidate, // the builds outputs may be taken from (writer_candidates) + candidates_read:bool, + fast:bool, // a texture taken or built is Fast owner:base.Asset_Id, // of the parts: the prefab of a model - stored:bool, // the source has a sidecar whole:bool, // writes a model for Outputs: its prefab and clips are parts by their names given:int, // of outputs, those a part of the same kind and name is (parts_plan) - keeping:map[base.Asset_Id]string, // the documents changed since the import wrote them that it leaves as they are (Changed.Keep), with the file that takes what is written of each + keeping:map[base.Asset_Id]u64, // the documents changed since the import wrote them that it leaves as they are (Changed.Keep), with what the pin recorded of each rng:base.Rng, allocator:mem.Allocator, keep:mem.Allocator, // what lives as long as the writer: the temp allocator it began in error:Error, } +// A build of the package this machine has, whose objects an import takes. +@(private) +Candidate :: struct { + build:asset.Build, + current:bool, // of what the import is made from now (its key) +} + // The message goes to the allocator of the writer: the temp one where the // failure happens may be the one that ran out of memory. @(private) @@ -166,28 +187,24 @@ writer_warn :: proc(w:^Writer,format:string,args:..any) { append(&w.report.warnings,fmt.aprintf(format,..args,allocator=w.allocator)) } -// A document the writer follows (follows: what a person changed in it stays) -// records nothing of what it wrote: importing again never writes over a change. +// Lists an output the store took in the report. @(private) -record :: proc(w:^Writer,id:base.Asset_Id,kind:asset.Fourcc,name,path,source:string,existed:bool,detail:="",kept:=false,follows:=false) { - if _,keeping:=w.keeping[id];keeping { - progress_written(w.options.progress,len(w.sidecar.outputs)) - return - } - if !asset.import_cached(kind) { - for at:=len(w.outputs)-1;at>=0;at-=1 { - if w.outputs[at].id==id {w.outputs[at].written=0 if follows else written_hash(path);break} - } - } +record :: proc(w:^Writer,output:int,name:string,detail:="",kept:=false) { + o:=w.outputs[output] + path:=w.targets[output] size:int - if info,error:=os.stat(path,context.temp_allocator);error==nil {size=int(info.size)} + if path!="" { + if info,error:=os.stat(path,context.temp_allocator);error==nil {size=int(info.size)} + } else { + dependencies:=make([dynamic]base.Asset_Id,context.temp_allocator) + if header,error:=asset.object_header(w.options.store,w.objects[output],&dependencies);error==.None {size=int(header.total_size)} + } clone,_:=strings.clone(path,w.allocator) named,_:=strings.clone(name,w.allocator) - from,_:=strings.clone(source,w.allocator) + from,_:=strings.clone(w.source,w.allocator) described,_:=strings.clone(detail,w.allocator) - append(&w.report.assets,Written{id=id,kind=kind,name=named,path=clone,source=from,bytes=size,detail=described,kept=kept}) - if !existed {append(&w.created,clone)} - progress_written(w.options.progress,len(w.sidecar.outputs)) + append(&w.report.assets,Written{id=o.id,kind=o.kind,name=named,path=clone,object=w.objects[output],source=from,bytes=size,detail=described,kept=kept}) + progress_written(w.options.progress,len(w.previous.assets)) } @(private) @@ -204,74 +221,62 @@ import_info :: proc(source:string,name:string,index:int,size:u64,hash:u64,settin return info } -// The file of the cache an asset would be written to, open for its report, -// when it already is there: its name says it was built from what the import -// builds from now. Its owner, which the sidecar decides, is written again in -// its header when it changed. The caller closes it. -@(private="file") -writer_kept :: proc(w:^Writer,path:string,owner:base.Asset_Id,file:^asset.Asset) -> bool { - if w.options.rebuild || !os.exists(path) || asset.asset_load(file,path,context.temp_allocator)!=.None {return false} - if file.blob.header.owner==owner {return true} - if asset.header_owner_set(file.blob.data,owner)==.None && serial.file_write_atomic(path,file.blob.data)==.None { - file.blob.header.owner=owner - return true +// Reads the builds outputs may be taken from: that of what the import is made +// from now (which fixes it), Final or, with fast textures, Fast; and the one +// this machine chose for the previous manifest. Before any writer_reuse, on +// the writer's thread. +@(private) +writer_candidates :: proc(w:^Writer) { + if w.candidates_read {return} + w.candidates_read=true + w.candidates.allocator=w.keep + writer_identity(w) + s:=w.options.store + digits:[64]u8 + take :: proc(w:^Writer,s:^store.Store,namespace,name:string,current:bool) { + hash,error:=store.ref_get(s,namespace,name) + if error!=.None {return} + b,read_error:=asset.build_get(s,hash,w.keep) + if read_error==.None && b.package_id==w.package_id {append(&w.candidates,Candidate{b,current})} } - asset.asset_close(file) - return false + take(w,s,asset.REF_IMPORTS,store.hex(w.key,&digits),true) + if w.options.fast_textures {take(w,s,asset.REF_IMPORTS_FAST,store.hex(w.key,&digits),true)} + if w.stored {take(w,s,asset.REF_BUILDS,store.hex(asset.build_select(w.previous_manifest,&w.previous),&digits),false)} } -// The file of the imported folder that has the id and the extension of path -// under another key than that of the import, and its name without the key. -// The folder is read once, on the writer's thread. -@(private="file") -writer_previous :: proc(w:^Writer,path:string) -> (previous,unkeyed:string,found:bool) { - if !w.previous_read { - w.previous_read=true - w.previous=make(map[string]string,w.keep) - entries,_:=os.read_all_directory_by_path(w.imported,context.temp_allocator) - for entry in entries { - if _,key,named:=asset.imported_name(entry.name);named && key!=w.key && entry.type==.Regular { - w.previous[strings.concatenate({entry.name[:32],entry.name[49:]},w.keep)]=strings.clone(entry.fullpath,w.keep) - } +// The object of a build of the package (writer_candidates) for the output id, +// to take without building it: of kind in the version it is written in now, +// with owner, and made from one of expected, which records what decides its +// bytes beyond the key (the image of a texture and its encoders, the mesh as +// read and what builds it). Never with rebuild. Reads what it is given only: +// any thread, with types of its own. +@(private) +writer_reuse :: proc(w:^Writer,types:^asset.Types,id:base.Asset_Id,kind:asset.Fourcc,owner:base.Asset_Id,expected:..asset.Import_Info) -> (hash:store.Hash,info:asset.Import_Info,found:bool) { + if w.options.rebuild {return} + known,_:=asset.kind_info(kind) + for &c in w.candidates { + for o in c.build.outputs { + if o.id!=id || o.kind!=kind || o.kind_version!=known.version {continue} + header,recorded,imported,error:=asset.object_peek(w.options.store,o.object,types) + if error!=.None || !imported || header.kind!=kind || header.owner!=owner || !slice.contains(expected,recorded) {continue} + return o.object,recorded,true } } - name:=filepath.base(path) - if len(name)<49 {return} - unkeyed=strings.concatenate({name[:32],name[49:]},context.temp_allocator) - previous,found=w.previous[unkeyed] return } -// Takes the file an output of the cache had under another key as its file -// under the key of now (path), without building or writing it, when what it -// records it was built from is one of expected and it is of the version its -// kind is written in now: what is not in the key of the file but decides its -// bytes (the image of a texture and its encoders, the mesh as read and what -// builds it) is in what it records. False with rebuild or a file at path. +// The object a texture had, built from the same image with the same settings and +// encoders (Import_Info.settings_hash): Final, or Fast for an import of fast +// textures while the texture cache holds no Final of it. fast: it is Fast. @(private="file") -writer_reuse :: proc(w:^Writer,kind:asset.Fourcc,path:string,expected:..asset.Import_Info) -> bool { - if w.options.rebuild || os.exists(path) {return false} - previous,unkeyed,found:=writer_previous(w,path) - if !found {return false} - header,info,imported,error:=asset.import_peek(previous,&w.types) - known,_:=asset.kind_info(kind) - if error!=.None || !imported || header.kind!=kind || header.kind_version!=known.version || !slice.contains(expected,info) {return false} - if os.rename(previous,path)!=nil {return false} - delete_key(&w.previous,unkeyed) - return true -} - -// Takes the file a texture had under another key when it was built from the -// same image with the same settings and encoders (Import_Info.settings_hash): -// Final, or Fast for an import of fast textures while the texture cache holds -// no Final of it. -@(private="file") -texture_reuse :: proc(w:^Writer,job:Texture_Job,path,source:string) -> bool { +texture_reuse :: proc(w:^Writer,types:^asset.Types,job:Texture_Job,id,owner:base.Asset_Id,source:string) -> (hash:store.Hash,fast,found:bool) { options:=texture_options(w,job) final:=import_info(source,job.name,job.index,job.size,job.hash,textures.settings_hash(options),job.version) - fast:=final - if w.options.fast_textures && !textures.cache_has(w.options.texture_cache,textures.cache_key_build(job.bytes,job.made,options)) {fast.settings_hash=fast_settings(options)} - return writer_reuse(w,asset.KIND_TEXTURE,path,final,fast) + quick:=final + if w.options.fast_textures && !textures.cache_has(import_cache(w.options),textures.cache_key_build(job.bytes,job.made,options)) {quick.settings_hash=fast_settings(options)} + info:asset.Import_Info + hash,info,found=writer_reuse(w,types,id,asset.KIND_TEXTURE,owner,final,quick) + return hash,found && info.settings_hash!=final.settings_hash,found } // The slots of a mesh as its file stores them, naming the materials by row; in @@ -301,23 +306,6 @@ mesh_info :: proc(source:string,mesh:^Imported_Mesh,index:int,content:u64,slots: return import_info(source,mesh.name,index,size,hash,mesh_build_hash(settings),version) } -// Writes a material the source gives as desc. Over the file of the same -// material it follows what the source gives now and keeps what was changed -// after the last import, with the parameters, layers and hidden names it has. -@(private="file") -material_write :: proc(w:^Writer,path:string,existed:bool,desc:asset.Material_Desc) -> asset.Error { - desc:=desc - file:asset.Asset - if !existed || asset.asset_load(&file,path,context.temp_allocator)!=.None {return asset.material_save(path,&w.types,desc)} - defer asset.asset_close(&file) - if view,error:=asset.material_view(&file,&w.types);error==.None && file.blob.header.id==desc.id { - desc.info=asset.material_follow(&w.types,view,desc.imported^) - desc.parameters=asset.material_parameters_follow(view,desc.imported_parameters,context.temp_allocator) - desc.overlays,desc.hidden=view.overlays,view.hidden - } - return asset.material_save(path,&w.types,desc) -} - // What the source gives a material, its maps naming the textures of the // images by row. @(private) @@ -365,27 +353,31 @@ texture_options :: proc(w:^Writer,job:Texture_Job) -> textures.Build_Options { return image_build_options({srgb=job.srgb,normal_map=job.normal_map,green_down=job.green_down,alpha_cutoff=job.alpha_cutoff,alpha_fill=job.alpha_fill,keep_precision=job.keep_precision,channels=job.channels},w.options.settings) } -// Builds and writes one texture of source as the asset id at path; false when -// the image cannot be turned into one (a warning says why). +// Builds one texture of source as the asset id at path, or takes the object it +// had, and puts it in the store as output; false when the image cannot be +// turned into one (a warning says why) or the store does not take it (the +// writer failed). @(private) -write_texture :: proc(w:^Writer,job:Texture_Job,id,owner:base.Asset_Id,path,source:string) -> bool { - return texture_written(w,job,texture_write(w,job,id,owner,path,source,w.options.worker),id,path,source) +write_texture :: proc(w:^Writer,job:Texture_Job,id,owner:base.Asset_Id,path,source:string,output:int) -> bool { + writer_candidates(w) + return texture_written(w,job,texture_write(w,job,id,owner,path,source,w.options.worker),output) } // What building and saving a texture gave, for its writer to record. @(private) Texture_Write :: struct { - kept:bool, // its file already was the one the import gives - existed:bool, // a file was at its path before + kept:bool, // an object of a build was taken (hash) + hash:store.Hash, + fast:bool, // its blocks are Fast format:asset.Texture_Format, width,height,mips:int, built:textures.Error, saved:asset.Error, } -// Builds and saves a texture as write_texture does, or keeps its file, -// without changing w: on worker, so that the textures of a model are written -// on the job system at once. +// Builds and saves a texture as write_texture does, or finds the object to take +// (writer_reuse), without changing w: on worker, so that the textures of a +// model are written on the job system at once. @(private) texture_write :: proc(w:^Writer,job:Texture_Job,id,owner:base.Asset_Id,path,source:string,worker:jobs.Worker_Ref) -> (out:Texture_Write) { import_options:=w.options @@ -393,13 +385,15 @@ texture_write :: proc(w:^Writer,job:Texture_Job,id,owner:base.Asset_Id,path,sour types:asset.Types asset.types_init(&types,context.temp_allocator) defer asset.types_destroy(&types) - types.codecs=w.types.codecs options:=texture_options(w,job) info:=import_info(source,job.name,job.index,job.size,job.hash,textures.settings_hash(options),job.version) - if file:asset.Asset;writer_kept(w,path,owner,&file) { - defer asset.asset_close(&file) - if view,error:=asset.texture_view(&file,&types);error==.None { - return {kept=true,existed=true,format=view.info.format,width=int(view.info.width),height=int(view.info.height),mips=len(view.mips)} + if hash,fast,found:=texture_reuse(w,&types,job,id,owner,source);found { + file:asset.Asset + if asset.asset_open_object(&file,w.options.store,hash,context.temp_allocator)==.None { + defer asset.asset_close(&file) + if view,error:=asset.texture_view(&file,&types);error==.None { + return {kept=true,hash=hash,fast=fast,format=view.info.format,width=int(view.info.width),height=int(view.info.height),mips=len(view.mips)} + } } } cooked:=job.cooked @@ -412,20 +406,21 @@ texture_write :: proc(w:^Writer,job:Texture_Job,id,owner:base.Asset_Id,path,sour out.built=cooked.error if cooked.error!=.None {return} built:=&cooked.built + out.fast=cooked.settings!=info.settings_hash info.settings_hash=cooked.settings - desc:=textures.built_desc(built,id,&info,types.codecs==.Best) + desc:=textures.built_desc(built,id,&info) desc.owner=owner - out.existed=os.exists(path) out.format,out.width,out.height,out.mips=built.format,built.width,built.height,len(built.mips) zone(import_options,"texture.save") out.saved=asset.texture_save(path,&types,desc) return } -// Records what texture_write gave; false, with a warning, for an image that -// does not become a texture. +// Puts what texture_write gave in the store as output and records it; false, +// with a warning, for an image that does not become a texture, or with the +// writer failed. @(private) -texture_written :: proc(w:^Writer,job:Texture_Job,out:Texture_Write,id:base.Asset_Id,path,source:string) -> bool { +texture_written :: proc(w:^Writer,job:Texture_Job,out:Texture_Write,output:int) -> bool { switch { case out.built==.Too_Large: writer_warn(w,"image %s is not used: a side is above the %d texels a texture holds; scale it down in the source",job.name,asset.TEXTURE_MAX_SIZE) @@ -434,9 +429,11 @@ texture_written :: proc(w:^Writer,job:Texture_Job,out:Texture_Write,id:base.Asse writer_warn(w,"image %s is not used: %v",job.name,out.built) return false case out.saved!=.None: - return save_failed(w,out.saved,path) + return save_failed(w,out.saved,w.paths[output]) } - record(w,id,asset.KIND_TEXTURE,job.name,path,source,out.existed,texture_detail(out.format,out.width,out.height,out.mips),kept=out.kept) + if !(writer_take(w,output,out.hash) if out.kept else writer_put(w,output,w.paths[output])) {return false} + w.fast||=out.fast + record(w,output,job.name,texture_detail(out.format,out.width,out.height,out.mips),kept=out.kept) return true } @@ -444,6 +441,7 @@ texture_written :: proc(w:^Writer,job:Texture_Job,out:Texture_Write,id:base.Asse // them, each a job of the job system of the import: into out by row. @(private) textures_write :: proc(w:^Writer,texture_jobs:[]Texture_Job,parts:[]Part,source:string,out:[]Texture_Write) { + writer_candidates(w) Work :: struct {w:^Writer,jobs:[]Texture_Job,parts:[]Part,source:string,out:[]Texture_Write} work:=Work{w,texture_jobs,parts,source,out} run :: proc(work:^Work,first,last:int,worker:int) { @@ -470,54 +468,51 @@ texture_detail :: proc(format:asset.Texture_Format,width,height,mips:int) -> str return fmt.tprintf("%v %dx%d, %d mips",format,width,height,mips) } -// Starts the writes of the import of the source at path: reads its sidecar -// and, without options.registry, the assets of the root its documents are -// found among. folder: under the folder of the source, where the documents of -// a first import go ("" beside it). +// Starts the writes of the import of the source at path (writer_source). +// folder: under the folder of the documents, where the documents of a first +// import go ("" in it). @(private) writer_init :: proc(w:^Writer,path,folder:string,options:Import_Options,report:^Import_Report,allocator:mem.Allocator) -> bool { w^={options=options,report=report,allocator=allocator,version=import_version(0)} w.created.allocator=context.temp_allocator w.names.allocator=context.temp_allocator + w.followed.allocator=context.temp_allocator w.rng=base.rng_make(u64(time.time_to_unix_nano(time.now())),u64(uintptr(w))) asset.types_init(&w.types) - w.types.codecs=asset.project_codecs(options.root,project.engine_root_find("",context.temp_allocator)) zone(options,"import.scan") return writer_source(w,path,folder) && writer_changed(w) } -// Whether every output of the cache the sidecar lists has its file under the -// key of what the import builds from now, or one under another key that the -// writer may take (writer_reuse): then nothing is built ahead. +// Whether the build of what the import is made from now has an object in the +// store for every output of the cache the previous manifest lists: then +// nothing is built ahead. It fixes what the import is made from. @(private) writer_current :: proc(w:^Writer) -> bool { - if w.options.rebuild || len(w.sidecar.outputs)==0 {return false} - for output in w.sidecar.outputs { - info,known:=asset.kind_info(output.kind) - if !known || !asset.import_cached(output.kind) {continue} - path:=imported_file(w,output.id,info.extension) - if _,_,previous:=writer_previous(w,path);!previous && !os.exists(path) {return false} + if w.options.rebuild || !w.stored {return false} + writer_candidates(w) + for a in w.previous.assets { + if !asset.import_cached(a.kind) {continue} + held:=false + for &c in w.candidates { + if !c.current {continue} + for o in c.build.outputs {held||=o.id==a.id && asset.object_whole(w.options.store,o.object)} + } + if !held {return false} } return true } -@(private) -writer_rollback :: proc(w:^Writer) { - for path in w.created {os.remove(path)} - for _,kept in w.keeping {os.remove(kept)} -} - // Writes the meshes, materials, textures, skeletons and clips of a model as -// assets. Importing a source again writes each part again with the id its -// sidecar lists (identity.odin); a new document gets a random id and -// .mat, .skeleton or .anim in the folder of the import -// (model.folder beside the source the first time), and what nobody edits is a -// file of the imported folder of the root named by its id (sidecar.odin). -// The parts name the prefab of the model as their owner -// (Import_Report.prefab), which import_source writes. A mesh or a texture -// whose file already is the one it would write is kept as it is -// (Written.kept), unless options.rebuild. A failure removes the files this -// call created. +// the objects of a build of its package (package.odin). Importing a source again +// gives each part the asset the previous manifest lists (identity.odin); a new +// one gets a random id. A document is also written or followed in the folder +// of the documents (documents.odin): .mat, .skeleton or +// .anim there (under model.folder the first time). The parts name the +// prefab of the model as their owner (Import_Report.prefab), which +// import_source writes. A mesh or a texture a build of the package already +// has, made from what it would be made from now, is taken as it is +// (Written.kept), unless options.rebuild. A failure removes the documents this +// call created and records nothing. import_write :: proc(model:^Imported_Model,options:Import_Options={},allocator:=context.allocator) -> (report:Import_Report,error:Error) { caller:=context.temp_allocator scratch:mem.Dynamic_Arena @@ -528,11 +523,11 @@ import_write :: proc(model:^Imported_Model,options:Import_Options={},allocator:= w:Writer ok:=writer_init(&w,model.source,model.folder,options,&report,allocator) - defer asset.types_destroy(&w.types) + defer writer_close(&w) defer model_cooked_destroy(model) if ok { model_begin(&w,model) - ok=write_model(&w,model) && writer_finish(&w) + ok=write_model(&w,model) && writer_finish(&w,model.name if model.name!="" else filepath.stem(model.source),model.format) } if !ok { writer_rollback(&w) @@ -545,18 +540,20 @@ import_write :: proc(model:^Imported_Model,options:Import_Options={},allocator:= // Writes a model as parts of the import of o, with what import_write writes // one with: its meshes, materials, textures, skeletons and clips, each the -// output of the sidecar of its kind and name (then index), named as the model -// names them, a name with '/' in that folder under the one of the import. A -// part of the kind and name of one o already gave is that asset, which is not -// written again. When the model has nodes, its prefab is the scene of its -// name, the owner of its meshes and skeletons: the caller writes it -// (cook/prefab.model_prefab) and records it (output_written), unless -// prefab_path is "" (o gave it already, or the import keeps it as it is). The report gives, in allocator, the -// ids by row as that of import_write and the prefab; what is written, the -// warnings and the programs of the model go to the report of o. The files the -// model was read from are inputs of the import (output_input). Its assets name -// the source of o, with its size, hash and version, and are made with its -// settings. A failure leaves in o what was written, for outputs_end. +// asset of the previous manifest of its kind and name (then index), named as +// the model names them, a name with '/' in that folder under the one of the +// documents. A part of the kind and name of one o already gave is that asset, +// which is not written again. When the model has nodes, its prefab is the +// scene of its name, the owner of its meshes and skeletons: the caller writes +// it at prefab_path (cook/prefab.model_prefab) and records it +// (output_written), unless prefab_path is "" (o gave it already, or the import +// keeps it as it is). The report +// gives, in allocator, the ids by row as that of import_write and the prefab; +// what is written, the warnings and the programs of the model go to the +// report of o. The files the model was read from are inputs of the import +// (output_input). Its assets name the source of o, with its size, hash and +// version, and are made with its settings. A failure leaves in o what was +// written, for outputs_end. output_model :: proc(o:^Outputs,model:^Imported_Model,allocator:=context.allocator) -> (report:Import_Report,error:Error) { caller:=context.temp_allocator w:=&o.w @@ -604,17 +601,17 @@ output_model :: proc(o:^Outputs,model:^Imported_Model,allocator:=context.allocat return report,{} } -// The error of an input given once the key of the import was fixed. +// The error of an input given once what the import is made from was fixed. @(private) input_late :: proc(file:string,allocator:mem.Allocator) -> Error { - return error_make(.Malformed,"%s is an input given after the first output of the cache was planned, when what the import is built from was fixed: give every input (output_input) before",file,allocator=allocator) + return error_make(.Malformed,"%s is an input given after the first output of the cache was planned, when what the import is made from was fixed: give every input (output_input) before",file,allocator=allocator) } // What a model is written with: its settings, its warnings and programs, and // the files it was read from. @(private="file") model_begin :: proc(w:^Writer,model:^Imported_Model) { - // The settings kept with the source are those of every import of it. + // The settings its package keeps are those of every import of it. if model.stored && !w.options.ignore_stored {w.options.settings=model.settings} w.report.settings=w.options.settings for warning in model.warnings {writer_warn(w,"%s",warning)} @@ -634,17 +631,18 @@ write_model :: proc(w:^Writer,model:^Imported_Model) -> bool { name:=model.name if model.name!="" else filepath.stem(source) if chosen.skeleton!={} {return write_clips_onto(w,model,source,name,chosen.skeleton)} - // The prefab of the model: the one its sidecar lists, which a model that lost its nodes keeps. - // For Outputs, a scene of its name. + // The prefab of the model: the one its package lists, which a model that lost its nodes keeps. + // For Outputs, a scene of its name, written to the file output_written takes. listed:=false - for output in w.sidecar.outputs {listed||=output.kind==asset.KIND_SCENE && (!w.whole || output.name==name)} + for a in w.previous.assets {listed||=a.kind==asset.KIND_SCENE && (!w.whole || a.name==name)} w.owner={} if listed || len(model.nodes)>0 { prefab:=[1]Part{{kind=asset.KIND_SCENE,name=name,extension=".scene",only=!w.whole}} parts_plan(w,prefab[:]) w.owner=prefab[0].id _,keeping:=w.keeping[w.owner] - w.report.prefab,w.report.prefab_path=w.owner,strings.clone("" if keeping else prefab[0].path,w.allocator) + if keeping && !w.whole {writer_written(w,w.owner,asset.KIND_SCENE,0)} + w.report.prefab,w.report.prefab_path=w.owner,strings.clone(prefab[0].path if w.whole else prefab[0].target,w.allocator) w.report.prefab_info=import_info(source,name,0,model.source_size,model.source_hash,0,model.version) } @@ -679,43 +677,48 @@ write_model :: proc(w:^Writer,model:^Imported_Model) -> bool { parts_plan(w,parts[:]) reasons:=make([]string,len(model.materials),context.temp_allocator) parts_own(w,model,jobs[:],parts[:],first_material,reasons) - parts_record(w,parts[:]) - // What the writer keeps (its file under the key of now, or taken from - // another key) is not built; the rest is built ahead on a job system. - // Without one the writer builds each when it gets to it, holding one at a time. + // What a build of the package has, made from what it is made from now, + // is taken, not built; the rest is built ahead on a job system. Without + // one the writer builds each when it gets to it, holding one at a time. + writer_candidates(w) kept_images:=make([]bool,len(model.images),context.temp_allocator) - kept_meshes:=make([]bool,len(model.meshes),context.temp_allocator) + kept_meshes:=make([]store.Hash,len(model.meshes),context.temp_allocator) + given_meshes:=make([]bool,len(model.meshes),context.temp_allocator) if !w.options.rebuild { reuse_zone:=zone_begin(w.options,"import.reuse") defer base.profile_end(reuse_zone) for job,i in jobs { part:=parts[i] - kept_images[job.index]=part.given || os.exists(part.path) || texture_reuse(w,job,part.path,source) + _,_,found:=texture_reuse(w,&w.types,job,part.id,part.owner,source) + kept_images[job.index]=part.given || found } material_part_ids:=make([]base.Asset_Id,len(model.materials),context.temp_allocator) for i in 0..0 && mesh.skeleton>=0 && mesh.skeleton bool { if part.given {continue} info:=material_given(&material,image_ids) import_data:=import_info(source,material.name,i,model.source_size,model.source_hash,0,model.version) - existed:=os.exists(part.path) parameters:=material_given_parameters(&material,image_ids) material_zone:=zone_begin(w.options,"material.save") - if error:=material_write(w,part.path,existed,{id=part.id,owner=part.owner,info=info,imported=&info,parameters=parameters,imported_parameters=parameters,source=&import_data});error!=.None {return save_failed(w,error,part.path)} - record(w,part.id,asset.KIND_MATERIAL,material.name,part.path,source,existed,fmt.tprintf("%s: %s",part.owner!={}?fmt.tprintf("part of %s",name):"of its own",reasons[i]),follows=true) + if error:=asset.material_save(part.path,&w.types,{id=part.id,owner=part.owner,info=info,imported=&info,parameters=parameters,imported_parameters=parameters,source=&import_data});error!=.None {return save_failed(w,error,part.path)} + writer_put(w,part.output,part.path) or_return + record(w,part.output,material.name,fmt.tprintf("%s: %s",part.owner!={}?fmt.tprintf("part of %s",name):"of its own",reasons[i])) base.profile_end(material_zone) } @@ -738,36 +741,17 @@ write_model :: proc(w:^Writer,model:^Imported_Model) -> bool { for skeleton,i in model.skeletons { part:=&parts[first_skeleton+i] w.report.skeleton_ids[i]=part.id - if part.given { - body:asset.Skeleton_Body - body_parts:[len(anim.Body_Part)]asset.Skeleton_Body_Part - count,map_:=skeleton_body(nil,skeleton.bones,&body,&body_parts) - root_bones[i]=skeleton_root_bone(skeleton.bones,map_) if count>0 else -1 - continue - } - import_data:=import_info(source,skeleton.name,i,model.source_size,model.source_hash,0,model.version) - existed:=os.exists(part.path) - // The file it replaces: what a person set on it stays (body parts, sockets of their own). - previous_file:asset.Asset - previous:^asset.Skeleton_View - previous_view:asset.Skeleton_View - loaded:bool - if existed && asset.asset_load(&previous_file,part.path,context.temp_allocator)==.None { - loaded=true - view,error:=asset.skeleton_view(&previous_file,&w.types) - if error==.None {previous_view=view;previous=&previous_view} - if error==.Version {writer_warn(w,"%s is of a version this build does not read: the body parts, axes and sockets set by hand on it were not kept; set them again",part.path)} - } - defer if loaded {asset.asset_close(&previous_file)} body:asset.Skeleton_Body body_parts:[len(anim.Body_Part)]asset.Skeleton_Body_Part - count,map_:=skeleton_body(previous,skeleton.bones,&body,&body_parts) + count,map_:=skeleton_body(nil,skeleton.bones,&body,&body_parts) root_bones[i]=skeleton_root_bone(skeleton.bones,map_) if count>0 else -1 + if part.given {continue} + import_data:=import_info(source,skeleton.name,i,model.source_size,model.source_hash,0,model.version) sockets:=skeleton_sockets(model,skeleton) if count>0 {skeleton_grips(w,model,i,map_,&sockets)} - skeleton_sockets_kept(previous,skeleton.bones,&sockets) if error:=asset.skeleton_save(part.path,&w.types,{id=part.id,owner=part.owner,bones=skeleton.bones,sockets=sockets[:],body=&body if count>0 else nil,body_parts=body_parts[:count],source=&import_data});error!=.None {return save_failed(w,error,part.path)} - record(w,part.id,asset.KIND_SKELETON,skeleton.name,part.path,source,existed,fmt.tprintf("%d bones",len(skeleton.bones))) + writer_put(w,part.output,part.path) or_return + record(w,part.output,skeleton.name,fmt.tprintf("%d bones",len(skeleton.bones))) } w.report.mesh_ids=make([]base.Asset_Id,len(model.meshes),w.allocator) for &mesh,i in model.meshes { @@ -780,17 +764,22 @@ write_model :: proc(w:^Writer,model:^Imported_Model) -> bool { if mesh.skeleton<0 || mesh.skeleton>=len(w.report.skeleton_ids) {return writer_fail(w,.Malformed,"mesh %s has no skeleton",mesh.name)} skeleton=w.report.skeleton_ids[mesh.skeleton] } - file:asset.Asset - kept_zone:=zone_begin(w.options,"mesh.kept") - if writer_kept(w,path,owner,&file) { - view,view_error:=asset.mesh_view(&file,&w.types) - if view_error==.None { - record(w,id,asset.KIND_MESH,mesh.name,path,source,true,mesh_detail(view.levels,view.colliders,len(view.collision_points),len(view.collision_indices)),kept=true) + if kept:=kept_meshes[i];kept!={} { + kept_zone:=zone_begin(w.options,"mesh.kept") + file:asset.Asset + if asset.asset_open_object(&file,w.options.store,kept,context.temp_allocator)==.None { + view,view_error:=asset.mesh_view(&file,&w.types) + detail:=mesh_detail(view.levels,view.colliders,len(view.collision_points),len(view.collision_indices)) if view_error==.None else "" + asset.asset_close(&file) + if view_error==.None { + writer_take(w,part.output,kept) or_return + record(w,part.output,mesh.name,detail,kept=true) + base.profile_end(kept_zone) + continue + } } - asset.asset_close(&file) - if view_error==.None {base.profile_end(kept_zone);continue} + base.profile_end(kept_zone) } - base.profile_end(kept_zone) slots:=mesh_slots(&mesh,material_ids) cooked:=mesh.cooked own:Cooked_Mesh @@ -809,49 +798,40 @@ write_model :: proc(w:^Writer,model:^Imported_Model) -> bool { id=id,owner=owner,geometry=geometry,slots=slots,joints=mesh.joints,skeleton=skeleton, colliders=collision.colliders,collision_points=collision.points,collision_indices=collision.indices,source=&import_data, } - existed:=os.exists(path) save_zone:=zone_begin(w.options,"mesh.save") saved:=asset.mesh_save(path,&w.types,desc) base.profile_end(save_zone) - if saved==.None {record(w,id,asset.KIND_MESH,mesh.name,path,source,existed,mesh_detail(geometry.levels,collision.colliders,len(collision.points),len(collision.indices)))} + detail:=mesh_detail(geometry.levels,collision.colliders,len(collision.points),len(collision.indices)) if cooked==&own {cooked_mesh_destroy(&own)} if saved!=.None {return save_failed(w,saved,path)} + writer_put(w,part.output,path) or_return + record(w,part.output,mesh.name,detail) } if len(model.animations)>0 && parts[animation].given {w.report.animation_id=parts[animation].id} if len(model.animations)>0 && !parts[animation].given { part:=&parts[animation] import_data:=import_info(source,name,0,model.source_size,model.source_hash,0,model.version) - existed:=os.exists(part.path) clips:=make([]asset.Animation_Clip_Desc,len(model.animations),context.temp_allocator) copy(clips,model.animations) - // The file it replaces: the root a person set on a clip stays. - previous_file:asset.Asset - previous:asset.Animation_View - loaded:=existed && asset.asset_load(&previous_file,part.path,context.temp_allocator)==.None - defer if loaded {asset.asset_close(&previous_file)} - if loaded { - view_error:asset.Error - previous,view_error=asset.animation_view(&previous_file,&w.types) - if view_error==.Version {writer_warn(w,"%s is of a version this build does not read: the root bones set by hand on its clips were not kept; set them again",part.path)} - } for &clip in clips { index:int clip.skeleton,index=clip_skeleton(model,clip,w.report.skeleton_ids) - if index>=0 {clip.root=clip_root(model.skeletons[index].bones,root_bones[index],clip,previous)} + if index>=0 {clip.root=clip_root(model.skeletons[index].bones,root_bones[index],clip)} } desc:=asset.Animation_Desc{id=part.id,owner=part.owner,clips=clips,source=&import_data} if error:=asset.animation_save(part.path,&w.types,desc);error!=.None {return save_failed(w,error,part.path)} + writer_put(w,part.output,part.path) or_return w.report.animation_id=part.id - record(w,part.id,asset.KIND_ANIMATION,name,part.path,source,existed,fmt.tprintf("%d clips",len(model.animations))) + record(w,part.output,name,fmt.tprintf("%d clips",len(model.animations))) } return true } // Only the clips of a model, as an .anim of its own bound to a skeleton the project has // (Import_Settings.skeleton): the motion of another file onto a character imported before. Each clip -// keeps the root the model's own skeleton gives it, and the root a person set on the .anim it -// replaces. A clip that moves no bone at all (its markers alone, or curves) goes on that skeleton as -// well: only one whose bones the skeleton lacks is refused. +// keeps the root the model's own skeleton gives it. A clip that moves no bone at all (its markers +// alone, or curves) goes on that skeleton as well: only one whose bones the skeleton lacks is +// refused. @(private="file") write_clips_onto :: proc(w:^Writer,model:^Imported_Model,source,name:string,skeleton:base.Asset_Id) -> bool { if len(model.animations)==0 {return writer_fail(w,.Malformed,"%s has no clips to put on the skeleton of Import_Settings.skeleton",source)} @@ -861,14 +841,8 @@ write_clips_onto :: proc(w:^Writer,model:^Imported_Model,source,name:string,skel w.report.animation_id=part.id if part.given {return true} import_data:=import_info(source,name,0,model.source_size,model.source_hash,0,model.version) - existed:=os.exists(part.path) clips:=make([]asset.Animation_Clip_Desc,len(model.animations),context.temp_allocator) copy(clips,model.animations) - previous_file:asset.Asset - previous:asset.Animation_View - loaded:=existed && asset.asset_load(&previous_file,part.path,context.temp_allocator)==.None - defer if loaded {asset.asset_close(&previous_file)} - if loaded {previous,_=asset.animation_view(&previous_file,&w.types)} ids:=make([]base.Asset_Id,len(model.skeletons),context.temp_allocator) for &id in ids {id=skeleton} for &clip in clips { @@ -881,13 +855,14 @@ write_clips_onto :: proc(w:^Writer,model:^Imported_Model,source,name:string,skel body_parts:[len(anim.Body_Part)]asset.Skeleton_Body_Part bones:=model.skeletons[index].bones count,map_:=skeleton_body(nil,bones,&body,&body_parts) - clip.root=clip_root(bones,skeleton_root_bone(bones,map_) if count>0 else -1,clip,previous) + clip.root=clip_root(bones,skeleton_root_bone(bones,map_) if count>0 else -1,clip) } desc:=asset.Animation_Desc{id=part.id,owner=part.owner,clips=clips,source=&import_data} if error:=asset.animation_save(part.path,&w.types,desc);error!=.None {return save_failed(w,error,part.path)} + writer_put(w,part.output,part.path) or_return w.report.animation_id=part.id text:[base.ASSET_ID_TEXT]u8 - record(w,part.id,asset.KIND_ANIMATION,name,part.path,source,existed,fmt.tprintf("%d clips on skeleton %s",len(model.animations),base.asset_id_format(skeleton,text[:]))) + record(w,part.output,name,fmt.tprintf("%d clips on skeleton %s",len(model.animations),base.asset_id_format(skeleton,text[:]))) return true } @@ -929,16 +904,12 @@ skeleton_root_bone :: proc(bones:[]asset.Skeleton_Bone,map_:anim.Body_Map) -> in return parent if parent>=0 && parent (value:asset.Animation_Root) { - for &kept in previous.clips { - if base.text_get(kept.name[:])!=clip.name || !kept.root.translation && !kept.root.rotation {continue} - for &bone in bones {if bone.path==kept.root.bone {return kept.root}} - } +clip_root :: proc(bones:[]asset.Skeleton_Bone,root:int,clip:asset.Animation_Clip_Desc) -> (value:asset.Animation_Root) { if root<0 {return} for track in clip.tracks { if track.property!=.Position && track.property!=.Rotation || base.name_hash(track.target)!=bones[root].target || len(track.keys)<2 {continue} @@ -977,13 +948,13 @@ mesh_detail :: proc(levels:[]asset.Mesh_Level,colliders:[]asset.Collider,points, // Imports an image file as a texture of its own, read as options.image says // and keeping the coverage of options.alpha_coverage, unless the settings -// its sidecar keeps say otherwise. The texture is the root of the import: -// importing the file again writes the asset its sidecar lists. +// its package keeps say otherwise. The texture is the root of the import: +// importing the file again gives the asset its package lists. import_texture :: proc(path:string,options:Import_Options={},allocator:=context.allocator) -> (report:Import_Report,error:Error) { caller:=context.temp_allocator options:=options if !options.ignore_stored { - stored,found,settings_error:=settings_load(path) + stored,found,settings_error:=settings_load(options,path) if settings_error.code!=.None {return {},settings_error} if found {options.settings=stored} } @@ -996,7 +967,7 @@ import_texture :: proc(path:string,options:Import_Options={},allocator:=context. w:Writer ok:=writer_init(&w,path,"",options,&report,allocator) - defer asset.types_destroy(&w.types) + defer writer_close(&w) if ok { data,read_error:=read_whole(path) if read_error!=.None { @@ -1008,10 +979,10 @@ import_texture :: proc(path:string,options:Import_Options={},allocator:=context. name=filepath.stem(path),bytes=data,size=u64(len(data)),hash=base.hash64(data), srgb=use==.Color,normal_map=use==.Normal || use==.Normal_Green_Down,green_down=use==.Normal_Green_Down,alpha_cutoff=options.alpha_coverage,version=import_version(0), } - id,target:=root_take(&w,asset.KIND_TEXTURE,job.name,".tex") - built:=write_texture(&w,job,id,{},target,source) + id,target,output:=root_take(&w,asset.KIND_TEXTURE,job.name,".tex") + built:=write_texture(&w,job,id,{},target,source,output) if w.error.code!=.None {ok=false} else if !built {ok=writer_fail(&w,.Unsupported,"%s is not an image that can be decoded",path)} - ok=ok && writer_finish(&w) + ok=ok && writer_finish(&w,job.name,"image") } } if !ok { @@ -1071,37 +1042,3 @@ skeleton_grips :: proc(w:^Writer,model:^Imported_Model,skeleton:int,map_:anim.Bo if found {append(sockets,socket)} } } - -// The sockets of the file an import replaces that it does not make again (one a person or an agent -// added, such as where a weapon hangs), on a bone it still has. Those it makes (named by the -// source, bone.*, grips) are made again from the source. -@(private="file") -skeleton_sockets_kept :: proc(previous:^asset.Skeleton_View,bones:[]asset.Skeleton_Bone,sockets:^[dynamic]asset.Skeleton_Socket) { - if previous==nil {return} - made:=len(sockets) - for &socket in previous.sockets { - name:=base.text_get(socket.name[:]) - if strings.has_prefix(name,"bone.") {continue} - again:=false - for &other in sockets[:made] {if other.name==socket.name {again=true;break}} - if again {continue} - for &bone in bones {if bone.path==socket.bone {append(sockets,socket);break}} - } -} - -// The body of an imported skeleton: found on its bones, keeping the parts a -// person set in the file it replaces when their bones are still there. -// Returns how many parts it wrote, 0 for a skeleton without a body, and the map they come from. -@(private="file") -skeleton_body :: proc(previous_file:^asset.Skeleton_View,bones:[]asset.Skeleton_Bone,body:^asset.Skeleton_Body,parts:^[len(anim.Body_Part)]asset.Skeleton_Body_Part) -> (count:int,map_:anim.Body_Map) { - previous:=anim.body_map_empty() - if previous_file!=nil { - previous=asset.skeleton_body_map(bones,previous_file.body,previous_file.body_parts) - for part in anim.Body_Part {if part not_in previous.manual {previous.bones[part],previous.confidence[part]=-1,0}} - } - found:bool - map_,found=asset.skeleton_body_detect(bones,previous) - if !found {return 0,map_} - count,_=asset.skeleton_body_rows(bones,map_,body,parts) - return count,map_ -} diff --git a/cook/prefab/prefab.odin b/cook/prefab/prefab.odin index a2b2239a4..bc99d40ab 100644 --- a/cook/prefab/prefab.odin +++ b/cook/prefab/prefab.odin @@ -154,7 +154,7 @@ model_prefab :: proc(model:^importer.Imported_Model,assets:^importer.Import_Repo bytes,problem:=author.scene_file_bytes(&scene,types,info,&scratch) if problem!="" {return report,.Write} os.make_directory_all(filepath.dir(path)) - if serial.file_write_atomic(path,bytes)!=.None {return report,.Save} + if _,write_error:=serial.file_write_changed(path,bytes);write_error!=.None {return report,.Save} report.entities=asset.scene_entity_count(&scene) return report,.None } @@ -282,12 +282,10 @@ model_import :: proc(path:string,formats:[]importer.Format,options:importer.Impo // prefab of its hierarchy; only its clips with Import_Settings.skeleton. model_write :: proc(model:^importer.Imported_Model,path:string,options:importer.Import_Options,allocator:=context.allocator) -> (report:importer.Import_Report,error:importer.Error) { if report,error=importer.import_write(model,options,allocator);error.code!=.None {return} - if len(model.nodes)==0 || report.settings.skeleton!={} {return report,{}} + if len(model.nodes)==0 || report.settings.skeleton!={} || report.prefab_path=="" {return report,{}} name:=filepath.stem(path) - source:=importer.source_path(options,path) + source:=importer.source_path(path) id,target:=report.prefab,report.prefab_path - // A prefab changed since the import wrote it, which the import keeps (Changed.Keep). - if target=="" {return report,{}} if options.world==nil { importer.report_destroy(&report) return {},importer.error_make(.Write,"the prefab %s is written in the types of the game, which the library of the project's editor has: %s",target,author.BUILD_FIX) @@ -303,10 +301,7 @@ model_write :: proc(model:^importer.Imported_Model,path:string,options:importer. importer.report_destroy(&report) return {},importer.error_make(.Write,"the prefab %s could not be written: %v",target,prefab_error) } - if !importer.output_recorded(path,id,target) { - importer.report_destroy(&report) - return {},importer.error_make(.Write,"%s%s, which records what the import wrote, could not be written",path,importer.SIDECAR_EXTENSION) - } + append(&report.written,asset.Package_Written{id,importer.written_hash(target)}) size:int if info,stat_error:=os.stat(target,context.temp_allocator);stat_error==nil {size=int(info.size)} append(&report.assets,importer.Written{ diff --git a/cook/share/command.odin b/cook/share/command.odin new file mode 100644 index 000000000..a23919c91 --- /dev/null +++ b/cook/share/command.odin @@ -0,0 +1,415 @@ +package share + +import "base:runtime" +import "core:fmt" +import "core:io" +import "core:os" +import "core:strconv" +import "core:strings" +import "core:sync" +import "core:thread" +import "core:time" +import "mirai:asset" +import "mirai:cook/import/sources" +import "mirai:project" +import "mirai:std/base" +import "mirai:std/jobs" +import "mirai:store" +import wire "mirai:net" + +// The commands mirai share and mirai join, which the import program of the +// engine runs. share takes a snapshot of a project, opens a host with its port +// mapped by the router, prints one invitation and serves until it is stopped: +// the snapshot, its files and the library of the machine store, to whoever +// holds the invitation. join connects with an invitation, writes the project +// into a folder and refreshes its packages from the peers; while it runs it +// serves what it has to others, with an invitation of its own. --address +// joins the host at that address, with the key of the invitation, in place +// of the addresses it carries (the same machine at 127.0.0.1, or an address +// known to reach it). The store is the machine's, or the one at --store. +// Exit codes: 0 done, 1 failed, 2 misuse, 3 incomplete (a join wrote all but +// the files it lists, and run again once they are fixed completes them). + +SHARE_USAGE :: "Usage: mirai share [project] [--port N] [--store ]" +JOIN_USAGE :: "Usage: mirai join [folder] [--address ] [--port N] [--store ]" +PORT :: 7950 // a share's, unless --port; taken, the system gives another +@(private) STATUS_SECONDS :: 10.0 +@(private) MAPPING_SECONDS :: 15.0 // the most an invitation waits for the router + +// share or join, by the first argument. library reads the formats the plugins +// of the project add (editor.import_library in the import program). +command :: proc(args:[]string,output,errors:io.Writer,library:sources.Library=nil) -> int { + if len(args)>0 && args[0]=="share" {return share_command(args[1:],output,errors)} + if len(args)>0 && args[0]=="join" {return join_command(args[1:],output,errors,library)} + fmt.wprintfln(errors,"%s\n%s",SHARE_USAGE,JOIN_USAGE) + return 2 +} + +// A host on UDP with its port mapping. +@(private) +Udp_Host :: struct { + udp:wire.Udp, + host:wire.Host, + mapping:wire.Port_Mapping, + mapping_on,open:bool, + key:wire.Invite_Key, // of the invitation: of this share for a server, what the code carries for a client + port:int, +} + +@(private) +host_open :: proc(h:^Udp_Host,mode:wire.Mode,port:int,key:wire.Invite_Key) -> bool { + // The addresses of an invitation are IPv4. + if mode==.Client { + if !wire.udp_open(&h.udp,wire.address_ip4(0,0,0,0,0)) {return false} + } else if !wire.udp_open(&h.udp,wire.address_any6(port)) && !wire.udp_open(&h.udp,wire.address_any6(0)) { + return false + } + chans:=channels() + desc:=wire.Host_Desc{mode=mode,transport=wire.udp_transport(&h.udp),protocol_id=PROTOCOL_ID,version=VERSION,invite=key,max_connections=PEERS-1,channels=chans[:]} + if wire.host_init(&h.host,desc)!=.None || !wire.worker_start(&h.host) { + wire.host_destroy(&h.host) + wire.udp_close(&h.udp) + return false + } + h.open,h.key,h.port=true,key,int(h.udp.address.port) + if mode==.Server {h.mapping_on=wire.mapping_start(&h.mapping,u16(h.port))} + return true +} + +@(private) +host_close :: proc(h:^Udp_Host) { + if !h.open {return} + // The others hear the end instead of waiting for silence. + wire.host_disconnect(&h.host,0,.Closed) + wire.host_poll(&h.host) + time.sleep(5*time.Millisecond) + if h.mapping_on {wire.mapping_close(&h.mapping)} + wire.host_destroy(&h.host) + wire.udp_close(&h.udp) + h.open=false +} + +// The invitation of a server once its router answered, or MAPPING_SECONDS +// passed; mapped: whether it carries an address outside this network. +@(private) +invitation_of :: proc(h:^Udp_Host,wait:bool) -> (code:string,mapped:bool,ready:bool) { + if h.mapping_on && wait { + started:=time.tick_now() + for wire.mapping_poll(&h.mapping).phase==.Pending && time.duration_seconds(time.tick_since(started)) (s:store.Store,ok:bool) { + folder:=root!=""?root:store.default_root(context.temp_allocator) + if folder=="" { + fmt.wprintfln(errors,"mirai %s: this machine has no folder of user data for its store: pass one with --store",what) + return + } + error:store.Error + s,error=store.open(folder) + #partial switch error { + case .None: return s,true + case .Unsupported_Layout: + fmt.wprintfln(errors,"mirai %s: the store at %s was written by another version of Mirai: remove that folder, whose content is fetched or built again, or pass another with --store",what,folder) + case: + fmt.wprintfln(errors,"mirai %s: the store at %s could not be opened (%v): pass another with --store",what,folder,error) + } + return +} + +@(private) +options_parse :: proc(args:[]string,errors:io.Writer,usage,what:string,positional:^[dynamic]string,address:^wire.Address=nil) -> (port:int,store_root:string,ok:bool) { + port=-1 + for at:=0;at=len(args) {fmt.wprintln(errors,usage);return} + at+=1 + parsed,valid:=wire.address_parse(args[at]) + if !valid || parsed.kind!=.IP4 || parsed.port==0 {fmt.wprintfln(errors,"mirai %s: --address needs an IPv4 address with its port, as 127.0.0.1:7950",what);return} + address^=parsed + case "--store": + if at+1>=len(args) {fmt.wprintln(errors,usage);return} + at+=1 + store_root=args[at] + case "--port": + if at+1>=len(args) {fmt.wprintln(errors,usage);return} + at+=1 + value,parsed:=strconv.parse_int(args[at]) + if !parsed || value<0 || value>65535 {fmt.wprintfln(errors,"mirai %s: --port needs a UDP port from 0 (any) to 65535",what);return} + port=value + case: + if strings.has_prefix(args[at],"-") {fmt.wprintfln(errors,"mirai %s: '%s' is not an option\n%s",what,args[at],usage);return} + append(positional,args[at]) + } + } + return port,store_root,true +} + +@(private) +share_command :: proc(args:[]string,output,errors:io.Writer) -> int { + positional:=make([dynamic]string,context.temp_allocator) + port,store_root,ok:=options_parse(args,errors,SHARE_USAGE,"share",&positional) + if !ok || len(positional)>1 { + if ok {fmt.wprintln(errors,SHARE_USAGE)} + return 2 + } + root:=project.path_absolute(len(positional)>0?positional[0]:".",context.temp_allocator) + s,opened:=store_open(store_root,errors,"share") + if !opened {return 1} + defer store.close(&s) + + // The exchange's jobs, owned by this thread, which pumps. + exchange_jobs:jobs.Job_System + if !jobs.system_init(&exchange_jobs,{threads=jobs.THREADS_AUTO,allocator=runtime.heap_allocator()}) { + fmt.wprintln(errors,"mirai share: the threads of the exchange could not be started") + return 1 + } + defer jobs.system_destroy(&exchange_jobs) + n:=new(Node,context.temp_allocator) + if node_init(n,&s,runtime.heap_allocator(),{system=&exchange_jobs,worker=jobs.OWNER})!=.None { + fmt.wprintln(errors,"mirai share: no memory for the exchange") + return 1 + } + defer node_destroy(n) + key,made:=wire.invite_key() + if !made {fmt.wprintln(errors,"mirai share: this system gives no secure random bytes to seal the connections with");return 1} + + js:jobs.Job_System + if !jobs.system_init(&js,{threads=jobs.THREADS_AUTO,allocator=runtime.heap_allocator()}) { + fmt.wprintln(errors,"mirai share: the threads to read the files on could not be started") + return 1 + } + started:=time.tick_now() + o,offer_error:=offer(n,root,key.secret,{system=&js,worker=jobs.OWNER},context.temp_allocator) + jobs.system_destroy(&js) + for line in o.taken.skipped {fmt.wprintfln(errors,"left out: %s",line)} + if offer_error!="" { + fmt.wprintfln(errors,"mirai share: %s",offer_error) + return 1 + } + fmt.wprintfln(output,"snapshot of %s: %d files, %s (%d read into the store at %s, %s)",root,o.taken.files,bytes_text(o.taken.bytes),o.taken.put,s.root,seconds_text(time.duration_seconds(time.tick_since(started)))) + for problem in o.problems {fmt.wprintfln(errors,"warning: a joiner will not get it: %s",problem)} + h:=new(Udp_Host,context.temp_allocator) + if !host_open(h,.Server,port>=0?port:PORT,key) { + fmt.wprintfln(errors,"mirai share: no UDP port could be opened%s: give another with --port",port>0?fmt.tprintf(" at %d",port):"") + return 1 + } + defer host_close(h) + node_host(n,&h.host) + name:=o.name + + code,mapped,_:=invitation_of(h,true) + fmt.wprintfln(output,"Sharing %s on UDP port %d. Invitation:\n\n %s\n\nOn the other machine: mirai join %s [folder]",name,h.port,code,code) + if !mapped {fmt.wprintfln(output,"The router mapped no port, so only machines on this network can join; from another network, the joiner's router must map one, or this one must allow UPnP, NAT-PMP or PCP.")} + fmt.wprintln(output,"Stop with Ctrl+C.") + events:[EVENTS]Event + status_at:=0.0 + last:Node_Totals + for { + now:=wire.host_time(&h.host) + node_pump(n,now) + for e in node_events(n,events[:]) {event_print(output,e,node_totals(n).connected)} + if now-status_at>=STATUS_SECONDS { + status_at=now + t:=node_totals(n) + if t.sent!=last.sent || t.connected!=last.connected { + fmt.wprintfln(output,"%d connected, %s sent, %s/s",t.connected,bytes_text(i64(t.sent)),bytes_text(i64(t.send_rate))) + } + last=t + } + time.sleep(time.Millisecond) + } +} + +// What a host offers: the snapshot of its project, its address and name, +// what taking it read, and the packages a joiner will not get, each why. +Offer :: struct { + snapshot:Snapshot, + address:store.Hash, + name:string, + taken:Take_Report, + problems:[]string, +} + +// Offers the project at root to whoever holds the invitation of secret: its +// snapshot put in the store of the node under offer_name of the secret, with +// its folders, files and what the packages it pins need held by the lease of +// the node. Taking it again after a change offers the new snapshot, and a +// joined peer that joins again gets only what differs. Jobs of +// worker, when given, read the files. In allocator. +offer :: proc(n:^Node,root:string,secret:u64,worker:=jobs.Worker_Ref{},allocator:=context.allocator) -> (o:Offer,error:string) { + manifest,manifest_error:=project.manifest_read(project.path(root,project.PROJECT_FILE,allocator=context.temp_allocator)) + if manifest_error.code!=.None { + return {},fmt.aprintf("%s is not a project (its %s does not read: %v): give the folder of one",root,project.PROJECT_FILE,manifest_error.code,allocator=allocator) + } + o.name=base.text_get(manifest.name[:]) + if !name_valid(o.name) {o.name=os.base(root)} + if !name_valid(o.name) {o.name="project"} + o.name=strings.clone(o.name,allocator) + served:=make([dynamic]store.Hash,context.temp_allocator) + o.snapshot,o.address,o.taken,error=snapshot_take(n.s,root,o.name,&served,worker,allocator) + if error!="" {return} + packages,_:=asset.root_packages(root,n.s.root,allocator) + o.problems=packages.problems + for e in o.snapshot.entries {append(&served,e.address)} + library_served(n.s,root,&served) + // An offer taken again replaces what the lease held for the one before. + sync.mutex_lock(&n.lock) + clear(&n.served) + sync.mutex_unlock(&n.lock) + node_serve(n,..served[:]) + digits:[64]u8 + offer_set(n.s,root,offer_name(secret,&digits),o.address) + return +} + +// What the packages the project at root pins need: their manifests and the +// builds this machine chose for them, with every blob they list. +@(private) +library_served :: proc(s:^store.Store,root:string,served:^[dynamic]store.Hash) { + list,_,error:=asset.packages_load(root,context.temp_allocator) + if error!=.None {return} + digits:[64]u8 + for &entry in list.packages { + append(served,entry.manifest) + p,package_error:=asset.package_load(s,root,&entry,context.temp_allocator) + if package_error!=.None {continue} + hash,ref_error:=store.ref_get(s,asset.REF_BUILDS,store.hex(asset.build_select(entry.manifest,&p),&digits)) + if ref_error!=.None {continue} + append(served,hash) + b,build_error:=asset.build_get(s,hash,context.temp_allocator) + if build_error!=.None {continue} + for o in b.outputs { + append(served,o.object) + if o.thumbnail!={} {append(served,o.thumbnail)} + } + } +} + +@(private) +event_print :: proc(output:io.Writer,e:Event,connected:int) { + switch e.kind { + case .Joined: fmt.wprintfln(output,"peer %d joined (%d connected)",e.peer,connected) + case .Left: fmt.wprintfln(output,"peer %d left (%v): %s sent to it, %s received (%d connected)",e.peer,e.reason,bytes_text(i64(e.sent)),bytes_text(i64(e.received)),connected) + } +} + +// The network of a join: its node pumped on a thread of its own until stopped. +@(private) +Pump :: struct { + node:^Node, + hosts:[2]^Udp_Host, + output:io.Writer, + stop:bool, +} + +@(private) +join_command :: proc(args:[]string,output,errors:io.Writer,library:sources.Library) -> int { + positional:=make([dynamic]string,context.temp_allocator) + at:wire.Address + port,store_root,ok:=options_parse(args,errors,JOIN_USAGE,"join",&positional,&at) + if !ok || len(positional)<1 || len(positional)>2 { + if ok {fmt.wprintln(errors,JOIN_USAGE)} + return 2 + } + invitation,refused:=wire.invite_decode(positional[0]) + if refused!=.None { + fmt.wprintfln(errors,"mirai join: '%s' is not an invitation of this Mirai: %s",positional[0],wire.invite_error_text(refused)) + return 1 + } + folder:=len(positional)>1?positional[1]:"" + s,opened:=store_open(store_root,errors,"join") + if !opened {return 1} + defer store.close(&s) + js:jobs.Job_System + if !jobs.system_init(&js,{threads=jobs.THREADS_AUTO,allocator=runtime.heap_allocator()}) { + fmt.wprintln(errors,"mirai join: the threads to import on could not be started") + return 1 + } + defer jobs.system_destroy(&js) + + // The exchange's jobs, owned by the thread that pumps; the join's own are for its imports. + exchange_jobs:jobs.Job_System + if !jobs.system_init(&exchange_jobs,{threads=jobs.THREADS_AUTO,allocator=runtime.heap_allocator()}) { + fmt.wprintln(errors,"mirai join: the threads of the exchange could not be started") + return 1 + } + defer jobs.system_destroy(&exchange_jobs) + n:=new(Node,context.temp_allocator) + if node_init(n,&s,runtime.heap_allocator(),{system=&exchange_jobs,worker=jobs.OWNER})!=.None { + fmt.wprintln(errors,"mirai join: no memory for the exchange") + return 1 + } + client,server:=new(Udp_Host,context.temp_allocator),new(Udp_Host,context.temp_allocator) + own,made:=wire.invite_key() + if !made {fmt.wprintln(errors,"mirai join: this system gives no secure random bytes to seal the connections with");return 1} + if !host_open(client,.Client,0,invitation.key) { + fmt.wprintln(errors,"mirai join: no UDP port could be opened: close what holds them, or let this program use the network") + node_destroy(n) + return 1 + } + node_host(n,&client.host) + if host_open(server,.Server,port>=0?port:PORT,own) {node_host(n,&server.host)} + if at.kind!=.None { + wire.host_connect(&client.host,at) + } else { + wire.host_connect(&client.host,invitation.lan,invitation.public) + } + + pump:=Pump{node=n,hosts={client,server},output=output} + t:=thread.create_and_start_with_data(&pump,proc(data:rawptr) { + p:=(^Pump)(data) + code_said:=!p.hosts[1].open + for !sync.atomic_load(&p.stop) { + node_pump(p.node,wire.host_time(&p.hosts[0].host)) + if !code_said { + if code,_,ready:=invitation_of(p.hosts[1],false);ready { + code_said=true + if code!="" {fmt.wprintfln(p.output,"while this runs, others can join from here too: mirai join %s",code)} + } + } + time.sleep(time.Millisecond) + } + }) + if t==nil {fmt.wprintln(errors,"mirai join: no thread for the network");return 1} + digits,own_digits:[64]u8 + desc:=Join_Desc{ + node=n,offer=offer_name(invitation.key.secret,&digits),own_offer=server.open?offer_name(own.secret,&own_digits):"", + folder=folder,library=library,worker={system=&js,worker=jobs.OWNER},output=output, + } + report,error:=join(desc,context.temp_allocator) + t_received:=node_totals(n).received + sync.atomic_store(&pump.stop,true) + thread.join(t) + thread.destroy(t) + host_close(client) + host_close(server) + node_destroy(n) + if report.folder!="" && desc.own_offer!="" {store.ref_remove(&s,NAMESPACE,desc.own_offer)} + + for line in report.kept {fmt.wprintfln(output,"kept: %s",line)} + for line in report.failed {fmt.wprintfln(errors,"not written: %s",line)} + if error!="" { + fmt.wprintfln(errors,"mirai join: %s",error) + return 1 + } + code:=0 + for warning in report.packages.warnings {fmt.wprintfln(output,"warning: %s",warning)} + for failure in report.packages.failures {fmt.wprintfln(errors,"mirai join: %s",failure);code=1} + for missing in report.packages.missing {fmt.wprintfln(errors,"mirai join: %s",missing);code=1} + fmt.wprintfln(output,"Joined into %s: %d files, %d written, %d removed, %d kept as you left them; %d packages, %d builds fetched; %s received.", + report.folder,report.files,report.written,report.removed,len(report.kept),report.packages.packages,report.packages.fetched,bytes_text(i64(t_received))) + if len(report.failed)>0 { + fmt.wprintfln(errors,"mirai join: incomplete, %d files were not written or removed; fix what each says and run the same join again, which does only those",len(report.failed)) + return 3 + } + open:=fmt.tprintf("mirai edit %s",report.folder) + if store_root!="" {open=fmt.tprintf("%s --store %s",open,store_root)} + fmt.wprintfln(output,"Open it with: %s",open) + return code +} diff --git a/cook/share/join.odin b/cook/share/join.odin new file mode 100644 index 000000000..45398b6b9 --- /dev/null +++ b/cook/share/join.odin @@ -0,0 +1,1027 @@ +package share + +import "base:runtime" +import "core:fmt" +import "core:io" +import "core:os" +import "core:slice" +import "core:strings" +import "core:sync" +import "core:thread" +import "mirai:asset" +import "mirai:cook/import/sources" +import "mirai:exchange" +import "mirai:project" +import "mirai:std/base" +import "mirai:std/jobs" +import "mirai:store" +import wire "mirai:net" + +// A join: the snapshot a host offers, its files written into a folder, and +// the packages of the project refreshed with the peers as the source of the +// library. The tree is fetched a depth at a time, with the base put in the +// store first, so only the folders that changed are asked. The files that +// must be written are fetched small first, many at once, the project's +// manifest and start scene before the rest; each is written chunk by chunk +// into a temporary file and renamed into place as it completes. The store +// keeps what arrived, so a join stopped and run again fetches only the rest, +// and a file changed here before is put in the store first, so the chunks +// its old version shares with the new are not asked. +// +// A folder joined before keeps the snapshot it last took +// (/.mirai/share/snapshot), the base of three ways: a file the host +// changed or removed is replaced or removed when the joiner left it as it +// was; a file both changed is kept and reported, never overwritten, also +// when the joiner changes it while the join runs. A file that cannot be +// written or removed is reported with why and how to fix it, the rest is +// done, and the folder keeps as its base the last entry of each such file, so +// running the join again completes it. + +Join_Desc :: struct { + node:^Node, + offer:string, // the name of the snapshot under NAMESPACE the host offers (offer_name of the secret of its invitation) + own_offer:string, // the name this node offers it under to others; "": none + folder:string, // where the project goes; "": a new folder by its name under parent + parent:string, // "": the current folder + library:sources.Library, + worker:jobs.Worker_Ref, + output:io.Writer, // progress and what was kept; nil: nowhere + pace:f64, // seconds between two progress lines, default 1 + connect_seconds:f64, // waiting for the host, default 15 + progress:^Join_Progress, // what it is doing, for whoever shows it; nil: none +} + +Join_Phase :: enum u8 {Connecting,Snapshot,Files,Packages,Done} + +// What a join is doing while it runs. files and bytes: those it fetches; +// rate: chunk bytes received per second lately; left: seconds the files +// still need at that rate, <0 when unknown; kept: files left as the joiner +// changed them so far. +Join_State :: struct { + phase:Join_Phase, + files,files_done:int, + bytes,bytes_done:i64, + received:u64, + rate,left:f64, + kept:int, +} + +// The state of a join, written by its thread and read by any other. +Join_Progress :: struct { + lock:sync.Mutex, + state:Join_State, +} + +join_progress_read :: proc(p:^Join_Progress) -> Join_State { + sync.mutex_guard(&p.lock) + return p.state +} + +@(private) +progress_set :: proc(d:^Join_Desc,state:Join_State) { + if d.progress==nil {return} + sync.mutex_guard(&d.progress.lock) + d.progress.state=state +} + +@(private) +progress_phase :: proc(d:^Join_Desc,phase:Join_Phase,report:^Join_Report=nil) { + if d.progress==nil {return} + sync.mutex_guard(&d.progress.lock) + d.progress.state.phase=phase + d.progress.state.left=-1 + if report!=nil {d.progress.state.kept=len(report.kept)} +} + +Join_Report :: struct { + folder:string, + files:int, // of the snapshot + written:int, // files written now + removed:int, // files the host removed, removed here + kept:[dynamic]string, // files left as they are on purpose, each with why + failed:[dynamic]string, // files not written or removed, each with why and how to fix it: the join is incomplete + fetched:i64, // bytes of files received now + packages:sources.Library_Report, +} + +// What the files of the snapshot need here. +@(private) +Plan :: struct { + write:[dynamic]Write, + remove:[dynamic]Remove, + folds:bool, // the folder's file system folds case: paths compare in lower case + last_folded:map[string]int, // with folds: the entries of the last snapshot by their path in lower case + snap_folded:map[string]int, // with folds: those of the snapshot +} + +// An entry of the snapshot to write, with the file as the plan found it here: +// absent, or with the address base. done: written, or left on purpose. +// refused: its path passes through a link here, and it is not written. +@(private) +Write :: struct {at:int,present:bool,base:store.Hash,done:bool,refused:bool} + +// An entry of the last snapshot the host removed and the joiner did not change. +@(private) +Remove :: struct {at:int,done:bool,refused:bool} + +@(private) MISSING_SECONDS :: 10.0 // an object no connected peer has for this long fails the join +@(private) STALL_SECONDS :: 20.0 // without a byte received or an object complete, what is wanted fails +@(private) WINDOW :: 256 // files wanted at once + +join :: proc(d:Join_Desc,allocator:=context.allocator) -> (report:Join_Report,error:string) { + n:=d.node + d:=d + if d.pace<=0 {d.pace=1} + if d.connect_seconds<=0 {d.connect_seconds=15} + report.kept=make([dynamic]string,allocator) + report.failed=make([dynamic]string,allocator) + progress_phase(&d,.Connecting) + defer progress_phase(&d,.Done,&report) + if error=join_connect(&d,allocator);error!="" {return} + progress_phase(&d,.Snapshot) + + // The root first: it names the project, whose folder keeps the base. + say(&d,"asking for the snapshot") + address:store.Hash + if address,error=join_offer(&d,allocator);error!="" {return} + roots:=[1]store.Hash{address} + if error=join_fetch(&d,roots[:],"the snapshot",allocator);error!="" {return} + name,name_error:=snapshot_name(n.s,address,allocator) + if name_error!=.None { + error=refused(name_error,allocator) + return + } + folder:=d.folder + if folder=="" {folder=project.path(d.parent==""?".":d.parent,name,allocator=context.temp_allocator)} + folder=project.path_absolute(folder,allocator) + report.folder=folder + if os.exists(folder) && !os.is_dir(folder) { + error=fmt.aprintf("%s is a file: give the join a folder",folder,allocator=allocator) + return + } + // Its state and temporary files are not written through a link either. + names:=project.FOLDER_NAMES + for part in ([2]string{"tmp",SHARE_FOLDER}) { + if link:=link_on(folder,fmt.tprintf("%s/%s",names[.Generated],part));link!="" { + error=fmt.aprintf("%s is a link, and a join never writes through a link inside its folder: replace it with a folder of its own and run the join again",link,allocator=allocator) + return + } + } + node_serve(n,address) + if d.own_offer!="" {offer_set(n.s,folder,d.own_offer,address)} + // What a join stopped in the middle of a write left. + temp_folder:=project.path(project.folder_path(folder,.Generated,context.temp_allocator),"tmp",allocator=context.temp_allocator) + if left,list_error:=os.read_all_directory_by_path(temp_folder,context.temp_allocator);list_error==nil { + for info in left { + if strings.has_prefix(info.name,"share-") {os.remove(info.fullpath)} + } + } + + // The base as a tree in the store, so what the host did not change since is + // not asked; then the snapshot, a depth of folders at a time. + index:=index_load(folder,context.temp_allocator) + last:Snapshot + if previous,read_error:=os.read_entire_file(share_path(folder,"snapshot",context.temp_allocator),context.temp_allocator);read_error==nil { + last,_=snapshot_decode(previous,context.temp_allocator) + } + if len(last.entries)>0 {snapshot_put(n.s,&last)} + fetching:=Tree_Fetching{d=&d,allocator=allocator} + folders:=make([dynamic]store.Hash,context.temp_allocator) + snap,tree_error:=snapshot_read(n.s,address,{procedure=join_tree_fetch,data=&fetching},&folders,allocator) + if tree_error==.Unavailable && fetching.error!="" { + error=fetching.error + return + } + if tree_error!=.None { + error=refused(tree_error,allocator) + return + } + report.files=len(snap.entries) + node_serve(n,..folders[:]) + plan:=join_plan(&snap,&last,&index,folder,&report,allocator) + if error=join_room(&d,&snap,&plan,folder,allocator);error!="" {return} + + // The files, smallest first, then what the host removed. + error=join_files(&d,&snap,&plan,&index,folder,&report,allocator) + if error=="" {join_remove(&last,&plan,&index,folder,&report)} + base_save(folder,&snap,&last,&plan) + index_save(folder,&index) + if error!="" {return} + + // The packages, their builds from the peers. + if os.exists(project.path(folder,asset.PACKAGES_FILE,allocator=context.temp_allocator)) { + say(&d,"refreshing the packages of the project") + progress_phase(&d,.Packages,&report) + report.packages=join_packages(&d,folder,allocator) + } + return +} + +@(private) +say :: proc(d:^Join_Desc,format:string,args:..any) { + if d.output.procedure!=nil {fmt.wprintfln(d.output,format,..args)} +} + +// Waits for the connection to the host, or says why there is none. +@(private) +join_connect :: proc(d:^Join_Desc,allocator:runtime.Allocator) -> string { + n:=d.node + // The clock of the node is its owner's from its first step on. + if !node_wait(n,0) {return strings.clone("the join was stopped",allocator)} + started:=node_now(n) + for { + sync.mutex_lock(&n.lock) + joined,reason:=n.client_joined,n.client_reason + sync.mutex_unlock(&n.lock) + if joined {return ""} + if reason!=.None || node_now(n)-started>d.connect_seconds {return refusal(reason,allocator)} + if !node_wait(n,0.05) {return strings.clone("the join was stopped",allocator)} + } +} + +// Why the connection to a host ended, and what to do. +refusal :: proc(reason:wire.Disconnect_Reason,allocator:=context.allocator) -> string { + #partial switch reason { + case .Denied_Version,.Denied_Protocol: + return strings.clone("the host runs another version of Mirai: both sides need the same engine",allocator) + case .Denied_Full: + return strings.clone("the host has all the peers it takes: join again when one leaves, or join from another peer that has the project",allocator) + case .Closed,.Kicked: + return strings.clone("the host closed the connection: it stopped sharing; run the join again with a new invitation and it resumes",allocator) + case .Timeout: + return strings.clone("the host stopped answering: run the same join again when it shares, and it resumes where it was",allocator) + } + return strings.clone("the host did not answer. Either its share has ended (an invitation lasts while the share that printed it runs: ask for a new one), or it cannot be reached from here: on another network one side needs a router that maps a port (UPnP, NAT-PMP or PCP) and the host must give the code it prints once its router answered; a firewall must let UDP through",allocator) +} + +// The address of the snapshot the host offers, asked of the host alone: a +// peer that joined from this one could name another. +@(private) +join_offer :: proc(d:^Join_Desc,allocator:runtime.Allocator) -> (address:store.Hash,error:string) { + n:=d.node + host,connection,connected:=node_host_peer(n) + if !connected {return {},refusal(.Timeout,allocator)} + q,ask_error:=exchange.ref_ask(&n.x,NAMESPACE,d.offer,host,connection) + if ask_error!=.None {return {},fmt.aprintf("the snapshot could not be asked for (%v)",ask_error,allocator=allocator)} + defer exchange.query_end(&n.x,q) + for { + result,ok:=exchange.query_result(&n.x,q) + if !ok {return {},strings.clone("the question for the snapshot was lost",allocator)} + switch result.state { + case .Found: return result.address,"" + case .Not_Found: return {},strings.clone("the host offers no project under this invitation, or did not answer: its share ended and another runs; ask for its new invitation",allocator) + case .Waiting: + } + if error=join_alive(d,allocator);error!="" {return} + } +} + +// Fails once the host is gone; otherwise waits a step. +@(private) +join_alive :: proc(d:^Join_Desc,allocator:runtime.Allocator) -> string { + n:=d.node + sync.mutex_lock(&n.lock) + connected,reason:=n.connected,n.client_reason + sync.mutex_unlock(&n.lock) + if connected==0 {return refusal(reason==.None?.Timeout:reason,allocator)} + if !node_wait(n,0.02) {return strings.clone("the join was stopped",allocator)} + return "" +} + +// Why a snapshot the host sent is refused. +@(private) +refused :: proc(error:Snapshot_Error,allocator:runtime.Allocator) -> string { + return fmt.aprintf("the host sent a snapshot this engine refuses (%v): it was not written; the host runs another version of Mirai, or is not honest",error,allocator=allocator) +} + +@(private) +Tree_Fetching :: struct {d:^Join_Desc,error:string,allocator:runtime.Allocator} + +@(private) +join_tree_fetch :: proc(data:rawptr,addresses:[]store.Hash) -> bool { + f:=(^Tree_Fetching)(data) + f.error=join_fetch(f.d,addresses,"the snapshot",f.allocator) + return f.error=="" +} + +// Fetches the folders of a snapshot at addresses from the peers, WINDOW at a +// time; one whose object is larger than a folder's is refused. +@(private) +join_fetch :: proc(d:^Join_Desc,addresses:[]store.Hash,what:string,allocator:runtime.Allocator) -> string { + n:=d.node + Active :: struct {address:store.Hash,missing:f64} + active:=make([dynamic]Active,context.temp_allocator) + defer for a in active {exchange.forget(&n.x,a.address)} + next,complete:=0,0 + stall:Stall + for next0 { + for len(active)LISTING_MAX {return fmt.aprintf("the host sent a folder of %s of %d bytes, more than the %d this engine takes: it was refused",what,p.total,i64(LISTING_MAX),allocator=allocator)} + #partial switch p.state { + case .Complete: + exchange.forget(&n.x,a.address) + unordered_remove(&active,k) + complete+=1 + continue + case .Failed: + return fmt.aprintf("%s could not be kept in the store at %s (%v, %v): free space there",what,n.s.root,p.error,p.store_error,allocator=allocator) + } + if join_missing(d,p.state,&a.missing) {return fmt.aprintf("no connected peer has %s: the host changed it while this join ran; run the join again",what,allocator=allocator)} + k+=1 + } + if len(active)==0 && next==len(addresses) {break} + if stalled(d,&stall,complete) {return fmt.aprintf("no connected peer sent any of %s for %.0f s: the host withholds it, or stopped; run the join again",what,STALL_SECONDS,allocator=allocator)} + if error:=join_alive(d,allocator);error!="" {return error} + } + return "" +} + +// Whether the join made no progress for STALL_SECONDS: no chunk byte received +// from any peer and no object complete, whatever the peers do or withhold. +@(private) +Stall :: struct {at:f64,received:u64,complete:int} + +@(private) +stalled :: proc(d:^Join_Desc,s:^Stall,complete:int) -> bool { + now:=node_now(d.node) + received:=node_totals(d.node).received + if s.at==0 || received!=s.received || complete!=s.complete { + s^={at=now,received=received,complete=complete} + return false + } + return now-s.at>=STALL_SECONDS +} + +// Whether an object stayed missing at every connected peer for MISSING_SECONDS. +@(private) +join_missing :: proc(d:^Join_Desc,state:exchange.State,since:^f64) -> bool { + if state!=.Missing { + since^=-1 + return false + } + now:=node_now(d.node) + if since^<0 {since^=now} + return now-since^>=MISSING_SECONDS +} + +// What the snapshot needs here, against the last one this folder took. +@(private) +join_plan :: proc(snap,last:^Snapshot,index:^Index,folder:string,report:^Join_Report,allocator:runtime.Allocator) -> (plan:Plan) { + plan.write=make([dynamic]Write,allocator) + plan.remove=make([dynamic]Remove,allocator) + if plan.folds=folds_case(folder);plan.folds { + plan.last_folded=make(map[string]int,len(last.entries),allocator) + plan.snap_folded=make(map[string]int,len(snap.entries),allocator) + for e,at in last.entries {plan.last_folded[strings.to_lower(e.path,allocator)]=at} + for e,at in snap.entries {plan.snap_folded[strings.to_lower(e.path,allocator)]=at} + case_follow(snap,last,&plan,index,folder) + } + for &e,at in snap.entries { + if link_refused(report,folder,e.path) { + append(&plan.write,Write{at=at,refused=true}) + continue + } + here,_,present:=index_address(index,e.path,project.path(folder,e.path,allocator=context.temp_allocator)) + if present && here==e.address {continue} + base,based:=base_find(&plan,last,e.path) + switch { + case !present && !based: + append(&plan.write,Write{at=at}) + case !present: + if base.address!=e.address {kept(report,"%s: you removed it and the host changed it; it stays removed",e.path)} + case based && here==base.address: + append(&plan.write,Write{at=at,present=true,base=here}) + case based && base.address==e.address: + // Only the joiner changed it: it is the joiner's. + case: + kept(report,"%s: you changed it and the host has another one; yours is kept",e.path) + } + } + for &e,at in last.entries { + if _,still:=snapshot_find(snap,e.path);still {continue} + // Renamed by the host changing only its case: on this system it is the same file. + if plan.folds && strings.to_lower(e.path,context.temp_allocator) in plan.snap_folded {continue} + if link_refused(report,folder,e.path) { + append(&plan.remove,Remove{at=at,refused=true}) + continue + } + here,_,present:=index_address(index,e.path,project.path(folder,e.path,allocator=context.temp_allocator)) + switch { + case !present: + case here==e.address: append(&plan.remove,Remove{at=at}) + case: kept(report,"%s: the host removed it and you changed it; yours is kept",e.path) + } + } + return +} + +// The first part of relative under folder that is a link (a symbolic link, or +// a junction on Windows), as a path; "" when none. The folder itself may be +// one: it is the place the joiner named. +@(private) +link_on :: proc(folder,relative:string) -> string { + for at in 1..=len(relative) { + if at bool { + link:=link_on(folder,relative) + if link=="" {return false} + failed(report,"%s: %s is a link, and a join never writes or removes through a link inside its folder: replace it with a folder or file of its own and run the join again",relative,link) + return true +} + +// The entry of path in the last snapshot, under any case where the system folds it. +@(private) +base_find :: proc(plan:^Plan,last:^Snapshot,path:string) -> (^Entry,bool) { + if e,found:=snapshot_find(last,path);found {return e,true} + if !plan.folds {return nil,false} + at,found:=plan.last_folded[strings.to_lower(path,context.temp_allocator)] + if !found {return nil,false} + return &last.entries[at],true +} + +// Whether the file system of folder folds case, as APFS and NTFS do unless +// formatted otherwise: a file made with a name in lower case is found under it +// in upper case. +@(private) +folds_case :: proc(folder:string) -> bool { + temp_folder:=project.path(project.folder_path(folder,.Generated,context.temp_allocator),"tmp",allocator=context.temp_allocator) + if os.make_directory_all(temp_folder)!=nil && !os.is_dir(temp_folder) {return false} + name:=fmt.tprintf("share-case-%d",os.get_pid()) + lower:=project.path(temp_folder,name,allocator=context.temp_allocator) + if os.write_entire_file(lower,nil)!=nil {return false} + defer os.remove(lower) + return os.exists(project.path(temp_folder,strings.to_upper(name,context.temp_allocator),allocator=context.temp_allocator)) +} + +// Where the system folds case, a file the host renamed changing only the case +// of its path is renamed here too, folders first, where the joiner left the +// name of the last snapshot; a name the joiner changed stays. The index follows. +@(private) +case_follow :: proc(snap,last:^Snapshot,plan:^Plan,index:^Index,folder:string) { + temp_folder:=project.path(project.folder_path(folder,.Generated,context.temp_allocator),"tmp",allocator=context.temp_allocator) + for &e in snap.entries { + if _,same:=snapshot_find(last,e.path);same {continue} + at,renamed:=plan.last_folded[strings.to_lower(e.path,context.temp_allocator)] + if !renamed {continue} + old:=last.entries[at].path + if len(old)!=len(e.path) || link_on(folder,old)!="" {continue} + start:=0 + for i in 0..=len(e.path) { + if i string { + s:=d.node.s + fetch,write:i64 + seen:=make(map[store.Hash]bool,context.temp_allocator) + // The root read once: each file then asks only what this value knows. + store.chunk_has(s,{}) + for w in plan.write { + if w.refused {continue} + e:=&snap.entries[w.at] + write+=e.size + if e.address in seen || store.chunk_here(s,e.address) {continue} + seen[e.address]=true + fetch+=e.size + } + existing:=folder + for !os.exists(existing) { + parent:=os.dir(existing) + if parent==existing {break} + existing=parent + } + store_free,store_known:=free_bytes(s.root) + folder_free,folder_known:=free_bytes(existing) + store_info,_:=os.stat(s.root,context.temp_allocator) + folder_info,_:=os.stat(existing,context.temp_allocator) + if store_known && folder_known && store_info.device==folder_info.device && store_info.device!=0 { + if fetch+write>store_free {return fmt.aprintf("the disk of %s and %s has %s free and the join needs %s: free space, or give another folder or --store",s.root,folder,bytes_text(store_free),bytes_text(fetch+write),allocator=allocator)} + return "" + } + if store_known && fetch>store_free {return fmt.aprintf("the disk of the store at %s has %s free and the join needs %s there: free space, or give another --store",s.root,bytes_text(store_free),bytes_text(fetch),allocator=allocator)} + if folder_known && write>folder_free {return fmt.aprintf("the disk of %s has %s free and the join needs %s there: free space, or give another folder",folder,bytes_text(folder_free),bytes_text(write),allocator=allocator)} + return "" +} + +// Fetches the objects of the files to write, WINDOW at once: the project's +// manifest and its start scene first, then the others smallest first; each +// file is written as its object completes. A file whose object does not come +// is reported and the others go on; the error is what stops the whole join. +@(private) +join_files :: proc(d:^Join_Desc,snap:^Snapshot,plan:^Plan,index:^Index,folder:string,report:^Join_Report,allocator:runtime.Allocator) -> string { + n:=d.node + s:=n.s + // The writes of each address, and the addresses to fetch in order. + paths:=make(map[store.Hash][dynamic]int,context.temp_allocator) + order:=make([dynamic]int,context.temp_allocator) + total:i64 + for w,k in plan.write { + if w.refused {continue} + e:=&snap.entries[w.at] + if e.size==0 { + one:=[1]int{k} + join_write(d,snap,plan,one[:],index,folder,report) + continue + } + list,known:=&paths[e.address] + if !known { + paths[e.address]=make([dynamic]int,context.temp_allocator) + list=&paths[e.address] + append(&order,k) + if !store.chunk_here(s,e.address) {total+=e.size} + } + append(list,k) + } + if len(order)==0 {return ""} + scene:=join_scene(d,snap,index,folder) + Sized :: struct {first:bool,size:i64,k:int} + sized:=make([]Sized,len(order),context.temp_allocator) + for k,at in order { + e:=&snap.entries[plan.write[k].at] + sized[at]={e.path==project.PROJECT_FILE || e.path==scene,e.size,k} + } + slice.sort_by(sized,proc(a,b:Sized) -> bool { + if a.first!=b.first {return a.first} + return a.size0 { + for len(active)=d.pace { + t:=node_totals(n) + dt:=now-printed + rate=f64(t.received-received_at)/dt + received_at,printed=t.received,now + left:=total-done-here + eta:=rate>0?f64(left)/rate:-1 + say(d,"%d of %d files, %s of %s, %s/s, %s left",next-len(active),len(order),bytes_text(done+here),bytes_text(total),bytes_text(i64(rate)),eta<0?"time":seconds_text(max(eta,0))) + } + if d.progress!=nil { + left:=total-done-here + progress_set(d,{phase=.Files,files=len(order),files_done=complete_count,bytes=total,bytes_done=done+here,received=node_totals(n).received,rate=rate,left=rate>0?f64(left)/rate:-1,kept=len(report.kept)}) + } + if len(active)==0 && next==len(order) {break} + // Nothing arrives: what is wanted fails by path, and the files after it are asked. + if stalled(d,&stall,complete_count) { + for a in active { + e:=&snap.entries[plan.write[a.k].at] + for w in paths[e.address] {failed(report,"%s: no connected peer sent any of it for %.0f s: the host withholds it, or stopped; run the join again",snap.entries[plan.write[w].at].path,STALL_SECONDS)} + exchange.forget(&n.x,e.address) + } + clear(&active) + stall={} + continue + } + if error:=join_alive(d,allocator);error!="" {return error} + } + elapsed:=node_now(n)-started + say(d,"fetched %d files, %s in %s (%s/s)",len(order),bytes_text(total),seconds_text(elapsed),bytes_text(i64(f64(total)/max(elapsed,1e-3)))) + return "" +} + +// The path of the scene the project's manifest starts with, "" when it names +// none: read from the folder when the file there is the host's, otherwise +// fetched first. +@(private) +join_scene :: proc(d:^Join_Desc,snap:^Snapshot,index:^Index,folder:string) -> string { + e,found:=snapshot_find(snap,project.PROJECT_FILE) + if !found || e.size>1<<20 {return ""} + path:=project.path(folder,project.PROJECT_FILE,allocator=context.temp_allocator) + // One store per value until odin-lang/Odin#7937 (see ODIN_WORKAROUNDS): each manifest is read where it is declared. + if here,_,present:=index_address(index,project.PROJECT_FILE,path);present && here==e.address { + m,read_error:=project.manifest_read(path) + if read_error.code!=.None {return ""} + return start_scene(&m) + } + roots:=[1]store.Hash{e.address} + if join_fetch(d,roots[:],project.PROJECT_FILE,context.temp_allocator)!="" {return ""} + o,open_error:=store.object_open(d.node.s,e.address) + if open_error!=.None {return ""} + defer store.object_close(&o) + text:=make([]u8,int(store.object_size(&o)),context.temp_allocator) + if store.object_read(&o,0,text)!=.None {return ""} + m,parse_error:=project.manifest_parse(string(text)) + if parse_error.code!=.None {return ""} + return start_scene(&m) +} + +// The path of the start scene of a manifest, in the temp allocator; "": none. +@(private) +start_scene :: proc(m:^project.Manifest) -> string { + scene:=base.text_get(m.start_scene[:]) + if scene=="" {return ""} + return fmt.tprintf("assets/%s",scene) +} + +// Puts in the store the file the joiner has of the writes of list where the +// store lacks the base it was planned at, cut as the host's is: the chunks its +// old version shares with the new are then not asked. +@(private) +join_reuse :: proc(d:^Join_Desc,snap:^Snapshot,plan:^Plan,list:[]int,folder:string) { + s:=d.node.s + for k in list { + w:=&plan.write[k] + if !w.present || store.chunk_here(s,w.base) {continue} + file_put_object(s,project.path(folder,snap.entries[w.at].path,allocator=context.temp_allocator)) + store.commit(s) + return + } +} + +// Writes the planned writes list of a complete object into the folder, each +// whole through a rename, unless the joiner changed the file since the plan; +// what cannot be written is reported with how to fix it. +@(private) +join_write :: proc(d:^Join_Desc,snap:^Snapshot,plan:^Plan,list:[]int,index:^Index,folder:string,report:^Join_Report) { + s:=d.node.s + temp_folder:=project.path(project.folder_path(folder,.Generated,context.temp_allocator),"tmp",allocator=context.temp_allocator) + if os.make_directory_all(temp_folder)!=nil && !os.is_dir(temp_folder) { + for k in list {failed(report,"%s: %s could not be made: give this user access to %s, and run the join again",snap.entries[plan.write[k].at].path,temp_folder,folder)} + return + } + for k in list { + w:=&plan.write[k] + if w.refused {continue} + e:=&snap.entries[w.at] + target:=project.path(folder,e.path,allocator=context.temp_allocator) + if link_refused(report,folder,e.path) {continue} + here,_,present:=index_address(index,e.path,target) + if present!=w.present || present && here!=w.base { + if !present || here!=e.address {kept(report,"%s: you changed it while the join ran; yours is kept",e.path)} + w.done=true + continue + } + if present && !writable(target) { + failed(report,"%s is read-only: make it writable (chmod u+w on macOS and Linux, attrib -r on Windows) and run the join again",target) + continue + } + temp:=fmt.tprintf("%s/share-%d-%d",temp_folder,os.get_pid(),w.at) + mode:=os.Permissions_Default_File + if e.executable {mode+=os.Permissions_Execute_All} + #partial switch saved,store_error,os_error:=object_save(s,e.address,e.size,temp);saved { + case .None: + case .Size: + os.remove(temp) + failed(report,"%s: the snapshot says %d bytes and its object has others: the host is not honest; it was not written",e.path,e.size) + continue + case .Store: + os.remove(temp) + failed(report,"%s: its object does not read from the store at %s (%v): run the join again",e.path,s.root,store_error) + continue + case: + os.remove(temp) + failed(report,"%s could not be written (%v): free space on its disk, or give this user access to %s, and run the join again",target,os_error,temp_folder) + continue + } + os.change_mode(temp,mode) + if h:=d.node.hooks;h!=nil && h.saved!=nil {h.saved(h.data,e.path)} + // The copy of a large file takes time: an edit made meanwhile is kept too. + if again,_,still:=index_address(index,e.path,target);still!=present || still && again!=here { + os.remove(temp) + if !still || again!=e.address {kept(report,"%s: you changed it while the join ran; yours is kept",e.path)} + w.done=true + continue + } + parent:=os.dir(target) + if make_error:=os.make_directory_all(parent);make_error!=nil && !os.is_dir(parent) { + os.remove(temp) + failed(report,"%s could not be made (%v): a file of yours has the name of a folder of the host's, or this user cannot write there; move it or give access, and run the join again",parent,make_error) + continue + } + if !writable(parent) { + os.remove(temp) + failed(report,"%s: its folder %s is read-only: give this user write access to it (chmod u+w on macOS and Linux) and run the join again",e.path,parent) + continue + } + // A link made inside the folder while the file was written is not followed either. + if link_refused(report,folder,e.path) { + os.remove(temp) + continue + } + if rename_error:=os.rename(temp,target);rename_error!=nil { + os.remove(temp) + failed(report,"%s could not be replaced (%v): close what holds it, or give this user access, and run the join again",target,rename_error) + continue + } + if info,stat_error:=os.stat(target,context.temp_allocator);stat_error==nil {index_learn(index,e.path,info,e.address)} + report.written+=1 + w.done=true + } +} + +@(private) +Save_Error :: enum u8 {None,Size,Store,Write} + +// Writes the object at address, of size bytes, into a new file at path chunk +// by chunk at the offset of each, walking its tree one node per level: what +// it holds does not grow with the file. Size: the object holds other bytes. +@(private) +object_save :: proc(s:^store.Store,address:store.Hash,size:i64,path:string) -> (error:Save_Error,store_error:store.Error,os_error:os.Error) { + if size==0 { + if os_error=os.write_entire_file(path,nil);os_error!=nil {return .Write,.None,os_error} + return + } + o,open_error:=store.object_open(s,address) + if open_error!=.None {return .Store,open_error,nil} + defer store.object_close(&o) + if store.object_size(&o)!=u64(size) {return .Size,.None,nil} + file,create_error:=os.open(path,{.Write,.Create,.Trunc},os.Permissions_Default_File) + if create_error!=nil {return .Write,.None,create_error} + defer os.close(file) + if os_error=os.truncate(file,size);os_error!=nil {return .Write,.None,os_error} + Frame :: struct {node:store.Node_View,next:int} + stack:[store.NODE_LEVEL_MAX+1]Frame + buffers:[store.NODE_LEVEL_MAX+1][]u8 + defer for b in buffers {delete(b,context.temp_allocator)} + chunk,_:=make([]u8,store.CHUNK_MAX,context.temp_allocator) + defer delete(chunk,context.temp_allocator) + stack[0]={node=o.root} + depth:=1 + at:i64 + for depth>0 { + top:=&stack[depth-1] + if top.next==len(top.node.entries) { + depth-=1 + continue + } + e:=top.node.entries[top.next] + top.next+=1 + level:=top.node.level + if level==0 { + bytes,read_error:=store.chunk_read(s,e.address,chunk) + if read_error!=.None {return .Store,read_error,nil} + if u64(len(bytes))!=e.size {return .Store,.Corrupt,nil} + for written:=0;written0 || len(child.meta)>0 {return .Store,.Corrupt,nil} + stack[depth]={node=child} + depth+=1 + } + if at!=size {return .Size,.None,nil} + return +} + +// Removes what the host removed, unless the joiner changed it since the plan +// or it cannot be removed, which is reported with how to fix it. +@(private) +join_remove :: proc(last:^Snapshot,plan:^Plan,index:^Index,folder:string,report:^Join_Report) { + for &r in plan.remove { + if r.refused {continue} + e:=&last.entries[r.at] + target:=project.path(folder,e.path,allocator=context.temp_allocator) + if link_refused(report,folder,e.path) {continue} + here,_,present:=index_address(index,e.path,target) + switch { + case !present: + r.done=true + case here!=e.address: + kept(report,"%s: the host removed it and you changed it; yours is kept",e.path) + r.done=true + case !writable(target): + failed(report,"%s is read-only and the host removed it: make it writable (chmod u+w on macOS and Linux, attrib -r on Windows) and run the join again",target) + case: + if remove_error:=os.remove(target);remove_error!=nil { + failed(report,"%s could not be removed (%v): close what holds it, or give this user write access to its folder, and run the join again",target,remove_error) + continue + } + report.removed+=1 + delete_key(index,e.path) + folders_prune(folder,e.path) + r.done=true + } + } +} + +// Keeps as the base of the next join the host's snapshot, but for what this +// one did not do: a file not written keeps its entry of the last snapshot, and +// a file not removed its removed entry, so that the next join does them. +@(private) +base_save :: proc(folder:string,snap,last:^Snapshot,plan:^Plan) { + undone:=make(map[int]bool,context.temp_allocator) + for w in plan.write { + if !w.done {undone[w.at]=true} + } + entries:=make([dynamic]Entry,0,len(snap.entries)+len(plan.remove),context.temp_allocator) + for e,at in snap.entries { + if at not_in undone { + append(&entries,e) + } else if base,based:=base_find(plan,last,e.path);based { + append(&entries,base^) + } + } + for r in plan.remove { + if !r.done {append(&entries,last.entries[r.at])} + } + slice.sort_by(entries[:],proc(a,b:Entry) -> bool {return a.path0;at=strings.last_index_byte(path[:at],'/') { + folder:=project.path(root,path[:at],allocator=context.temp_allocator) + listed,error:=os.read_all_directory_by_path(folder,context.temp_allocator) + if error!=nil || len(listed)>0 || os.remove(folder)!=nil {return} + } +} + +// The project's packages refreshed with the peers as the source, on a thread +// of its own, while this one says how far it is: as an editor refreshes them, +// a Final or a Fast build taken from the store and then from the peers, and a +// source imported only when none gives one, in the quality an editor imports. +@(private) +join_packages :: proc(d:^Join_Desc,folder:string,allocator:runtime.Allocator) -> sources.Library_Report { + n:=d.node + Work :: struct { + options:sources.Library_Options, + folder:string, + report:sources.Library_Report, + allocator:runtime.Allocator, + done:bool, + } + work:=Work{folder=folder,allocator=allocator} + work.options={store=n.s,source=node_source(n),fast_textures=true,root=folder,plugins=d.library,worker=d.worker,progress=d.output} + t:=thread.create_and_start_with_data(&work,proc(data:rawptr) { + w:=(^Work)(data) + w.report=sources.project_refresh(w.folder,w.options,w.allocator) + free_all(context.temp_allocator) + sync.atomic_store(&w.done,true) + }) + if t==nil { + work.report=sources.project_refresh(folder,work.options,allocator) + return work.report + } + started:=node_now(n) + from:=node_totals(n).received + printed:=started + last:=from + for !sync.atomic_load(&work.done) { + if !node_wait(n,0.05) {break} + now:=node_now(n) + if now-printed string { + value:=f64(bytes) + switch { + case value>=1<<30: return fmt.tprintf("%.2f GiB",value/(1<<30)) + case value>=1<<20: return fmt.tprintf("%.1f MiB",value/(1<<20)) + case value>=1<<10: return fmt.tprintf("%.1f KiB",value/(1<<10)) + } + return fmt.tprintf("%d bytes",bytes) +} + +seconds_text :: proc(seconds:f64) -> string { + whole:=int(seconds+0.5) + if whole<60 {return fmt.tprintf("%d s",whole)} + if whole<3600 {return fmt.tprintf("%d min %02d s",whole/60,whole%60)} + return fmt.tprintf("%d h %02d min",whole/3600,whole/60%60) +} diff --git a/cook/share/library.odin b/cook/share/library.odin new file mode 100644 index 000000000..b9456508c --- /dev/null +++ b/cook/share/library.odin @@ -0,0 +1,342 @@ +package share + +import "core:fmt" +import "core:os" +import "core:slice" +import "core:strings" +import "core:sync" +import "mirai:asset" +import "mirai:cook/import/sources" +import "mirai:exchange" +import "mirai:project" +import "mirai:std/base" +import "mirai:std/jobs" +import "mirai:store" +import wire "mirai:net" + +// The libraries of the peers of a node: the packages each one's machine +// exposes (asset.REF_PACKAGES, with the builds of asset.REF_IMPORTS and +// asset.REF_IMPORTS_FAST, which node_expose already lets it read), and taking +// one of them into this machine's library, builds included, so a project then +// pins it as a package ingested here. Everything a peer names is fetched by +// address and read only through asset's checks of manifests and builds. + +// A connected peer of the node. +Peer :: struct { + host:^wire.Host, + connection:wire.Connection, + number:int, // given in order of arrival, as Event.peer +} + +// The connected peers, into buffer. +node_peers :: proc(n:^Node,buffer:[]Peer) -> []Peer { + sync.mutex_guard(&n.lock) + count:=0 + for p in n.peers { + if !p.used || count==len(buffer) {continue} + buffer[count]={p.host,p.connection,p.number} + count+=1 + } + return buffer[:count] +} + +// A package a peer's library exposes. bytes: of the objects of the build it +// has, -1 when unknown; built: it has a build this engine reads; here: this +// machine has its manifest and that build whole, so adding it fetches nothing. +Peer_Package :: struct { + id:base.Asset_Id, + manifest:store.Hash, + name:string, + origin:string, // where its source was on the machine that ingested it, which tells apart two of one name + kinds:[]asset.Fourcc, + assets:int, + root:store.Hash, // the object its card shows (asset.build_shown) in the build it has; zero: none + objects:[]store.Hash, // of every asset of that build, sorted + + bytes:i64, + built,here:bool, +} + +@(private) LIBRARY_NAMES_MAX :: 1<<14 // names of one namespace a listing reads of a peer +@(private) SIZE_SECONDS :: 10.0 // the most a listing waits for the sizes of objects + +// The packages the library of peer exposes, by name, with their builds; error +// says why there is none (the peer left, or answers nothing). For a thread +// other than the pump's. In allocator. +peer_library :: proc(n:^Node,peer:Peer,allocator:=context.allocator) -> (list:[]Peer_Package,error:string) { + packages,packages_error:=peer_names(n,peer,asset.REF_PACKAGES) + if packages_error!="" {return nil,packages_error} + builds:=make([dynamic]exchange.Name,context.temp_allocator) + for namespace in ([2]string{asset.REF_IMPORTS,asset.REF_IMPORTS_FAST}) { + names,names_error:=peer_names(n,peer,namespace) + if names_error!="" {return nil,names_error} + append(&builds,..names) + } + // The manifests and the builds are small: fetched together. A manifest is + // named by the hash of its text, its object by asset.REF_MANIFESTS. + manifests_fetch(n,packages[:]) + wanted:=make([dynamic]store.Hash,context.temp_allocator) + for b in builds {append(&wanted,b.address)} + objects_fetch(&n.x,wanted[:]) + s:=n.s + chosen:=make(map[base.Asset_Id]asset.Build,context.temp_allocator) + for b in builds { + if !asset.object_whole(s,b.address) {continue} + build,build_error:=asset.build_get(s,b.address,context.temp_allocator) + if build_error!=.None {continue} + // Final over Fast. + if had,known:=chosen[build.package_id];known && had.final {continue} + chosen[build.package_id]=build + } + out:=make([dynamic]Peer_Package,allocator) + sizes:=make([dynamic]store.Hash,context.temp_allocator) + for named in packages { + p,package_error:=asset.package_get(s,named.address,context.temp_allocator) + if package_error!=.None {continue} + kinds:=make([dynamic]asset.Fourcc,allocator) + for a in p.assets { + known:=false + for k in kinds {known||=k==a.kind} + if !known {append(&kinds,a.kind)} + } + entry:=Peer_Package{id=p.id,manifest=named.address,name=strings.clone(p.name,allocator),origin=strings.clone(p.origin.path,allocator),kinds=kinds[:],assets=len(p.assets),bytes=-1} + // Whether it is here is read before the sizes are asked, which may bring a small object whole. + if b,built:=chosen[p.id];built { + entry.built,entry.here=true,true + objects:=make([]store.Hash,len(b.outputs),allocator) + entry.root,_=asset.build_shown(&b,&p) + for o,k in b.outputs { + append(&sizes,o.object) + entry.here&&=asset.object_whole(s,o.object) + objects[k]=o.object + } + slice.sort_by(objects,proc(a,b:store.Hash) -> bool { + a,b:=a,b + return string(a[:]) bool {return a.name (names:[]exchange.Name,error:string) { + out:=make([dynamic]exchange.Name,context.temp_allocator) + after:="" + for len(out)0 + if more {after=out[len(out)-1].name} + exchange.query_end(&n.x,q) + if !more {break} + } + return out[:],"" +} + +@(private) +node_peer_alive :: proc(n:^Node,peer:Peer) -> bool { + sync.mutex_guard(&n.lock) + for p in n.peers { + if p.used && p.host==peer.host && p.connection==peer.connection {return true} + } + return false +} + +// The size of each object of addresses: here, or as its root says once it +// arrives from a peer (exchange.measure): nothing else of the object is fetched. Those whose root no +// peer sends within SIZE_SECONDS are left out. +@(private) +objects_sizes :: proc(n:^Node,addresses:[]store.Hash) -> map[store.Hash]i64 { + sizes:=make(map[store.Hash]i64,context.temp_allocator) + asking:=make([dynamic]store.Hash,context.temp_allocator) + for address in addresses { + if address in sizes {continue} + if o,error:=store.object_open(n.s,address);error==.None { + sizes[address]=i64(store.object_size(&o)) + store.object_close(&o) + continue + } + // Its root alone: none of what the object holds is fetched to know its size. + if exchange.measure(&n.x,address)==.None {append(&asking,address)} + } + defer for address in asking {exchange.forget(&n.x,address)} + started:=node_now(n) + for len(asking)>0 && node_now(n)-started=0 || p.state==.Failed || p.state==.Complete { + if p.total>=0 {sizes[asking[k]]=p.total} + exchange.forget(&n.x,asking[k]) + unordered_remove(&asking,k) + continue + } + k+=1 + } + if len(asking)>0 && !node_wait(n,0.02) {break} + } + return sizes +} + +// Takes the package of manifest from the peers into this machine's library: +// its manifest and the build of it this engine reads, Fast textures allowed, +// fetched as a join fetches the builds of a project, and its name under +// asset.REF_PACKAGES, so it stays in the library once the peers leave. root: +// the project it is for, whose plugins read a source should one be here. +// Nothing is pinned: on failure no project changed. For a thread other than +// the pump's. +package_take :: proc(n:^Node,manifest:store.Hash,root:string,library:sources.Library=nil,worker:=jobs.Worker_Ref{}) -> (error:string) { + s:=n.s + named:=[1]exchange.Name{{address=manifest}} + manifests_fetch(n,named[:]) + p,package_error:=asset.package_get(s,manifest,context.temp_allocator) + if package_error!=.None { + digits:[64]u8 + if _,ref_error:=store.ref_get(s,asset.REF_MANIFESTS,store.hex(manifest,&digits));ref_error!=.None {return "no connected peer has its manifest any more: the peer that offered it left; add it again once it is back"} + return fmt.tprintf("its manifest is not one this engine reads (%v): the peer runs another version of Mirai",package_error) + } + // A project of its own under the generated folder of root, which pins the + // package and refreshes it from the peers, then goes: what it fetched stays in the store. + digits:[64]u8 + scratch:=project.path(project.folder_path(root,.Generated,context.temp_allocator),"tmp",fmt.tprintf("share-take-%s",store.hex(manifest,&digits)[:16]),allocator=context.temp_allocator) + base.remove_all(scratch) + if os.make_directory_all(scratch)!=nil {return fmt.tprintf("%s could not be made: give this user access to %s",scratch,root)} + defer base.remove_all(scratch) + options:=sources.Library_Options{store=s,source=node_source(n),fast_textures=true,root=root,plugins=library,worker=worker} + report:=sources.project_add(scratch,manifest,"",nil,options,context.temp_allocator) + if len(report.failures)>0 {return report.failures[0]} + if len(report.missing)>0 || report.fetched+report.ingested==0 && !package_built(s,manifest,&p) { + return fmt.tprintf("%s: no connected peer gave its build any more, and its source is not on this machine: the peer that offered it left; add it again once it is back",p.name) + } + buffer:[base.ASSET_ID_TEXT]u8 + if store.ref_set(s,asset.REF_PACKAGES,base.asset_id_format(p.id,buffer[:]),manifest)!=.None { + return fmt.tprintf("%s could not be kept in the library of the store at %s: free space there",p.name,s.root) + } + return "" +} + +// Whether this machine has a build of p it chose and has whole. +@(private) +package_built :: proc(s:^store.Store,manifest:store.Hash,p:^asset.Package) -> bool { + digits:[64]u8 + hash,ref_error:=store.ref_get(s,asset.REF_BUILDS,store.hex(asset.build_select(manifest,p),&digits)) + if ref_error!=.None {return false} + b,error:=asset.build_get(s,hash,context.temp_allocator) + if error!=.None {return false} + for o in b.outputs { + if !asset.object_whole(s,o.object) {return false} + } + return true +} + +// A ref one peer names, and the small object it names, fetched without +// waiting: for derived data a peer has or lacks (a thumbnail), never asked of +// another peer, and never made on request. fetch_poll steps it from the thread +// that pumps; a peer that leaves ends it as Absent. +Fetch_State :: enum u8 {Asking,Fetching,Done,Absent} + +Fetch :: struct { + peer:Peer, + namespace,name:string, // held by the caller while it fetches + query:exchange.Query, + address:store.Hash, + state:Fetch_State, +} + +fetch_start :: proc(n:^Node,peer:Peer,namespace,name:string) -> (f:Fetch) { + f={peer=peer,namespace=namespace,name=name} + if _,error:=store.ref_get(n.s,namespace,name);error==.None { + f.state=.Done + return + } + q,error:=exchange.ref_ask(&n.x,namespace,name,peer.host,peer.connection) + if error!=.None {f.state=.Absent;return} + f.query=q + return +} + +// One step of a fetch: the answer of the peer, then the object, which this +// store keeps under the same ref once it is here. +fetch_poll :: proc(n:^Node,f:^Fetch) -> Fetch_State { + switch f.state { + case .Done,.Absent: + case .Asking: + r,ok:=exchange.query_result(&n.x,f.query) + if !ok || r.state==.Not_Found || r.state==.Waiting && !node_peer_alive(n,f.peer) { + if ok {exchange.query_end(&n.x,f.query)} + f.state=.Absent + break + } + if r.state==.Waiting {break} + f.address=r.address + exchange.query_end(&n.x,f.query) + if exchange.want(&n.x,f.address,1<<20)!=.None {f.state=.Absent;break} + f.state=.Fetching + case .Fetching: + p,_:=exchange.progress(&n.x,f.address) + #partial switch p.state { + case .Complete: + exchange.forget(&n.x,f.address) + f.state=store.ref_set(n.s,f.namespace,f.name,f.address)==.None?.Done:.Absent + case .Failed,.Missing: + exchange.forget(&n.x,f.address) + f.state=.Absent + case: + if !node_peer_alive(n,f.peer) { + exchange.forget(&n.x,f.address) + f.state=.Absent + } + } + } + return f.state +} + +// Lets go of a fetch that did not end. +fetch_end :: proc(n:^Node,f:^Fetch) { + #partial switch f.state { + case .Asking: exchange.query_end(&n.x,f.query) + case .Fetching: exchange.forget(&n.x,f.address) + } + f.state=.Absent +} diff --git a/cook/share/node.odin b/cook/share/node.odin new file mode 100644 index 000000000..2f656e2b9 --- /dev/null +++ b/cook/share/node.odin @@ -0,0 +1,388 @@ +package share + +import "base:runtime" +import "core:crypto" +import "core:fmt" +import "core:strings" +import "core:sync" +import "mirai:asset" +import "mirai:cook/cache" +import "mirai:exchange" +import "mirai:std/jobs" +import "mirai:store" +import wire "mirai:net" + +// A peer of a shared project: a store, the exchange over the hosts of its +// owner (a server others join, the client of a join), the lease that keeps +// what it serves and fetches from a sweep, and what it saw of its peers. +// Whoever owns it calls node_pump once per step, or, when it polls its hosts +// itself (an editor's session, whose hosts carry its own channels too), +// node_command for each command of a host and node_step; a join runs on +// another thread and reads the clock and waits through the node. + +NAMESPACE :: "shares" // : the snapshot a share offers under it +PROTOCOL_ID :: 0x4d495348 // of the hosts of sharing: datagrams of any other application are ignored +VERSION :: 4 // of what two peers say to each other: the snapshot and the exchange over it +PEERS :: 8 // connections of a node at once +LEASE_SECONDS :: 60.0 // between two refreshes of the lease +@(private) WANTS :: 512 +@(private) EVENTS :: 64 + +// What the hosts of a node carry: the channels of the exchange, from the first. +channels :: proc() -> [exchange.CHANNELS]wire.Channel_Desc {return exchange.channels()} + +Event_Kind :: enum u8 {Joined,Left} + +Event :: struct { + kind:Event_Kind, + peer:int, // a number given in order of arrival + reason:wire.Disconnect_Reason, // Left + sent,received:u64, // chunk bytes with that peer, Left +} + +@(private) +Node_Peer :: struct { + used:bool, + host:^wire.Host, + connection:wire.Connection, + number:int, +} + +Node :: struct { + s:^store.Store, + x:exchange.Exchange, + hosts:[2]^wire.Host, + host_count:int, + allocator:runtime.Allocator, + // Between the pump and the thread that joins. + lock:sync.Mutex, + tick:sync.Cond, + now:f64, + stepped:bool, // now is a time of the owner's clock: before the first step it is not + waiting:int, // threads in node_wait + peers:[PEERS]Node_Peer, + numbers:int, + connected:int, + client_reason:wire.Disconnect_Reason, // why the client host was refused or dropped + client_joined:bool, + events:[EVENTS]Event, + event_head,event_count:int, + sent_gone,received_gone:u64, // of peers that left + served:[dynamic]store.Hash, // what the lease holds besides the wants + lease:store.Lease, + leased:bool, + lease_at:f64, + lease_due:bool, + stopped:bool, + hooks:^Join_Hooks, +} + +// What a test does at the crash points of a join. +@(private) +Join_Hooks :: struct { + saved:proc(data:rawptr,path:string), // a file's replacement is whole in its temporary file + data:rawptr, +} + +// worker: the job system and worker of the thread that pumps, which reads, +// hashes, stores and commits for the exchange; none: all of it runs on the pump. +// channel: where the channels of channels() start in the channels of every host. +node_init :: proc(n:^Node,s:^store.Store,allocator:=context.allocator,worker:=jobs.Worker_Ref{},channel:=0) -> exchange.Error { + n^={s=s,allocator=allocator} + n.served=make([dynamic]store.Hash,allocator) + return exchange.init(&n.x,{store=s,channel=channel,peers=PEERS,wants=WANTS,expose=node_expose,expose_user=n,allocator=allocator,worker=worker}) +} + +node_destroy :: proc(n:^Node) { + node_stop(n) + exchange.destroy(&n.x) + if n.leased {store.lease_release(&n.lease)} + delete(n.served) +} + +// Wakes whoever waits on the node, which waits no more: a join running ends. +node_stop :: proc(n:^Node) { + sync.mutex_guard(&n.lock) + n.stopped=true + sync.cond_broadcast(&n.tick) +} + +// Stops the node and lets go of every peer, as if they all left: what a join +// waits for from them returns at once. For an owner that ends while a join runs. +node_close :: proc(n:^Node) { + node_stop(n) + sync.mutex_lock(&n.lock) + peers:=n.peers + sync.mutex_unlock(&n.lock) + for p in peers { + if p.used {node_left(n,p.host,p.connection,.Closed)} + } +} + +// Adds a host whose channels start with those of channels(); the node polls it. +node_host :: proc(n:^Node,h:^wire.Host) -> bool { + if n.host_count==len(n.hosts) {return false} + n.hosts[n.host_count]=h + n.host_count+=1 + return true +} + +// What peers may read: the snapshots offered, the manifests of packages, their +// builds, Final and Fast each under its own namespace, and the results of +// cook/cache. Never the choices, the origins or the sources of this machine. +@(private) +node_expose :: proc(user:rawptr,namespace:string) -> bool { + switch namespace { + case NAMESPACE,asset.REF_PACKAGES,asset.REF_MANIFESTS,asset.REF_IMPORTS,asset.REF_IMPORTS_FAST: return true + } + return strings.has_prefix(namespace,asset.REF_CACHE_PREFIX) +} + +// One step: every host polled, the exchange updated at now (seconds, never +// back), the lease refreshed when due, and whoever waits on the node woken. +node_pump :: proc(n:^Node,now:f64) { + for h in n.hosts[:n.host_count] { + for command in wire.host_poll(h) {node_command(n,h,command)} + } + node_step(n,now) +} + +// A command of a host its owner polled: a connection that came or went, or a +// message, which it says was the exchange's. The owner sees every command too. +node_command :: proc(n:^Node,h:^wire.Host,command:wire.Command) -> (own:bool) { + #partial switch command.kind { + case .Connected: node_joined(n,h,command.connection) + case .Disconnected: node_left(n,h,command.connection,command.reason) + case .Message: return exchange.receive(&n.x,h,command) + } + return false +} + +// The step of node_pump after its hosts were polled. +node_step :: proc(n:^Node,now:f64) { + exchange.update(&n.x,now) + sync.mutex_lock(&n.lock) + n.now=now + n.stepped=true + due:=n.lease_due || now-n.lease_at>=LEASE_SECONDS + sync.cond_broadcast(&n.tick) + sync.mutex_unlock(&n.lock) + if due {node_lease(n,now)} +} + +@(private) +node_joined :: proc(n:^Node,h:^wire.Host,connection:wire.Connection) { + if exchange.peer_add(&n.x,h,connection)!=.None { + wire.host_disconnect(h,connection,.Capacity) + return + } + sync.mutex_guard(&n.lock) + for &p in n.peers { + if p.used {continue} + n.numbers+=1 + p={used=true,host=h,connection=connection,number=n.numbers} + n.connected+=1 + if h.desc.mode==.Client {n.client_joined=true} + node_event(n,{kind=.Joined,peer=p.number}) + return + } +} + +@(private) +node_left :: proc(n:^Node,h:^wire.Host,connection:wire.Connection,reason:wire.Disconnect_Reason) { + progress,known:=exchange.peer_progress(&n.x,h,connection) + exchange.peer_remove(&n.x,h,connection) + sync.mutex_guard(&n.lock) + if h.desc.mode==.Client {n.client_reason=reason} + if !known {return} + n.sent_gone+=progress.sent + n.received_gone+=progress.received + for &p in n.peers { + if !p.used || p.host!=h || p.connection!=connection {continue} + node_event(n,{kind=.Left,peer=p.number,reason=reason,sent=progress.sent,received=progress.received}) + p={} + n.connected-=1 + } +} + +@(private) +node_event :: proc(n:^Node,e:Event) { + if n.event_count==EVENTS { + n.event_head=(n.event_head+1)%EVENTS + n.event_count-=1 + } + n.events[(n.event_head+n.event_count)%EVENTS]=e + n.event_count+=1 +} + +// The connection of the client host of the node: to the host its join connects to. +@(private) +node_host_peer :: proc(n:^Node) -> (h:^wire.Host,connection:wire.Connection,ok:bool) { + sync.mutex_guard(&n.lock) + for p in n.peers { + if p.used && p.host.desc.mode==.Client {return p.host,p.connection,true} + } + return +} + +// The events since the last call, oldest first, into buffer. +node_events :: proc(n:^Node,buffer:[]Event) -> []Event { + sync.mutex_guard(&n.lock) + count:=min(n.event_count,len(buffer)) + for k in 0.. (t:Node_Totals) { + sync.mutex_guard(&n.lock) + t.connected,t.sent,t.received=n.connected,n.sent_gone,n.received_gone + for p in n.peers { + if !p.used {continue} + if progress,ok:=exchange.peer_progress(&n.x,p.host,p.connection);ok { + t.sent+=progress.sent + t.received+=progress.received + t.send_rate+=progress.send_rate + t.receive_rate+=progress.receive_rate + t.refused+=progress.refused + if progress.bad {t.bad+=1} + } + } + return +} + +// The clock of the node: that of its last step. +node_now :: proc(n:^Node) -> f64 { + sync.mutex_guard(&n.lock) + return n.now +} + +// Waits until the node stepped to at least now+seconds, and at least once; +// false once it stopped. +node_wait :: proc(n:^Node,seconds:f64) -> bool { + sync.mutex_guard(&n.lock) + until:=n.now+seconds + n.waiting+=1 + for (!n.stepped || n.now bool { + sync.mutex_guard(&n.lock) + return n.waiting>0 +} + +// Adds addresses to what the lease of the node holds. +node_serve :: proc(n:^Node,addresses:..store.Hash) { + sync.mutex_guard(&n.lock) + append(&n.served,..addresses) + n.lease_due=true +} + +@(private) +node_lease :: proc(n:^Node,now:f64) { + wants:[WANTS]store.Hash + wanted:=exchange.want_addresses(&n.x,wants[:]) + sync.mutex_lock(&n.lock) + live:=make([]store.Hash,len(n.served)+len(wanted),context.temp_allocator) + copy(live,n.served[:]) + copy(live[len(n.served):],wanted) + n.lease_at,n.lease_due=now,false + sync.mutex_unlock(&n.lock) + if len(live)>store.LEASE_MAX_HASHES {live=live[:store.LEASE_MAX_HASHES]} + if n.leased { + store.lease_update(&n.lease,live) + return + } + random:[8]u8 + crypto.rand_bytes(random[:]) + name:=fmt.tprintf("share-%x",random[:]) + // One store per value until odin-lang/Odin#7937 (see ODIN_WORKAROUNDS). + if lease,error:=store.lease_open(n.s,name,live);error==.None { + n.lease=lease + n.leased=true + } +} + +// The source a refresh of the library asks: refs of the peers with their +// objects, and for a build every object it lists, made complete in the store +// before the ref is set there. The names of one call are asked together, and +// the objects of their builds fetched together. For a thread other than the pump's. +node_source :: proc(n:^Node) -> cache.Source { + return {fetch=node_fetch,user=n} +} + +@(private) +node_fetch :: proc(user:rawptr,into:^store.Store,namespace:string,names:[]string,given:[]bool) { + n:=(^Node)(user) + if into!=n.s || len(names)==0 {return} + addresses:=make([]store.Hash,len(names),context.temp_allocator) + refs_fetch(&n.x,namespace,names,addresses) + if namespace==asset.REF_IMPORTS || namespace==asset.REF_IMPORTS_FAST { + objects:=make([dynamic]store.Hash,context.temp_allocator) + for &address in addresses { + if address=={} || !asset.object_whole(into,address) {continue} + b,build_error:=asset.build_get(into,address,context.temp_allocator) + if build_error!=.None { + address={} + continue + } + for o in b.outputs { + append(&objects,o.object) + if o.thumbnail!={} {append(&objects,o.thumbnail)} + } + } + objects_fetch(&n.x,objects[:]) + for &address in addresses { + if address=={} {continue} + b,_:=asset.build_get(into,address,context.temp_allocator) + for o in b.outputs { + if !asset.object_whole(into,o.object) || o.thumbnail!={} && !asset.object_whole(into,o.thumbnail) {address={}} + } + } + } + for address,k in addresses { + given[k]=address!={} && asset.object_whole(into,address) && store.ref_set(into,namespace,names[k],address)==.None + } +} + +// The addresses peers name with namespace and names, each with its object +// complete, or {} for a name none names. A name or an object that none has makes +// the batch Missing, which says nothing of the others: the batch is asked +// again by halves, so m missing of n cost about m log n round trips, not n. +@(private) +refs_fetch :: proc(x:^exchange.Exchange,namespace:string,names:[]string,addresses:[]store.Hash) { + error:=exchange.fetch_refs(x,namespace,names,addresses) + if error!=.Missing || len(names)==1 {return} + half:=len(names)/2 + refs_fetch(x,namespace,names[:half],addresses[:half]) + refs_fetch(x,namespace,names[half:],addresses[half:]) +} + +// The objects of addresses fetched together, by halves when some peer lacks one. +@(private) +objects_fetch :: proc(x:^exchange.Exchange,addresses:[]store.Hash) { + if len(addresses)==0 {return} + error:=exchange.fetch_all(x,addresses) + if error!=.Missing || len(addresses)==1 {return} + half:=len(addresses)/2 + objects_fetch(x,addresses[:half]) + objects_fetch(x,addresses[half:]) +} + +// The name of the snapshot a share offers under the secret of its invitation: +// a hash of the secret, so the names peers list reveal no secret. +offer_name :: proc(secret:u64,buffer:^[64]u8) -> string { + bytes:[8]u8 + for k in 0..<8 {bytes[k]=u8(secret>>(8*uint(k)))} + return store.hex(store.address_of_bytes(bytes[:]),buffer) +} diff --git a/cook/share/share_test.odin b/cook/share/share_test.odin new file mode 100644 index 000000000..f7506cbc3 --- /dev/null +++ b/cook/share/share_test.odin @@ -0,0 +1,1321 @@ +package share + +import "base:runtime" +import "core:fmt" +import "core:mem" +import "core:os" +import "core:strings" +import "core:sync" +import "core:testing" +import "core:thread" +import "core:time" +import "mirai:asset" +import "mirai:cook/import/sources" +import "mirai:exchange" +import "mirai:platform" +import "mirai:project" +import "mirai:std/base" +import "mirai:store" +import wire "mirai:net" + +// Peers of one simulated network, each with a store, a node, a server host +// others join and a client host it joins with, stepped together at 1 ms. +@(private="file") +Test_Peer :: struct { + s:store.Store, + n:Node, + server,client:^wire.Host, + endpoint:int, // of its server + key:wire.Invite_Key, // of the invitation of its server + pumped:bool, + mute:bool, // its hosts keep their connections, and its node reads and answers nothing +} + +@(private="file") +Test_Net :: struct { + sim:wire.Sim_Network, + now:f64, + peers:[3]Test_Peer, + count:int, + endpoints:int, + folder:string, +} + +@(private="file") +test_net :: proc(t:^testing.T,w:^Test_Net,name:string,peers:int) -> bool { + w.folder=base.test_folder(t,name) + if w.folder=="" {return false} + wire.sim_network_init(&w.sim,{seed=11,endpoints=8,packets=1<<16}) + w.count=peers + for k in 0.. ^wire.Host { + h:=new(wire.Host) + at:=w.endpoints + w.endpoints+=1 + chans:=channels() + wire.host_init(h,{mode=mode,transport=wire.sim_endpoint_transport(&w.sim,at),channels=chans[:],protocol_id=PROTOCOL_ID,version=VERSION,invite=key,seed=u64(200+at),max_connections=PEERS-1}) + if endpoint!=nil {endpoint^=at} + return h +} + +// Peer k joins the server of peer from with a client host. +@(private="file") +test_connect :: proc(w:^Test_Net,k,from:int) { + p:=&w.peers[k] + p.client=test_host(w,.Client,{public=w.peers[from].key.public,secret=w.peers[from].key.secret}) + node_host(&p.n,p.client) + wire.host_connect(p.client,wire.address_memory(w.peers[from].endpoint)) +} + +@(private="file") +test_net_destroy :: proc(w:^Test_Net) { + for &p in w.peers[:w.count] { + node_destroy(&p.n) + for h in ([2]^wire.Host{p.server,p.client}) { + if h!=nil {wire.host_destroy(h);free(h)} + } + store.close(&p.s) + } + wire.sim_network_destroy(&w.sim) +} + +@(private="file") +test_step :: proc(w:^Test_Net) { + w.now+=0.001 + wire.sim_network_advance(&w.sim,w.now) + for &p in w.peers[:w.count] { + if !p.pumped {continue} + if p.mute { + for h in ([2]^wire.Host{p.server,p.client}) { + if h==nil {continue} + wire.host_poll(h) + wire.host_update(h,w.now) + } + continue + } + node_pump(&p.n,w.now) + for h in ([2]^wire.Host{p.server,p.client}) { + if h!=nil {wire.host_update(h,w.now)} + } + } +} + +@(private="file") +Test_Join :: struct { + desc:Join_Desc, + report:Join_Report, + error:string, + done:bool, + allocator:mem.Allocator, +} + +// An arena of the heap that lives until the test ends: what a join gives back. +@(private="file") +test_arena :: proc(t:^testing.T) -> mem.Allocator { + arena:=new(mem.Dynamic_Arena,runtime.heap_allocator()) + mem.dynamic_arena_init(arena,runtime.heap_allocator(),runtime.heap_allocator()) + testing.cleanup(t,proc(data:rawptr) { + arena:=(^mem.Dynamic_Arena)(data) + mem.dynamic_arena_destroy(arena) + free(arena,runtime.heap_allocator()) + },arena) + return base.dynamic_arena_allocator(arena) +} + +// Runs a join of peer k on a thread while the network steps, at most 600 +// virtual seconds of its waits. stop_bytes: in the step its node has +// received that many chunk bytes, peer k leaves (its client disconnects and +// it is pumped no more) and its join is stopped, so nothing arrives after +// the stop whenever the thread of the join notices it. +@(private="file") +test_join :: proc(t:^testing.T,w:^Test_Net,k:int,desc:Join_Desc,stop_bytes:u64=0) -> (report:Join_Report,error:string) { + run:=Test_Join{desc=desc} + run.desc.node=&w.peers[k].n + run.allocator=test_arena(t) + run.desc.pace=1000 + th:=thread.create_and_start_with_data(&run,proc(data:rawptr) { + r:=(^Test_Join)(data) + r.report,r.error=join(r.desc,r.allocator) + free_all(context.temp_allocator) + sync.atomic_store(&r.done,true) + },init_context=context) + for steps:=0;!sync.atomic_load(&run.done); { + // The clock moves only while the join waits for it: its work between + // two waits takes no virtual time, however long it takes for real. + if w.peers[k].pumped && !node_waiting(run.desc.node) { + thread.yield() + continue + } + test_step(w) + steps+=1 + if steps==600_000 { + testing.expect(t,false,"the join did not end in 600 virtual seconds") + node_stop(run.desc.node) + } + if stop_bytes>0 && w.peers[k].pumped && node_totals(run.desc.node).received>=stop_bytes { + // As a process that ends: its client says goodbye and nothing more is read. + w.peers[k].pumped=false + node_stop(run.desc.node) + if c:=w.peers[k].client;c!=nil { + wire.host_disconnect(c,0,.Closed) + wire.host_poll(c) + wire.host_update(c,w.now) + } + } + } + thread.join(th) + thread.destroy(th) + return run.report,run.error +} + +@(private="file") +file_put :: proc(root,path:string,data:[]u8,executable:=false) { + full:=fmt.tprintf("%s/%s",root,path) + os.make_directory_all(os.dir(full)) + _=os.write_entire_file(full,data) + if executable {os.change_mode(full,os.Permissions_Default_File+os.Permissions_Execute_All)} +} + +// Gives the file at path a time an hour ago, as a file written long before it is read. +@(private="file") +settle :: proc(root,path:string) { + old:=time.time_add(time.now(),-time.Hour) + os.change_times(fmt.tprintf("%s/%s",root,path),old,old) +} + +@(private="file") +read_text :: proc(root,path:string) -> string { + data,error:=os.read_entire_file(fmt.tprintf("%s/%s",root,path),context.temp_allocator) + return error==nil?string(data):"(none)" +} + +// The address of the object of data, as a file of those bytes has. +@(private="file") +bytes_address :: proc(data:[]u8) -> store.Hash { + w:store.Object_Writer + store.object_writer_init(&w,{procedure=proc(data:rawptr,kind:store.Chunk_Kind,address:store.Hash,bytes:[]u8) -> store.Error {return .None}}) + defer store.object_writer_destroy(&w) + store.object_write(&w,data) + root,_,_:=store.object_finish(&w) + return root +} + +@(private="file") +noise :: proc(size:int,seed:u64) -> []u8 { + data:=make([]u8,size,context.temp_allocator) + h:=base.Hostile{state=seed} + for &b in data {b=u8(base.hostile_next(&h))} + return data +} + +// The files of root that are its own (project.own_path), by path, with their bytes and executable bit. +@(private="file") +own_files :: proc(root:string) -> map[string]string { + files:=make(map[string]string,context.temp_allocator) + todo:=make([dynamic]string,context.temp_allocator) + append(&todo,"") + for next:=0;next bool { + fa,fb:=own_files(a),own_files(b) + ok:=testing.expectf(t,len(fa)==len(fb),"%d files and %d",len(fa),len(fb),loc=loc) + for path,content in fa { + other,found:=fb[path] + ok&=testing.expectf(t,found && other==content,"%s is the same in both",path,loc=loc) + } + return ok +} + +@(private="file") +wav :: proc(samples:int,seed:u64) -> []u8 { + file:=make([]u8,44+samples*2,context.temp_allocator) + put :: proc(file:[]u8,at:int,value:u32,size:int) {for i in 0..>uint(8*i))}} + copy(file,"RIFF");put(file,4,u32(len(file)-8),4);copy(file[8:],"WAVEfmt ") + put(file,16,16,4);put(file,20,1,2);put(file,22,1,2);put(file,24,48000,4);put(file,28,96000,4) + put(file,32,2,2);put(file,34,16,2) + copy(file[36:],"data");put(file,40,u32(samples*2),4) + copy(file[44:],noise(samples*2,seed)) + return file +} + +// A project at /host with files of its own, some that are not, and a +// package whose source then leaves the machine; its snapshot offered by peer 0. +@(private="file") +host_project :: proc(t:^testing.T,w:^Test_Net) -> (root:string,ok:bool) { + root=fmt.tprintf("%s/host",w.folder) + file_put(root,"project.mirai",transmute([]u8)string("{}\n")) + file_put(root,"game/game.odin",transmute([]u8)string("package game\n")) + file_put(root,"tools/run.sh",transmute([]u8)string("#!/bin/sh\n"),executable=true) + file_put(root,"assets/big.bin",noise(300<<10,1)) + file_put(root,"assets/empty.txt",nil) + file_put(root,"assets/deep/a/b/c.txt",transmute([]u8)string("c")) + file_put(root,".gitignore",transmute([]u8)string(".mirai/\n")) + file_put(root,".mirai/cache/registry/x",transmute([]u8)string("machine")) + file_put(root,".git/HEAD",transmute([]u8)string("ref: main")) + sources_folder:=fmt.tprintf("%s/sources",w.folder) + file_put(sources_folder,"beep.wav",wav(4800,2)) + options:=sources.Library_Options{store=&w.peers[0].s,root=root} + report,error:=sources.ingest(fmt.tprintf("%s/beep.wav",sources_folder),options,allocator=context.temp_allocator) + if !testing.expectf(t,error.code==.None,"%s",error.message) {return} + added:=sources.project_add(root,report.manifest,"sounds",nil,options,context.temp_allocator) + if !testing.expectf(t,len(added.failures)==0 && len(added.missing)==0,"%v",added) {return} + // Its source is on no peer from now on. + testing.expect(t,os.rename(sources_folder,fmt.tprintf("%s/gone",w.folder))==nil) + return root,host_offer(t,w,root) +} + +@(private="file") +host_offer :: proc(t:^testing.T,w:^Test_Net,root:string) -> bool { + s:=&w.peers[0].s + folders:=make([dynamic]store.Hash,context.temp_allocator) + snap,address,_,error:=snapshot_take(s,root,"game",&folders,allocator=context.temp_allocator) + if !testing.expectf(t,error=="","%s",error) {return false} + digits:[64]u8 + offer_set(s,root,offer_name(w.peers[0].key.secret,&digits),address) + node_serve(&w.peers[0].n,..folders[:]) + for e in snap.entries {node_serve(&w.peers[0].n,e.address)} + return true +} + +@(private="file") +join_desc :: proc(w:^Test_Net,from:int,folder:string) -> Join_Desc { + digits:=new([64]u8,context.temp_allocator) + return {offer=offer_name(w.peers[from].key.secret,digits),folder=folder,parent=w.folder} +} + +@(test) +the_snapshot_leaves_out_what_the_layout_says_and_round_trips :: proc(t:^testing.T) { + folder:=base.test_folder(t,"share-snapshot") + if folder=="" {return} + root:=fmt.tprintf("%s/game",folder) + file_put(root,"project.mirai",transmute([]u8)string("{}\n")) + file_put(root,"tools/run.sh",transmute([]u8)string("#!/bin/sh\n"),executable=true) + file_put(root,"assets/b.txt",transmute([]u8)string("b")) + file_put(root,"assets/empty.txt",nil) + file_put(root,".gitignore",transmute([]u8)string(".mirai/\n")) + file_put(root,".mirai/build/out.o",transmute([]u8)string("built")) + file_put(root,".git/HEAD",transmute([]u8)string("ref: main")) + for path in ([]string{"project.mirai","tools/run.sh","assets/b.txt","assets/empty.txt",".gitignore"}) {settle(root,path)} + s,open_error:=store.open(fmt.tprintf("%s/store",folder)) + if !testing.expect_value(t,open_error,store.Error.None) {return} + defer store.close(&s) + snap,address,report,error:=snapshot_take(&s,root,"game",nil,allocator=context.temp_allocator) + if !testing.expectf(t,error=="","%s",error) {return} + paths:=make([dynamic]string,context.temp_allocator) + for e in snap.entries {append(&paths,e.path)} + testing.expectf(t,fmt.tprint(paths[:])==`[".gitignore", "assets/b.txt", "assets/empty.txt", "project.mirai", "tools/run.sh"]`,"the files of its own, sorted: %v",paths[:]) + run,_:=snapshot_find(&snap,"tools/run.sh") + b,_:=snapshot_find(&snap,"assets/b.txt") + empty,_:=snapshot_find(&snap,"assets/empty.txt") + testing.expect(t,run.executable && !b.executable && b.size==1 && store.chunk_has(&s,b.address) && b.address==bytes_address(transmute([]u8)string("b"))) + testing.expect(t,empty.address==empty_address() && empty.address==bytes_address(nil)) + testing.expect_value(t,report.put,5) + + // The tree reads back as the files, and the flat base a joiner keeps too. + again,read_error:=snapshot_read(&s,address,allocator=context.temp_allocator) + testing.expect_value(t,read_error,Snapshot_Error.None) + testing.expect(t,again.name=="game" && len(again.entries)==len(snap.entries)) + for e,at in again.entries {testing.expectf(t,e==snap.entries[at],"%s round-trips",e.path)} + name,name_error:=snapshot_name(&s,address,context.temp_allocator) + testing.expect(t,name_error==.None && name=="game") + bytes,_:=snapshot_encode(snap,context.temp_allocator) + based,decode_error:=snapshot_decode(bytes,context.temp_allocator) + testing.expect_value(t,decode_error,Snapshot_Error.None) + for e,at in based.entries {testing.expectf(t,e==snap.entries[at],"%s round-trips in the base",e.path)} + for cut in 0.. store.Hash { + bytes:=make([dynamic]u8,context.temp_allocator) + for r in records { + r:=r + append(&bytes,r.kind,u8(len(r.name))) + append(&bytes,..transmute([]u8)r.name) + append(&bytes,..r.address[:]) + for k in 0..<8 {append(&bytes,u8(r.size>>(8*uint(k))))} + if r.kind==KIND_FOLDER {for k in 0..<4 {append(&bytes,u8(r.files>>(8*uint(k))))}} + } + meta:=fmt.tprintf("%s%c%s",magic,rune(len(name)),name) + p,_:=store.put_begin(s) + store.put_write(&p,bytes[:len(bytes)-cut]) + address,_,_:=store.put_finish(&p,transmute([]u8)meta) + store.commit(s) + return address +} + +@(test) +hostile_trees_are_refused_without_a_crash :: proc(t:^testing.T) { + folder:=base.test_folder(t,"share-hostile") + if folder=="" {return} + s,open_error:=store.open(fmt.tprintf("%s/store",folder)) + if !testing.expect_value(t,open_error,store.Error.None) {return} + defer store.close(&s) + put :: proc(s:^store.Store,text:string) -> store.Hash { + address,_:=store.put(s,transmute([]u8)text) + return address + } + x:=put(&s,"x") + empty:=empty_address() + inner:=raw_folder(&s,{{name="f",address=x,size=1}}) + Case :: struct {records:[]Raw,error:Snapshot_Error} + cases:=[?]Case{ + {{{name="..",address=x,size=1}},.Path}, + {{{name=".",address=x,size=1}},.Path}, + {{{name="a/b",address=x,size=1}},.Path}, + {{{name="a\\b",address=x,size=1}},.Path}, + {{{name="",address=x,size=1}},.Path}, + {{{name="nul",address=x,size=1}},.Path}, + {{{name="Com1.txt",address=x,size=1}},.Path}, + {{{name="name.",address=x,size=1}},.Path}, + {{{name="a:b",address=x,size=1}},.Path}, + {{{kind=KIND_FOLDER,name=".mirai",address=inner,size=1,files=1}},.Path}, + {{{kind=KIND_FOLDER,name=".git",address=inner,size=1,files=1}},.Path}, + {{{name="a",address=x,size=1},{name="a",address=x,size=1}},.Duplicate}, + {{{name="b",address=x,size=1},{name="a",address=x,size=1}},.Duplicate}, + {{{name="A.txt",address=x,size=1},{name="a.txt",address=x,size=1}},.Duplicate}, + {{{name="a",address=x,size=0}},.Size}, + {{{name="a",address=x,size=1<<63}},.Size}, + {{{name="a",address=x,size=max(u64)/2},{name="b",address=x,size=max(u64)/2}},.Size}, + {{{kind=KIND_FOLDER,name="a",address=inner,size=1,files=0}},.Size}, + {{{kind=KIND_FOLDER,name="a",address=inner,size=1,files=2}},.Size}, + {{{kind=KIND_FOLDER,name="a",address=inner,size=2,files=1}},.Size}, + {{{kind=3,name="a",address=x,size=1}},.Not_Snapshot}, + {{{kind=KIND_FOLDER,name="a",address=x,size=1,files=1}},.Not_Snapshot}, + {{{kind=KIND_FOLDER,name="a",address=put(&s,"none"),size=1,files=1}},.Not_Snapshot}, + {{{name="a",address=empty,size=0},{name="b",address=x,size=1},{kind=KIND_FOLDER,name="c",address=inner,size=1,files=1}},.None}, + } + for c in cases { + root:=raw_folder(&s,c.records,"game") + _,error:=snapshot_read(&s,root,allocator=context.temp_allocator) + testing.expectf(t,error==c.error,"%v: %v, not %v",c.records,error,c.error) + free_all(context.temp_allocator) + } + read :: proc(s:^store.Store,root:store.Hash) -> Snapshot_Error { + _,error:=snapshot_read(s,root,allocator=context.temp_allocator) + return error + } + good:=[]Raw{{name="a",address=x,size=1},{kind=KIND_FOLDER,name="b",address=inner,size=1,files=1}} + // The meta: a root needs a folder's name, a folder below none; another magic is not a snapshot. + testing.expect_value(t,read(&s,raw_folder(&s,good,"..")),Snapshot_Error.Name) + testing.expect_value(t,read(&s,raw_folder(&s,good,"")),Snapshot_Error.Name) + testing.expect_value(t,read(&s,raw_folder(&s,good,"game",magic="mirai snapshot 1\n")),Snapshot_Error.Not_Snapshot) + named:=raw_folder(&s,{{name="f",address=x,size=1}},"inside") + testing.expect_value(t,read(&s,raw_folder(&s,{{kind=KIND_FOLDER,name="b",address=named,size=1,files=1}},"game")),Snapshot_Error.Not_Snapshot) + for cut in 1..<45 {testing.expectf(t,read(&s,raw_folder(&s,good,"game",cut=cut))==.Truncated,"a listing cut by %d is refused",cut)} + // What is not in the store is unavailable, never a crash. + testing.expect_value(t,read(&s,raw_folder(&s,{{kind=KIND_FOLDER,name="b",address=store.address_of_bytes(transmute([]u8)string("?")),size=1,files=1}},"game")),Snapshot_Error.Unavailable) + // Folders nested deeper than the bound. + deep:=inner + for _ in 0..0,"%v",listed) + for l in listed.listed { + a:asset.Asset + if testing.expectf(t,asset.asset_open_object(&a,&w.peers[1].s,l.object,context.temp_allocator)==.None,"%s loads on the joiner",l.path) { + testing.expect_value(t,a.blob.header.id,l.id) + asset.asset_close(&a) + } + } +} + +@(test) +a_second_join_updates_the_folder_by_three_ways :: proc(t:^testing.T) { + w:Test_Net + if !test_net(t,&w,"share-update",2) {return} + defer test_net_destroy(&w) + root:=fmt.tprintf("%s/host",w.folder) + for name in ([?]string{"host_changes","host_removes","both_change","host_removes_joiner_changes","joiner_removes_host_changes","joiner_changes","same","Case_only","Case_changed"}) { + file_put(root,fmt.tprintf("files/%s.txt",name),transmute([]u8)fmt.tprintf("%s 1",name)) + } + file_put(root,"Folder/inside.txt",transmute([]u8)string("inside 1")) + if !host_offer(t,&w,root) {return} + test_connect(&w,1,0) + joined:=fmt.tprintf("%s/joined",w.folder) + if _,error:=test_join(t,&w,1,join_desc(&w,0,joined));!testing.expectf(t,error=="","%s",error) {return} + + file_put(root,"files/host_changes.txt",transmute([]u8)string("host 2")) + os.remove(fmt.tprintf("%s/files/host_removes.txt",root)) + file_put(root,"files/added/new.txt",transmute([]u8)string("new")) + file_put(root,"files/both_change.txt",transmute([]u8)string("host 2")) + os.remove(fmt.tprintf("%s/files/host_removes_joiner_changes.txt",root)) + file_put(root,"files/joiner_removes_host_changes.txt",transmute([]u8)string("host 2")) + file_put(joined,"files/both_change.txt",transmute([]u8)string("joiner 2")) + file_put(joined,"files/host_removes_joiner_changes.txt",transmute([]u8)string("joiner 2")) + os.remove(fmt.tprintf("%s/files/joiner_removes_host_changes.txt",joined)) + file_put(joined,"files/joiner_changes.txt",transmute([]u8)string("joiner 2")) + // Renames that change only the case: of a file, of a file with its bytes, of a folder. + os.rename(fmt.tprintf("%s/files/Case_only.txt",root),fmt.tprintf("%s/files/case_only.txt",root)) + os.rename(fmt.tprintf("%s/files/Case_changed.txt",root),fmt.tprintf("%s/files/case_changed.txt",root)) + file_put(root,"files/case_changed.txt",transmute([]u8)string("changed 2")) + os.rename(fmt.tprintf("%s/Folder",root),fmt.tprintf("%s/folder",root)) + if !host_offer(t,&w,root) {return} + report,error:=test_join(t,&w,1,join_desc(&w,0,joined)) + if !testing.expectf(t,error=="","%s",error) {return} + read :: proc(root,name:string) -> string { + data,error:=os.read_entire_file(fmt.tprintf("%s/files/%s",root,name),context.temp_allocator) + return error==nil?string(data):"(none)" + } + testing.expect_value(t,read(joined,"host_changes.txt"),"host 2") + testing.expect_value(t,read(joined,"host_removes.txt"),"(none)") + testing.expect_value(t,read(joined,"added/new.txt"),"new") + testing.expect_value(t,read(joined,"both_change.txt"),"joiner 2") + testing.expect_value(t,read(joined,"host_removes_joiner_changes.txt"),"joiner 2") + testing.expect_value(t,read(joined,"joiner_removes_host_changes.txt"),"(none)") + testing.expect_value(t,read(joined,"joiner_changes.txt"),"joiner 2") + testing.expect_value(t,read(joined,"same.txt"),"same 1") + // Where case folds, a rename by case is one file renamed in place; elsewhere one removed and one written. + folds:=folds_case(joined) + testing.expectf(t,report.written==(folds?3:5) && report.removed==(folds?1:4),"the host's changes and its new file are written, its removals applied: %d, %d",report.written,report.removed) + names :: proc(folder:string) -> string { + listed,_:=os.read_all_directory_by_path(folder,context.temp_allocator) + list:=make([dynamic]string,context.temp_allocator) + for info in listed {append(&list,info.name)} + return strings.join(list[:]," ",context.temp_allocator) + } + renamed :: proc(t:^testing.T,joined:string) { + testing.expect_value(t,read_text(joined,"files/case_only.txt"),"Case_only 1") + testing.expect_value(t,read_text(joined,"files/case_changed.txt"),"changed 2") + testing.expect_value(t,read_text(joined,"folder/inside.txt"),"inside 1") + listed:=fmt.tprintf("%s %s",names(fmt.tprintf("%s/files",joined)),names(joined)) + testing.expectf(t,strings.contains(listed,"case_only.txt") && strings.contains(listed,"case_changed.txt") && strings.contains(listed,"folder") && !strings.contains(listed,"Case_") && !strings.contains(listed,"Folder"),"the names are the host's: %s",listed) + } + renamed(t,joined) + kept:=strings.join(report.kept[:],"\n",context.temp_allocator) + testing.expectf(t,len(report.kept)==3 && strings.contains(kept,"both_change") && strings.contains(kept,"host_removes_joiner_changes") && strings.contains(kept,"joiner_removes_host_changes"),"what both changed is reported: %s",kept) + + // Once more with nothing changed on either side: nothing is written, and what was kept stays kept. + again,again_error:=test_join(t,&w,1,join_desc(&w,0,joined)) + testing.expect(t,again_error=="" && again.written==0 && again.removed==0) + renamed(t,joined) +} + +// A join does not overwrite what the joiner changes after its plan: a file +// made where the host adds one, a file changed that the host changed too. +@(test) +a_join_keeps_what_the_joiner_changes_while_it_runs :: proc(t:^testing.T) { + folder:=base.test_folder(t,"share-meanwhile") + if folder=="" {return} + s,open_error:=store.open(fmt.tprintf("%s/store",folder)) + if !testing.expect_value(t,open_error,store.Error.None) {return} + defer store.close(&s) + n:Node + n.s=&s + d:=Join_Desc{node=&n} + joined:=fmt.tprintf("%s/joined",folder) + bytes :: proc(text:string) -> []u8 {return transmute([]u8)text} + entries:=make([]Entry,3,context.temp_allocator) + for name,k in ([3]string{"a","b","c"}) { + address,_:=store.put(&s,bytes(fmt.tprintf("host %s",name))) + entries[k]={path=fmt.tprintf("%s.txt",name),address=address,size=6} + } + store.commit(&s) + snap:=Snapshot{name="game",entries=entries} + file_put(joined,"b.txt",bytes("base b")) + file_put(joined,"c.txt",bytes("base c")) + plan:Plan + plan.write=make([dynamic]Write,context.temp_allocator) + append(&plan.write,Write{at=0},Write{at=1,present=true,base=bytes_address(bytes("base b"))},Write{at=2,present=true,base=bytes_address(bytes("base c"))}) + // After the plan, before the objects arrive. + file_put(joined,"a.txt",bytes("mine a")) + file_put(joined,"b.txt",bytes("mine b")) + index:=make(Index,context.temp_allocator) + report:=Join_Report{kept=make([dynamic]string,context.temp_allocator),failed=make([dynamic]string,context.temp_allocator)} + join_write(&d,&snap,&plan,{0,1,2},&index,joined,&report) + testing.expect_value(t,read_text(joined,"a.txt"),"mine a") + testing.expect_value(t,read_text(joined,"b.txt"),"mine b") + testing.expect_value(t,read_text(joined,"c.txt"),"host c") + testing.expectf(t,report.written==1 && len(report.kept)==2 && len(report.failed)==0,"%v",report) + for w in plan.write {testing.expect(t,w.done,"what is kept on purpose is done")} + + // Edited while its replacement is copied, after the check before the copy. + file_put(joined,"c.txt",bytes("base c")) + clear(&plan.write) + append(&plan.write,Write{at=2,present=true,base=bytes_address(bytes("base c"))}) + Edit :: struct {joined:string} + edit:=Edit{joined} + hooks:=Join_Hooks{data=&edit,saved=proc(data:rawptr,path:string) { + file_put((^Edit)(data).joined,path,transmute([]u8)string("mine c")) + }} + n.hooks=&hooks + late:=Join_Report{kept=make([dynamic]string,context.temp_allocator),failed=make([dynamic]string,context.temp_allocator)} + join_write(&d,&snap,&plan,{0},&index,joined,&late) + n.hooks=nil + testing.expect_value(t,read_text(joined,"c.txt"),"mine c") + testing.expectf(t,late.written==0 && len(late.kept)==1 && plan.write[0].done,"%v",late) +} + +// A join never writes or removes through a link inside its folder: a file of +// the host's under a linked folder, a linked file it replaces and a file it +// removed under a linked folder are refused and reported by path, and what +// the links lead to stays as it was. +@(test) +a_join_never_follows_a_link_inside_its_folder :: proc(t:^testing.T) { + folder:=base.test_folder(t,"share-links") + if folder=="" {return} + s,open_error:=store.open(fmt.tprintf("%s/store",folder)) + if !testing.expect_value(t,open_error,store.Error.None) {return} + defer store.close(&s) + n:Node + n.s=&s + d:=Join_Desc{node=&n} + joined:=fmt.tprintf("%s/joined",folder) + outside:=fmt.tprintf("%s/outside",folder) + bytes :: proc(text:string) -> []u8 {return transmute([]u8)text} + file_put(outside,"old.txt",bytes("old")) + file_put(outside,"target.txt",bytes("target")) + file_put(joined,"plain.txt",bytes("plain 1")) + if !testing.expect(t,os.symlink(outside,fmt.tprintf("%s/linked",joined))==nil,"a link can be made here") {return} + os.symlink(fmt.tprintf("%s/target.txt",outside),fmt.tprintf("%s/file.txt",joined)) + entry :: proc(s:^store.Store,path,text:string) -> Entry { + address,_:=store.put(s,transmute([]u8)text) + return {path=path,address=address,size=i64(len(text))} + } + last_entries:=[]Entry{entry(&s,"file.txt","target"),entry(&s,"linked/old.txt","old"),entry(&s,"plain.txt","plain 1")} + snap_entries:=[]Entry{entry(&s,"file.txt","host"),entry(&s,"linked/deep/new.txt","host"),entry(&s,"plain.txt","plain 2")} + store.commit(&s) + last:=Snapshot{name="game",entries=last_entries} + snap:=Snapshot{name="game",entries=snap_entries} + index:=make(Index,context.temp_allocator) + report:=Join_Report{kept=make([dynamic]string,context.temp_allocator),failed=make([dynamic]string,context.temp_allocator)} + plan:=join_plan(&snap,&last,&index,joined,&report,context.temp_allocator) + list:=make([dynamic]int,context.temp_allocator) + for _,k in plan.write {append(&list,k)} + join_write(&d,&snap,&plan,list[:],&index,joined,&report) + join_remove(&last,&plan,&index,joined,&report) + testing.expect_value(t,read_text(outside,"old.txt"),"old") + testing.expect_value(t,read_text(outside,"target.txt"),"target") + testing.expect_value(t,read_text(outside,"deep/new.txt"),"(none)") + testing.expect_value(t,read_text(joined,"plain.txt"),"plain 2") + listed:=strings.join(report.failed[:],"\n",context.temp_allocator) + testing.expectf(t,len(report.failed)==3 && strings.contains(listed,"file.txt:") && strings.contains(listed,"linked/deep/new.txt:") && strings.contains(listed,"linked/old.txt:") && strings.contains(listed,"is a link"),"each refused by path: %s",listed) + testing.expect_value(t,report.written,1) + testing.expect_value(t,report.removed,0) +} + +// A file that cannot be written does not stop the join: a read-only file, a +// read-only file the host removed and a file in a read-only folder are listed +// with how to fix them, the rest is written, and once fixed the same join +// completes them. +@(test) +a_join_writes_what_it_can_and_running_it_again_completes_it :: proc(t:^testing.T) { + w:Test_Net + if !test_net(t,&w,"share-incomplete",2) {return} + defer test_net_destroy(&w) + root:=fmt.tprintf("%s/host",w.folder) + for name in ([?]string{"files/a.txt","files/locked.txt","files/gone.txt","shut/x.txt"}) {file_put(root,name,transmute([]u8)fmt.tprintf("%s 1",name))} + if !host_offer(t,&w,root) {return} + test_connect(&w,1,0) + joined:=fmt.tprintf("%s/joined",w.folder) + if _,error:=test_join(t,&w,1,join_desc(&w,0,joined));!testing.expectf(t,error=="","%s",error) {return} + + file_put(root,"files/a.txt",transmute([]u8)string("a 2")) + file_put(root,"files/locked.txt",transmute([]u8)string("locked 2")) + os.remove(fmt.tprintf("%s/files/gone.txt",root)) + file_put(root,"shut/new.txt",transmute([]u8)string("new")) + if !host_offer(t,&w,root) {return} + read_only:=os.Permissions_Read_All + os.change_mode(fmt.tprintf("%s/files/locked.txt",joined),read_only) + os.change_mode(fmt.tprintf("%s/files/gone.txt",joined),read_only) + os.change_mode(fmt.tprintf("%s/shut",joined),read_only+os.Permissions_Execute_All) + fix :: proc(joined:string) { + os.change_mode(fmt.tprintf("%s/files/locked.txt",joined),os.Permissions_Default_File) + os.change_mode(fmt.tprintf("%s/files/gone.txt",joined),os.Permissions_Default_File) + os.change_mode(fmt.tprintf("%s/shut",joined),os.Permissions_Default_Directory) + } + defer fix(joined) + report,error:=test_join(t,&w,1,join_desc(&w,0,joined)) + if !testing.expectf(t,error=="","%s",error) {return} + testing.expect_value(t,read_text(joined,"files/a.txt"),"a 2") + testing.expect_value(t,read_text(joined,"files/locked.txt"),"files/locked.txt 1") + testing.expect_value(t,read_text(joined,"files/gone.txt"),"files/gone.txt 1") + testing.expect_value(t,read_text(joined,"shut/new.txt"),"(none)") + listed:=strings.join(report.failed[:],"\n",context.temp_allocator) + testing.expectf(t,len(report.failed)==3 && strings.contains(listed,"locked.txt is read-only") && strings.contains(listed,"gone.txt is read-only") && strings.contains(listed,"shut is read-only"),"each failure by path, with how to fix it: %s",listed) + testing.expect(t,report.written==1 && report.removed==0) + + fix(joined) + again,again_error:=test_join(t,&w,1,join_desc(&w,0,joined)) + if !testing.expectf(t,again_error=="","%s",again_error) {return} + testing.expectf(t,len(again.failed)==0 && len(again.kept)==0 && again.written==2 && again.removed==1,"%v",again) + same_files(t,root,joined) +} + +@(test) +a_stopped_join_resumes_without_fetching_again_what_it_has :: proc(t:^testing.T) { + w:Test_Net + if !test_net(t,&w,"share-resume",2) {return} + defer test_net_destroy(&w) + root:=fmt.tprintf("%s/host",w.folder) + for k in 0..<8 {file_put(root,fmt.tprintf("data/%d.bin",k),noise(512<<10,u64(10+k)))} + if !host_offer(t,&w,root) {return} + test_connect(&w,1,0) + joined:=fmt.tprintf("%s/joined",w.folder) + TOTAL :: 8*(512<<10) + _,stopped:=test_join(t,&w,1,join_desc(&w,0,joined),TOTAL/2) + testing.expect(t,strings.contains(stopped,"stopped"),"the join was stopped") + first:=node_totals(&w.peers[1].n).received + testing.expectf(t,first>=TOTAL/2 && first0,"the resume fetched the rest") + same_files(t,root,joined) +} + +@(test) +a_third_peer_joins_from_the_second :: proc(t:^testing.T) { + w:Test_Net + if !test_net(t,&w,"share-third",3) {return} + defer test_net_destroy(&w) + root,ok:=host_project(t,&w) + if !ok {return} + test_connect(&w,1,0) + second:=fmt.tprintf("%s/second",w.folder) + desc:=join_desc(&w,0,second) + digits:[64]u8 + desc.own_offer=offer_name(w.peers[1].key.secret,&digits) + if _,error:=test_join(t,&w,1,desc);!testing.expectf(t,error=="","%s",error) {return} + // The host is gone: all comes from the second. + w.peers[0].pumped=false + test_connect(&w,2,1) + third:=fmt.tprintf("%s/third",w.folder) + report,error:=test_join(t,&w,2,join_desc(&w,1,third)) + if !testing.expectf(t,error=="","%s",error) {return} + same_files(t,root,third) + testing.expectf(t,len(report.packages.missing)==0 && report.packages.fetched==1,"%v",report.packages) +} + +// A peer that joined from the joiner and names another snapshot under the +// host's invitation, answering before the host does, is not asked: the +// snapshot of an invitation comes only from the host it connects to. +@(test) +a_downstream_peer_cannot_substitute_the_snapshot :: proc(t:^testing.T) { + w:Test_Net + if !test_net(t,&w,"share-substitute",3) {return} + defer test_net_destroy(&w) + root:=fmt.tprintf("%s/host",w.folder) + file_put(root,"project.mirai",transmute([]u8)string("{}\n")) + file_put(root,"assets/real.txt",transmute([]u8)string("real")) + if !host_offer(t,&w,root) {return} + fake:=fmt.tprintf("%s/fake",w.folder) + file_put(fake,"project.mirai",transmute([]u8)string("{}\n")) + file_put(fake,"assets/fake.txt",transmute([]u8)string("fake")) + folders:=make([dynamic]store.Hash,context.temp_allocator) + snap,address,_,error:=snapshot_take(&w.peers[2].s,fake,"game",&folders,allocator=context.temp_allocator) + if !testing.expectf(t,error=="","%s",error) {return} + digits:[64]u8 + store.ref_set(&w.peers[2].s,NAMESPACE,offer_name(w.peers[0].key.secret,&digits),address) + node_serve(&w.peers[2].n,..folders[:]) + for e in snap.entries {node_serve(&w.peers[2].n,e.address)} + // The downstream peer is connected to the joiner first, and the host answers late. + test_connect(&w,2,1) + wire.sim_endpoint_conditions(&w.sim,w.peers[0].endpoint,{latency=0.2}) + test_connect(&w,1,0) + joined:=fmt.tprintf("%s/joined",w.folder) + if _,join_error:=test_join(t,&w,1,join_desc(&w,0,joined));!testing.expectf(t,join_error=="","%s",join_error) {return} + same_files(t,root,joined) +} + +// A host that stays connected and answers nothing, and one whose snapshot names +// a file it does not have, end the join within a bounded time: the first with +// why, the second with the file reported by path. +@(test) +a_host_that_withholds_ends_the_join_in_bounded_time :: proc(t:^testing.T) { + w:Test_Net + if !test_net(t,&w,"share-withholds",2) {return} + defer test_net_destroy(&w) + s:=&w.peers[0].s + here,_:=store.put(s,transmute([]u8)string("here")) + gone:=bytes_address(noise(1000,9)) + withheld:=Snapshot{name="game",entries={{path="assets/gone.bin",address=gone,size=1000},{path="here.txt",address=here,size=4}}} + snapshot,_,_:=snapshot_put(s,&withheld) + digits:[64]u8 + store.ref_set(s,NAMESPACE,offer_name(w.peers[0].key.secret,&digits),snapshot) + test_connect(&w,1,0) + started:=w.now + report,error:=test_join(t,&w,1,join_desc(&w,0,fmt.tprintf("%s/joined",w.folder))) + testing.expectf(t,error=="" && len(report.failed)==1 && strings.contains(report.failed[0],"assets/gone.bin") && report.written==1,"%s %v",error,report.failed) + testing.expectf(t,w.now-started<60,"the file is reported in bounded time: %.1f s",w.now-started) + + w.peers[0].mute=true + started=w.now + _,silent:=test_join(t,&w,1,join_desc(&w,0,fmt.tprintf("%s/other",w.folder))) + testing.expectf(t,silent!="" && w.now-started<60,"a silent host ends the join in bounded time: %.1f s, %s",w.now-started,silent) +} + +// A host whose snapshot lies about the size of a file, and one whose root is +// larger than a folder can be, are refused without writing anything. +@(test) +a_lying_size_and_an_oversized_folder_are_refused :: proc(t:^testing.T) { + w:Test_Net + if !test_net(t,&w,"share-lies",2) {return} + defer test_net_destroy(&w) + s:=&w.peers[0].s + address,_:=store.put(s,noise(200<<10,5)) + lying:=Snapshot{name="game",entries={{path="a.bin",address=address,size=100}}} + snapshot,_,_:=snapshot_put(s,&lying) + digits:[64]u8 + store.ref_set(s,NAMESPACE,offer_name(w.peers[0].key.secret,&digits),snapshot) + test_connect(&w,1,0) + joined:=fmt.tprintf("%s/joined",w.folder) + lied,error:=test_join(t,&w,1,join_desc(&w,0,joined)) + testing.expectf(t,error=="" && len(lied.failed)==1 && strings.contains(lied.failed[0],"not honest"),"%s %v",error,lied.failed) + testing.expect(t,!os.exists(fmt.tprintf("%s/a.bin",joined))) + + huge:=make([]u8,LISTING_MAX+1,context.temp_allocator) + p,_:=store.put_begin(s) + store.put_write(&p,huge) + meta:=fmt.tprintf("%s%cgame",SNAPSHOT_MAGIC,rune(4)) + large,_,_:=store.put_finish(&p,transmute([]u8)meta) + store.commit(s) + store.ref_set(s,NAMESPACE,offer_name(w.peers[0].key.secret,&digits),large) + _,refused:=test_join(t,&w,1,join_desc(&w,0,fmt.tprintf("%s/other",w.folder))) + testing.expectf(t,strings.contains(refused,"more than"),"%s",refused) + testing.expectf(t,node_totals(&w.peers[1].n).received<4<<20,"the folder was refused before it came: %d bytes",node_totals(&w.peers[1].n).received) +} + +// Invariant: a join after one file changed at a depth of d folders receives +// at most d+2 objects (the folders on its path, the root and the file), each +// as its root with at most exchange.ROOT_BUDGET of what follows it and the +// chunks its change rewrote (two at most) and one node per level: a bound of +// the depth alone, whatever the number of files of the project. +@(test) +a_rejoin_after_one_file_changed_receives_what_does_not_grow_with_the_project :: proc(t:^testing.T) { + DEPTH :: 3 // assets// + BOUND :: (DEPTH+2)*(exchange.ROOT_BUDGET+2*store.CHUNK_MAX) + for files in ([2]int{96,1536}) { + w:Test_Net + if !test_net(t,&w,fmt.tprintf("share-rejoin-%d",files),2) {return} + defer test_net_destroy(&w) + root:=fmt.tprintf("%s/host",w.folder) + file_put(root,"project.mirai",transmute([]u8)string("{}\n")) + for i in 0..>29 + b:=pool[h%POOL] + b=b[:min(i64(len(b)),SIZE-written)] + os.write(file,b) + written+=i64(len(b)) + } + os.close(file) + defer os.remove(path) + Sampler :: struct {base,peak:i64,stop:bool} + sample :: proc(data:rawptr) { + s:=(^Sampler)(data) + for !sync.atomic_load(&s.stop) { + sync.atomic_store(&s.peak,max(sync.atomic_load(&s.peak),i64(platform.process_memory().private))) + time.sleep(5*time.Millisecond) + } + } + measured :: proc(t:^testing.T,what:string,run:proc(data:rawptr),data:rawptr) -> i64 { + s:=Sampler{base=i64(platform.process_memory().private)} + s.peak=s.base + th:=thread.create_and_start_with_data(&s,sample) + started:=time.tick_now() + run(data) + sync.atomic_store(&s.stop,true) + thread.join(th) + thread.destroy(th) + fmt.printfln("%s: %.1f s, peak growth %d bytes",what,time.duration_seconds(time.tick_since(started)),s.peak-s.base) + return s.peak-s.base + } + Offer :: struct {t:^testing.T,w:^Test_Net,root:string} + offer:=Offer{t,&w,root} + shared:=measured(t,"share of 5 GB",proc(data:rawptr) { + o:=(^Offer)(data) + host_offer(o.t,o.w,o.root) + },&offer) + test_connect(&w,1,0) + joined:=fmt.tprintf("%s/joined",w.folder) + defer os.remove(fmt.tprintf("%s/assets/large.bin",joined)) + Run :: struct {t:^testing.T,w:^Test_Net,joined:string,report:Join_Report,error:string} + run:=Run{t=t,w=&w,joined=joined} + both:=measured(t,"join of 5 GB, both peers",proc(data:rawptr) { + r:=(^Run)(data) + r.report,r.error=test_join(r.t,r.w,1,join_desc(r.w,0,r.joined)) + },&run) + testing.expectf(t,run.error=="" && run.report.written==2,"%s %v",run.error,run.report) + info,_:=os.stat(fmt.tprintf("%s/assets/large.bin",joined),context.temp_allocator) + testing.expect_value(t,info.size,SIZE) + when #config(ODIN_TEST_THREADS,0)==1 { + testing.expectf(t,shared<=64<<20,"the share grew %d bytes",shared) + testing.expectf(t,both<=2*(64<<20),"the join grew %d bytes on both peers together",both) + } + } +} + +// A join leaves the list of packages and their manifests as the host has +// them, byte for byte, after its refresh: also the record of a document of a +// package the joiner's refresh does not write, as a model's prefab, which +// only a library with the game's types writes. +@(test) +the_packages_of_a_joined_project_are_the_hosts :: proc(t:^testing.T) { + w:Test_Net + if !test_net(t,&w,"share-packages",2) {return} + defer test_net_destroy(&w) + root:=fmt.tprintf("%s/host",w.folder) + if _,made:=host_project(t,&w);!made {return} + list,_,load_error:=asset.packages_load(root,context.temp_allocator) + if !testing.expect_value(t,load_error,asset.Error.None) || !testing.expect(t,len(list.packages)==1) {return} + entry:=&list.packages[0] + p,package_error:=asset.package_load(&w.peers[0].s,root,entry,context.temp_allocator) + if !testing.expect_value(t,package_error,asset.Error.None) {return} + append(&entry.written,asset.Package_Written{id=p.assets[0].id,hash=0x1234}) + testing.expect_value(t,asset.packages_save(root,&list),asset.Error.None) + if !host_offer(t,&w,root) {return} + test_connect(&w,1,0) + joined:=fmt.tprintf("%s/joined",w.folder) + report,error:=test_join(t,&w,1,join_desc(&w,0,joined)) + if !testing.expectf(t,error=="","%s",error) {return} + testing.expectf(t,len(report.packages.failures)==0 && report.packages.fetched==1,"%v",report.packages) + paths:=make([dynamic]string,context.temp_allocator) + append(&paths,asset.PACKAGES_FILE) + listed,_:=os.read_all_directory_by_path(fmt.tprintf("%s/packages",root),context.temp_allocator) + for info in listed {append(&paths,fmt.tprintf("packages/%s",info.name))} + testing.expect(t,len(paths)>1) + for path in paths {testing.expectf(t,read_text(root,path)==read_text(joined,path),"%s is the host's",path)} +} + +// A host that only opened its project in the editor holds Fast builds alone: +// a join takes them from it as an editor would and imports nothing, though +// the source is in the folder it wrote. +@(test) +a_join_from_a_host_with_only_fast_builds_fetches_them_and_imports_nothing :: proc(t:^testing.T) { + w:Test_Net + if !test_net(t,&w,"share-fast",2) {return} + defer test_net_destroy(&w) + root:=fmt.tprintf("%s/host",w.folder) + file_put(root,"project.mirai",transmute([]u8)string("{}\n")) + // An uncompressed TGA, top to bottom, of noise. + SIDE :: 16 + image:=make([]u8,18+SIDE*SIDE*4,context.temp_allocator) + image[2],image[12],image[14],image[16],image[17]=2,SIDE,SIDE,32,0x28 + copy(image[18:],noise(SIDE*SIDE*4,3)) + file_put(root,"assets/poster.tga",image) + options:=sources.Library_Options{store=&w.peers[0].s,root=root,fast_textures=true} + imported,import_error:=sources.ingest(fmt.tprintf("%s/assets/poster.tga",root),options,allocator=context.temp_allocator) + if !testing.expectf(t,import_error.code==.None,"%s",import_error.message) {return} + testing.expect(t,!imported.final,"the host's build is Fast") + added:=sources.project_add(root,imported.manifest,"",nil,options,context.temp_allocator) + if !testing.expectf(t,len(added.failures)==0 && len(added.missing)==0,"%v",added) {return} + if !host_offer(t,&w,root) {return} + test_connect(&w,1,0) + joined:=fmt.tprintf("%s/joined",w.folder) + report,error:=test_join(t,&w,1,join_desc(&w,0,joined)) + if !testing.expectf(t,error=="","%s",error) {return} + testing.expect(t,os.exists(fmt.tprintf("%s/assets/poster.tga",joined)),"the source is in the joined folder") + testing.expectf(t,report.packages.fetched==1 && report.packages.ingested==0 && len(report.packages.missing)==0 && len(report.packages.failures)==0,"%v",report.packages) + digits:[64]u8 + key:=store.hex(imported.key,&digits) + fast,fast_error:=store.ref_get(&w.peers[1].s,asset.REF_IMPORTS_FAST,key) + _,final_error:=store.ref_get(&w.peers[1].s,asset.REF_IMPORTS,key) + testing.expect(t,fast_error==.None && fast==imported.build && final_error==.Missing,"the joiner names the Fast build as Fast") +} + +// Runs work on a thread while the network steps, as test_join does; stop_bytes: +// in the step peer k has received that many chunk bytes, peer from leaves +// (its server says goodbye and is pumped no more). +@(private="file") +test_run :: proc(t:^testing.T,w:^Test_Net,k:int,work:proc(data:rawptr),data:rawptr,stop_bytes:u64=0,from:=0) { + Run :: struct {work:proc(data:rawptr),data:rawptr,done:bool} + run:=Run{work,data,false} + th:=thread.create_and_start_with_data(&run,proc(data:rawptr) { + r:=(^Run)(data) + r.work(r.data) + free_all(context.temp_allocator) + sync.atomic_store(&r.done,true) + },init_context=context) + n:=&w.peers[k].n + // The clock moves while the work waits on the node, or, as a blocking fetch waits on the + // exchange, after it yielded a while. + for steps,yields:=0,0;!sync.atomic_load(&run.done); { + if !node_waiting(n) && yields<200 { + yields+=1 + thread.yield() + continue + } + yields=0 + test_step(w) + steps+=1 + if steps==600_000 { + testing.expect(t,false,"the work did not end in 600 virtual seconds") + node_stop(n) + } + if stop_bytes>0 && w.peers[from].pumped && node_totals(n).received>=stop_bytes { + w.peers[from].pumped=false + h:=w.peers[from].server + wire.host_disconnect(h,0,.Closed) + wire.host_poll(h) + wire.host_update(h,w.now) + } + } + thread.join(th) + thread.destroy(th) +} + +// A package of the library of peer 0 that no project of it pins: a sound of samples. +@(private="file") +library_sound :: proc(t:^testing.T,w:^Test_Net,samples:int) -> (manifest:store.Hash,ok:bool) { + sources_folder:=fmt.tprintf("%s/library",w.folder) + file_put(sources_folder,"chime.wav",wav(samples,9)) + report,error:=sources.ingest(fmt.tprintf("%s/chime.wav",sources_folder),{store=&w.peers[0].s},allocator=context.temp_allocator) + if !testing.expectf(t,error.code==.None,"%s",error.message) {return} + // Its source leaves the machine: only its build gives it. + testing.expect(t,os.rename(sources_folder,fmt.tprintf("%s/library-gone",w.folder))==nil) + return report.manifest,true +} + +@(private="file") +Library_Run :: struct { + w:^Test_Net, + list:[]Peer_Package, + error:string, + manifest:store.Hash, + root:string, +} + +@(private="file") +library_list_run :: proc(t:^testing.T,w:^Test_Net,r:^Library_Run) { + test_run(t,w,1,proc(data:rawptr) { + r:=(^Library_Run)(data) + buffer:[PEERS]Peer + peers:=node_peers(&r.w.peers[1].n,buffer[:]) + for tries:=0;len(peers)==0 && tries<1000 && node_wait(&r.w.peers[1].n,0.01);tries+=1 {peers=node_peers(&r.w.peers[1].n,buffer[:])} + if len(peers)!=1 {r.error="not one peer";return} + r.list,r.error=peer_library(&r.w.peers[1].n,peers[0],context.allocator) + },r) +} + +// A peer's library is listed with name, kinds, size and whether this machine +// has it; one of its packages is taken with its build, pinned into a project +// that then has it built, stays in the library of this machine, and lists as here. +@(test) +a_peers_library_is_listed_and_a_package_of_it_taken_and_pinned :: proc(t:^testing.T) { + w:Test_Net + if !test_net(t,&w,"share-library",2) {return} + defer test_net_destroy(&w) + manifest,made:=library_sound(t,&w,4800) + if !made {return} + test_connect(&w,1,0) + r:=Library_Run{w=&w,root=fmt.tprintf("%s/guest",w.folder)} + os.make_directory_all(r.root) + library_list_run(t,&w,&r) + defer {for p in r.list {delete(p.name);delete(p.kinds)};delete(r.list)} + if !testing.expectf(t,r.error=="" && len(r.list)==1,"%s: %d",r.error,len(r.list)) {return} + listed:=r.list[0] + testing.expectf(t,listed.manifest==manifest && listed.name=="chime" && listed.built && !listed.here && listed.bytes>0 && len(listed.kinds)==1,"the package is listed with what it holds: %v",listed) + testing.expect(t,listed.root!={} && len(listed.objects)==1 && listed.objects[0]==listed.root,"with the objects of its build") + received:=node_totals(&w.peers[1].n).received + testing.expectf(t,received0,"%v",pinned) + for l in pinned.listed { + a:asset.Asset + if testing.expectf(t,asset.asset_open_object(&a,s,l.object,context.temp_allocator)==.None,"%s is built here",l.path) {asset.asset_close(&a)} + } + buffer:[base.ASSET_ID_TEXT]u8 + kept,ref_error:=store.ref_get(s,asset.REF_PACKAGES,base.asset_id_format(listed.id,buffer[:])) + testing.expect(t,ref_error==.None && kept==manifest,"it stays in the library of this machine") + + for p in r.list {delete(p.name);delete(p.kinds)} + delete(r.list) + r.list=nil + library_list_run(t,&w,&r) + testing.expect(t,r.error=="" && len(r.list)==1 && r.list[0].here,"listed again, it is here") +} + +// A peer that leaves while a package of it is taken leaves no package in the +// library of this machine and no project changed, and says why. +@(test) +a_peer_that_leaves_mid_take_leaves_nothing_taken :: proc(t:^testing.T) { + w:Test_Net + if !test_net(t,&w,"share-library-gone",2) {return} + defer test_net_destroy(&w) + manifest,made:=library_sound(t,&w,1<<20) + if !made {return} + test_connect(&w,1,0) + r:=Library_Run{w=&w,manifest=manifest,root=fmt.tprintf("%s/guest",w.folder)} + os.make_directory_all(r.root) + test_run(t,&w,1,proc(data:rawptr) { + r:=(^Library_Run)(data) + for tries:=0;tries<1000 && node_totals(&r.w.peers[1].n).connected==0 && node_wait(&r.w.peers[1].n,0.01);tries+=1 {} + r.error=strings.clone(package_take(&r.w.peers[1].n,r.manifest,r.root)) + },&r,stop_bytes=256<<10) + defer delete(r.error) + testing.expectf(t,strings.contains(r.error,"left"),"why: %s",r.error) + p,_:=asset.package_get(&w.peers[0].s,manifest,context.temp_allocator) + buffer:[base.ASSET_ID_TEXT]u8 + _,ref_error:=store.ref_get(&w.peers[1].s,asset.REF_PACKAGES,base.asset_id_format(p.id,buffer[:])) + testing.expect(t,ref_error!=.None,"nothing taken into the library") + testing.expect(t,!os.exists(fmt.tprintf("%s/%s",r.root,asset.PACKAGES_FILE)),"no project changed") +} + +// Derived data a peer has (a thumbnail, by its key) is fetched from it and +// kept here under the same ref; one it lacks is Absent, and nothing is made +// for it on either side. +@(test) +a_peer_gives_derived_data_it_has_and_nothing_it_lacks :: proc(t:^testing.T) { + w:Test_Net + if !test_net(t,&w,"share-derived",2) {return} + defer test_net_destroy(&w) + NAMESPACE_THUMBS :: asset.REF_CACHE_PREFIX+"test-thumbnails" + image:=noise(4096,5) + address,put_error:=store.put(&w.peers[0].s,image) + if !testing.expect_value(t,put_error,store.Error.None) {return} + testing.expect_value(t,store.ref_set(&w.peers[0].s,NAMESPACE_THUMBS,"00000000000000a1",address),store.Error.None) + test_connect(&w,1,0) + n:=&w.peers[1].n + for steps:=0;node_totals(n).connected==0 && steps<10000;steps+=1 {test_step(&w)} + buffer:[PEERS]Peer + peers:=node_peers(n,buffer[:]) + if !testing.expect_value(t,len(peers),1) {return} + Case :: struct {name:string,then:Fetch_State} + for c in ([]Case{{"00000000000000a1",.Done},{"00000000000000b2",.Absent}}) { + f:=fetch_start(n,peers[0],NAMESPACE_THUMBS,c.name) + for steps:=0;f.state==.Asking || f.state==.Fetching;steps+=1 { + if !testing.expect(t,steps<200_000,"the fetch ends") {fetch_end(n,&f);break} + test_step(&w) + fetch_poll(n,&f) + } + testing.expect_value(t,f.state,c.then) + here,ref_error:=store.ref_get(&w.peers[1].s,NAMESPACE_THUMBS,c.name) + testing.expect_value(t,ref_error==.None,c.then==.Done) + if c.then==.Done {testing.expect(t,here==address && asset.object_whole(&w.peers[1].s,address))} + _,lacking:=store.ref_get(&w.peers[0].s,NAMESPACE_THUMBS,c.name) + testing.expect_value(t,lacking==.None,c.then==.Done) + } +} diff --git a/cook/share/snapshot.odin b/cook/share/snapshot.odin new file mode 100644 index 000000000..5da15b916 --- /dev/null +++ b/cook/share/snapshot.odin @@ -0,0 +1,735 @@ +package share + +import "base:runtime" +import "core:fmt" +import "core:os" +import "core:slice" +import "core:strings" +import "core:time" +import importer "mirai:cook/import" +import "mirai:project" +import "mirai:std/jobs" +import "mirai:store" + +// A snapshot: the files of a project as a tree of objects of the store. A +// file is an object of its bytes, cut on content, so a changed file shares +// the chunks the change did not touch. A folder is an object whose bytes list +// its entries sorted by name, each a file (with its executable bit) or a +// folder, by the address and size of its object; a folder entry also carries +// the files and bytes under it. The meta of every folder is SNAPSHOT_MAGIC and +// the length of a name, the project's in the root and none below it. Two +// snapshots that differ in one file share every folder off its path. What +// each machine makes (project.own_path) is left out, and a folder holds at +// least one file. The same files give the same tree. +// +// In memory, and as the base a joined folder keeps, a snapshot is its files +// by path, sorted bytewise. + +SNAPSHOT_MAGIC :: "mirai snapshot 2\n" +FILES_MAX :: 1<<22 // files of a snapshot +FOLDER_ENTRIES_MAX :: 1<<16 // entries of one folder +DEPTH_MAX :: 64 // folders on the path of a file +PATH_MAX :: 1024 // bytes of a path of a snapshot +NAME_MAX :: 64 // bytes of the name of the project it carries +@(private) RECORD_MAX :: 1+1+255+32+8+4 // kind, name, address, size, files +LISTING_MAX :: FOLDER_ENTRIES_MAX*RECORD_MAX // bytes of the object of one folder + +@(private) KIND_FILE :: 0 +@(private) KIND_EXECUTABLE :: 1 +@(private) KIND_FOLDER :: 2 + +Entry :: struct { + path:string, + address:store.Hash, // of the file's object + size:i64, + executable:bool, +} + +Snapshot :: struct { + name:string, // of the project, the folder a join makes by default + entries:[]Entry, // sorted by path +} + +Snapshot_Error :: enum u8 { + None, + Truncated, // the bytes end before what they say + Not_Snapshot, // another magic or meta, bytes after the last entry, an unknown kind + Too_Large, // more files, entries of a folder or bytes than the bounds above + Path, // a name or path that would leave the folder, is absolute, names a device or the generated folder + Duplicate, // two names that name one file where case is folded, or out of order + Size, // a size or count out of range or unlike what is under it, or an empty file whose address is not that of no bytes + Name, // a name of the project that is not a folder name + Depth, // folders nested deeper than DEPTH_MAX + Cycle, // a folder inside itself + Unavailable, // an object of the tree is not in the store, or could not be fetched + Out_Of_Memory, +} + +// The address of the object of no bytes: an empty file's. +empty_address :: proc() -> store.Hash { + groups:=[1]store.Group{{}} + buffer:[size_of(store.Node)+size_of(store.Group)]u8 + bytes,_:=store.node_encode(buffer[:],0,nil,groups[:]) + return store.address_of_bytes(bytes) +} + +// An entry of a folder. +@(private) +Item :: struct { + name:string, + kind:u8, + address:store.Hash, + size:i64, // a file's bytes; the bytes of the files under a folder + files:i64, // under a folder +} + +// Puts the folders of snap in s, the files' objects being there already: +// the address of its root. Each folder put is added to folders. +snapshot_put :: proc(s:^store.Store,snap:^Snapshot,folders:^[dynamic]store.Hash=nil) -> (root:store.Hash,error:Snapshot_Error,store_error:store.Error) { + if !name_valid(snap.name) {return {},.Name,.None} + if len(snap.entries)>FILES_MAX {return {},.Too_Large,.None} + built:Item + built,error,store_error=folder_put(s,snap.entries,0,snap.name,folders,0) + if error==.None && store_error==.None {store_error=store.commit(s)} + return built.address,error,store_error +} + +// The folder of entries, all under the same path of prefix bytes (and a +// slash), with its subfolders put first. +@(private) +folder_put :: proc(s:^store.Store,entries:[]Entry,prefix:int,name:string,folders:^[dynamic]store.Hash,depth:int) -> (folder:Item,error:Snapshot_Error,store_error:store.Error) { + if depth>DEPTH_MAX {return {},.Depth,.None} + items:=make([dynamic]Item,context.temp_allocator) + for i:=0;iFOLDER_ENTRIES_MAX {return {},.Too_Large,.None} + slice.sort_by(items[:],proc(a,b:Item) -> bool {return a.name bool, + data:rawptr, +} + +// The name of the project the snapshot at root carries, from the meta of its +// root folder. +snapshot_name :: proc(s:^store.Store,root:store.Hash,allocator:=context.allocator) -> (name:string,error:Snapshot_Error) { + o,open_error:=store.object_open(s,root) + if open_error!=.None {return "",open_error==.Missing?.Unavailable:.Not_Snapshot} + defer store.object_close(&o) + name=meta_name(store.object_meta(&o),true) or_return + return strings.clone(name,allocator),.None +} + +@(private) +meta_name :: proc(meta:[]u8,root:bool) -> (name:string,error:Snapshot_Error) { + if len(meta) (snap:Snapshot,error:Snapshot_Error) { + t:=Tree_Read{ + s=s, + visits=make([dynamic]Visit,context.temp_allocator), + entries=make([dynamic]Entry,allocator), + allocator=allocator, + } + append(&t.visits,Visit{address=root,files=-1,parent=-1}) + addresses:=make([dynamic]store.Hash,context.temp_allocator) + for start:=0;start bool {return a.path Snapshot_Error { + v:=t.visits[k] + o,open_error:=store.object_open(t.s,v.address) + #partial switch open_error { + case .None: + case .Missing: return .Unavailable + case .Out_Of_Memory: return .Out_Of_Memory + case: return .Not_Snapshot + } + defer store.object_close(&o) + name:=meta_name(store.object_meta(&o),v.parent<0) or_return + if v.parent<0 {t.name=strings.clone(name,context.temp_allocator)} + total:=store.object_size(&o) + if total>LISTING_MAX {return .Too_Large} + if total==0 && v.parent>=0 {return .Size} + buffer:[64<<10+RECORD_MAX]u8 + r:=Reader{data=buffer[:0]} + at:u64 + // The name before, kept apart from the buffer, which is refilled. + previous_bytes:[255]u8 + previous:string + folded:=make(map[string]bool,context.temp_allocator) + files,size,count:i64 + empty:=empty_address() + for { + // Room for one whole record of any size before it is read. + if len(r.data)-r.atKIND_FOLDER {return .Not_Snapshot} + count+=1 + if count>FOLDER_ENTRIES_MAX {return .Too_Large} + if !part_valid(item_name) {return .Path} + if item_name<=previous && count>1 {return .Duplicate} + lower:=strings.to_lower(item_name,context.temp_allocator) + if lower in folded {return .Duplicate} + folded[lower]=true + previous=string(previous_bytes[:copy(previous_bytes[:],item_name)]) + path:=v.path==""?strings.clone(item_name,context.temp_allocator):fmt.tprintf("%s/%s",v.path,item_name) + if !path_valid(path) {return .Path} + if item_size>u64(max(i64)) || i64(item_size)>max(i64)-size {return .Size} + size+=i64(item_size) + if kind!=KIND_FOLDER { + if item_size==0 && address!=empty {return .Size} + files+=1 + if v.parent<0 && files>FILES_MAX {return .Too_Large} + append(&t.entries,Entry{path=strings.clone(path,t.allocator),address=address,size=i64(item_size),executable=kind==KIND_EXECUTABLE}) + continue + } + if item_files==0 {return .Size} + if v.depth+1>DEPTH_MAX {return .Depth} + for a:=k;a>=0;a=t.visits[a].parent { + if t.visits[a].address==address {return .Cycle} + } + files+=i64(item_files) + if v.parent<0 && files>FILES_MAX {return .Too_Large} + append(&t.visits,Visit{path=path,address=address,files=i64(item_files),size=i64(item_size),depth=v.depth+1,parent=k}) + } + if v.parent>=0 && (files!=v.files || size!=v.size) {return .Size} + return .None +} + +// Whether a path of a snapshot can be written under any folder of any +// system: relative, with forward slashes, no empty, . or .. part, no part +// a system reserves (a device of Windows, a name that ends in a dot or a +// space, a colon, a control character), and not under what is not the +// project's own. +path_valid :: proc(path:string) -> bool { + if path=="" || len(path)>PATH_MAX || !project.own_path(path) {return false} + start:=0 + for i in 0..=len(path) { + if i bool { + if part=="" || part=="." || part==".." || len(part)>255 {return false} + for c in transmute([]u8)part { + switch c { + case 0..<0x20,0x7f,'/','\\',':','*','?','"','<','>','|': return false + } + } + last:=part[len(part)-1] + if last=='.' || last==' ' {return false} + // A device of Windows, also with an extension: CON, PRN, AUX, NUL, COM1-9, LPT1-9. + stem:=part + if dot:=strings.index_byte(part,'.');dot>=0 {stem=part[:dot]} + stem=strings.trim_right(stem," ") + upper:[4]u8 + if len(stem)==3 || len(stem)==4 { + for c,i in transmute([]u8)stem {upper[i]=c>='a' && c<='z'?c-'a'+'A':c} + word:=string(upper[:len(stem)]) + switch word[:3] { + case "CON","PRN","AUX","NUL": if len(word)==3 {return false} + case "COM","LPT": if len(word)==4 && word[3]>='1' && word[3]<='9' {return false} + } + } + return true +} + +// Whether the project name can be a folder: one valid part, not empty. +name_valid :: proc(name:string) -> bool { + return name!="" && len(name)<=NAME_MAX && part_valid(name) && name!=".mirai" +} + +// No two paths name one file on a system that folds case, and no file is a +// folder of another. +@(private) +distinct_files :: proc(entries:[]Entry,allocator:=context.allocator) -> bool { + folded:=make(map[string]bool,len(entries),context.temp_allocator) + for e in entries { + lower:=strings.to_lower(e.path,context.temp_allocator) + if lower in folded {return false} + folded[lower]=false + } + for e in entries { + lower:=strings.to_lower(e.path,context.temp_allocator) + for at:=strings.last_index_byte(lower,'/');at>0;at=strings.last_index_byte(lower[:at],'/') { + if lower[:at] in folded {return false} + } + } + return true +} + +// The entry of path in a sorted snapshot. +snapshot_find :: proc(snap:^Snapshot,path:string) -> (^Entry,bool) { + at,found:=slice.binary_search_by(snap.entries,path,proc(e:Entry,path:string) -> slice.Ordering {return slice.cmp(e.path,path)}) + if !found {return nil,false} + return &snap.entries[at],true +} + +// The base a joined folder keeps (/.mirai/share/snapshot): the +// snapshot its last join took, by path. +@(private) BASE_MAGIC :: "mirai share base 2\n" + +snapshot_encode :: proc(snap:Snapshot,allocator:=context.allocator) -> (bytes:[]u8,error:Snapshot_Error) { + if len(snap.entries)>FILES_MAX {return nil,.Too_Large} + size:=len(BASE_MAGIC)+2+len(snap.name)+4 + for e in snap.entries {size+=2+len(e.path)+32+8+1} + made,make_error:=make([]u8,size,allocator) + if make_error!=nil {return nil,.Out_Of_Memory} + w:=Writer{data=made} + write_bytes(&w,transmute([]u8)string(BASE_MAGIC)) + write_uint(&w,u64(len(snap.name)),2) + write_bytes(&w,transmute([]u8)snap.name) + write_uint(&w,u64(len(snap.entries)),4) + for e in snap.entries { + address:=e.address + write_uint(&w,u64(len(e.path)),2) + write_bytes(&w,transmute([]u8)e.path) + write_bytes(&w,address[:]) + write_uint(&w,u64(e.size),8) + write_uint(&w,e.executable?1:0,1) + } + return made,.None +} + +// The base of bytes: every path, size and name is checked, and the paths +// must be sorted and name distinct files. The strings point into bytes. +snapshot_decode :: proc(bytes:[]u8,allocator:=context.allocator) -> (snap:Snapshot,error:Snapshot_Error) { + r:=Reader{data=bytes} + magic:=read_bytes(&r,len(BASE_MAGIC)) + if r.short {return {},.Truncated} + if string(magic)!=BASE_MAGIC {return {},.Not_Snapshot} + snap.name=string(read_bytes(&r,int(read_uint(&r,2)))) + count:=int(read_uint(&r,4)) + if r.short {return {},.Truncated} + if !name_valid(snap.name) {return {},.Name} + if count>FILES_MAX {return {},.Too_Large} + // Each entry takes at least 44 bytes: a count that the bytes cannot hold is a lie. + if count>(len(bytes)-r.at)/44 {return {},.Truncated} + entries,make_error:=make([]Entry,count,allocator) + if make_error!=nil {return {},.Out_Of_Memory} + empty:=empty_address() + for &e in entries { + e.path=string(read_bytes(&r,int(read_uint(&r,2)))) + copy(e.address[:],read_bytes(&r,32)) + size:=read_uint(&r,8) + flags:=read_uint(&r,1) + if r.short {return {},.Truncated} + if !path_valid(e.path) {return {},.Path} + if size>u64(max(i64)) || flags>1 || size==0 && e.address!=empty {return {},.Size} + e.size,e.executable=i64(size),flags==1 + } + if r.at!=len(bytes) {return {},.Not_Snapshot} + for k in 1..=entries[k].path {return {},.Duplicate} + } + if !distinct_files(entries,allocator) {return {},.Duplicate} + snap.entries=entries + return snap,.None +} + +// What a machine knows of the files of a folder (/.mirai/share/index): +// the address of each file as it was when last read or written, by its size and +// modification time, so a file that did not change is not read again. +Known :: struct {size,modified:i64,address:store.Hash} +Index :: map[string]Known + +@(private) INDEX_MAGIC :: "mirai share index 2\n" +@(private) SHARE_FOLDER :: "share" // under the generated folder of a project: what sharing keeps on this machine + +share_path :: proc(root,name:string,allocator:=context.allocator) -> string { + return project.path(project.folder_path(root,.Generated,context.temp_allocator),SHARE_FOLDER,name,allocator=allocator) +} + +// The index of root, empty when it does not read. +index_load :: proc(root:string,allocator:=context.allocator) -> Index { + index:=make(Index,allocator) + data,error:=os.read_entire_file(share_path(root,"index",context.temp_allocator),context.temp_allocator) + if error!=nil || len(data) bool { + size:=len(INDEX_MAGIC) + for path in index {size+=2+len(path)+48} + data:=make([]u8,size,context.temp_allocator) + w:=Writer{data=data} + write_bytes(&w,transmute([]u8)string(INDEX_MAGIC)) + for path,k in index { + address:=k.address + write_uint(&w,u64(len(path)),2) + write_bytes(&w,transmute([]u8)path) + write_uint(&w,u64(k.size),8) + write_uint(&w,u64(k.modified),8) + write_bytes(&w,address[:]) + } + return file_save(share_path(root,"index",context.temp_allocator),data) +} + +// The address of the file at path, from the index when its size and time are +// the ones it knows; the index learns it otherwise. False when it does not read. +index_address :: proc(index:^Index,relative,path:string) -> (address:store.Hash,size:i64,ok:bool) { + info,error:=os.stat(path,context.temp_allocator) + if error!=nil || info.type!=.Regular {return} + modified:=time.to_unix_nanoseconds(info.modification_time) + if k,known:=index[relative];known && k.size==info.size && k.modified==modified {return k.address,k.size,true} + hashed,hashed_size,hashed_ok:=file_address(path) + if !hashed_ok || hashed_size!=info.size {return} + index_learn(index,relative,info,hashed) + return hashed,hashed_size,true +} + +// Learns the address of a file whose size and time are those of info, unless +// it was modified too close to now (importer.SETTLED_NANOSECONDS): a change +// within the same time would not be seen. +@(private) +index_learn :: proc(index:^Index,relative:string,info:os.File_Info,address:store.Hash) { + entry:=Known{size=info.size,modified=time.to_unix_nanoseconds(info.modification_time),address=address} + if time.to_unix_nanoseconds(time.now())-entry.modified<=importer.SETTLED_NANOSECONDS {return} + if _,known:=index[relative];known { + index[relative]=entry + } else { + index[strings.clone(relative,index.allocator)]=entry + } +} + +@(private) READ_BYTES :: 1<<20 // of a file read at once + +// The address of the object of the file at path, read once and cut as the +// store cuts it, without putting it anywhere. +file_address :: proc(path:string) -> (address:store.Hash,size:i64,ok:bool) { + w:store.Object_Writer + if store.object_writer_init(&w,{procedure=proc(data:rawptr,kind:store.Chunk_Kind,address:store.Hash,bytes:[]u8) -> store.Error {return .None}})!=.None {return} + defer store.object_writer_destroy(&w) + size,ok=file_stream(path,proc(data:rawptr,bytes:[]u8) -> bool {return store.object_write((^store.Object_Writer)(data),bytes)==.None},&w) + if !ok {return} + root,_,error:=store.object_finish(&w) + return root,size,error==.None +} + +// Puts the file at path in s as an object, read once, its chunks hashed and +// appended by jobs of worker when given: its address and the bytes put, which +// are those the file had when the read reached its end. +file_put_object :: proc(s:^store.Store,path:string,worker:=jobs.Worker_Ref{}) -> (address:store.Hash,size:i64,error:store.Error) { + p:=store.put_begin(s,nil,worker) or_return + Put_Read :: struct {p:^store.Put,error:store.Error} + put_read:=Put_Read{p=&p} + ok:bool + size,ok=file_stream(path,proc(data:rawptr,bytes:[]u8) -> bool { + r:=(^Put_Read)(data) + r.error=store.put_write(r.p,bytes) + return r.error==.None + },&put_read) + if !ok { + store.put_abandon(&p) + return {},0,put_read.error!=.None?put_read.error:.Io + } + address,_=store.put_finish(&p) or_return + return address,size,.None +} + +// Gives the bytes of the file at path to each in pieces of at most +// READ_BYTES, up to its end: the bytes given. False when it does not read or +// each refuses a piece. +@(private) +file_stream :: proc(path:string,each:proc(data:rawptr,bytes:[]u8) -> bool,data:rawptr) -> (size:i64,ok:bool) { + file,open_error:=os.open(path) + if open_error!=nil {return} + defer os.close(file) + buffer,make_error:=make([]u8,READ_BYTES,context.temp_allocator) + if make_error!=nil {return} + defer delete(buffer,context.temp_allocator) + for { + n,read_error:=os.read(file,buffer) + if n>0 { + if !each(data,buffer[:n]) {return} + size+=i64(n) + } + if read_error==.EOF || n==0 && read_error==nil {return size,true} + if read_error!=nil {return} + } +} + +Take_Report :: struct { + files:int, + bytes:i64, + put:int, // files read into the store now: the others were current + skipped:[]string, // what was left out, why and how to share it +} + +// The snapshot of the project at root: every file of its own put in s (one +// the index knows unchanged and the store has is not read again), then its +// folders; address is its root. Each folder put is added to folders. Jobs of +// worker, when given, hash and append the chunks of a file. In allocator. +snapshot_take :: proc(s:^store.Store,root,name:string,folders:^[dynamic]store.Hash=nil,worker:=jobs.Worker_Ref{},allocator:=context.allocator) -> (snap:Snapshot,address:store.Hash,report:Take_Report,error:string) { + index:=index_load(root,allocator) + // The root read once: each file then asks only what this value knows. + store.chunk_has(s,{}) + entries:=make([dynamic]Entry,allocator) + skipped:=make([dynamic]string,allocator) + todo:=make([dynamic]string,context.temp_allocator) + append(&todo,"") + for next:=0;next bool {return a.path (e:Entry,put:bool,error:store.Error) { + path:=project.path(root,relative,allocator=context.temp_allocator) + modified:=time.to_unix_nanoseconds(info.modification_time) + e=Entry{path=strings.clone(relative,allocator),size=info.size,executable=info.mode&os.Permissions_Execute_All!={}} + if k,known:=index[relative];known && k.size==info.size && k.modified==modified && store.chunk_here(s,k.address) { + e.address=k.address + return + } + e.address,e.size=file_put_object(s,path,worker) or_return + if after,stat_error:=os.stat(path,context.temp_allocator);stat_error==nil && after.size==e.size && after.size==info.size && time.to_unix_nanoseconds(after.modification_time)==modified { + index_learn(index,relative,after,e.address) + } + return e,true,.None +} + +// Writes data whole at path through a rename, making its folder. +file_save :: proc(path:string,data:[]u8) -> bool { + folder,_:=os.split_path(path) + os.make_directory_all(folder) + temp:=fmt.tprintf("%s.%d.tmp",path,os.get_pid()) + if os.write_entire_file(temp,data)!=nil { + os.remove(temp) + return false + } + if os.rename(temp,path)!=nil { + os.remove(temp) + return false + } + return true +} + +@(private) +Writer :: struct {data:[]u8,at:int} + +@(private) +write_bytes :: proc(w:^Writer,bytes:[]u8) { + copy(w.data[w.at:],bytes) + w.at+=len(bytes) +} + +@(private) +write_uint :: proc(w:^Writer,value:u64,size:int) { + for k in 0..>(8*uint(k)))} + w.at+=size +} + +@(private) +Reader :: struct {data:[]u8,at:int,short:bool} + +@(private) +read_bytes :: proc(r:^Reader,count:int) -> []u8 { + if r.short || count<0 || count>len(r.data)-r.at { + r.short=true + return nil + } + bytes:=r.data[r.at:][:count] + r.at+=count + return bytes +} + +@(private) +read_uint :: proc(r:^Reader,size:int) -> (value:u64) { + bytes:=read_bytes(r,size) + for b,k in bytes {value|=u64(b)<<(8*uint(k))} + return +} diff --git a/cook/share/space_posix.odin b/cook/share/space_posix.odin new file mode 100644 index 000000000..fad8ea754 --- /dev/null +++ b/cook/share/space_posix.odin @@ -0,0 +1,19 @@ +#+build linux, darwin, freebsd, openbsd, netbsd +package share + +import "core:strings" +import "core:sys/posix" + +// Bytes this user may still write on the volume that holds path; false when +// the system does not say. +free_bytes :: proc(path:string) -> (bytes:i64,ok:bool) { + info:posix.statvfs_t + if posix.statvfs(strings.clone_to_cstring(path,context.temp_allocator),&info)!=.OK {return} + return i64(info.f_bavail)*i64(info.f_frsize),true +} + +// Whether this user may write the file or folder at path: a read-only file is +// not replaced, and a folder without write access takes no file. +writable :: proc(path:string) -> bool { + return posix.access(strings.clone_to_cstring(path,context.temp_allocator),{.W_OK})==.OK +} diff --git a/cook/share/space_windows.odin b/cook/share/space_windows.odin new file mode 100644 index 000000000..ad5d45599 --- /dev/null +++ b/cook/share/space_windows.odin @@ -0,0 +1,20 @@ +package share + +import win "core:sys/windows" + +// Bytes this user may still write on the volume that holds path; false when +// the system does not say. +free_bytes :: proc(path:string) -> (bytes:i64,ok:bool) { + available:win.ULARGE_INTEGER + if !win.GetDiskFreeSpaceExW(win.utf8_to_wstring(path,context.temp_allocator),&available,nil,nil) {return} + return i64(available),true +} + +// Whether this user may write the file or folder at path: a file with the +// read-only attribute is not replaced. Windows ignores that attribute on a +// folder. +writable :: proc(path:string) -> bool { + attributes:=win.GetFileAttributesW(win.utf8_to_wstring(path,context.temp_allocator)) + if attributes==win.INVALID_FILE_ATTRIBUTES {return false} + return attributes&win.FILE_ATTRIBUTE_READONLY==0 || attributes&win.FILE_ATTRIBUTE_DIRECTORY!=0 +} diff --git a/cook/textures/bc_test.odin b/cook/textures/bc_test.odin index f3eb7ca8a..ced52abac 100644 --- a/cook/textures/bc_test.odin +++ b/cook/textures/bc_test.odin @@ -170,6 +170,7 @@ psnr :: proc(format:asset.Texture_Format,blocks,pixels:[]u8,size,channels:int) - the_rdo_lambda_trades_error_for_files_that_compress_smaller :: proc(t:^testing.T) { SIZE :: 128 types:asset.Types;asset.types_init(&types);defer asset.types_destroy(&types) + types.codecs=.Open // the files a cook ships, which RDO makes smaller folder:=base.test_folder(t,"rdo") if folder=="" {return} pixels:=smooth_image(SIZE,true);defer delete(pixels) @@ -318,3 +319,14 @@ a_bc7_texture_restores_its_tail_without_its_larger_levels :: proc(t:^testing.T) } free_all(context.temp_allocator) } + +// The block encoder of a build is part of what a texture is built with only where it makes the +// bytes: uncompressed texels are the same for every build, and so is the key of their import. +@(test) +the_encoder_counts_only_for_textures_of_blocks :: proc(t:^testing.T) { + for options in ([]Build_Options{{srgb=true,encoding=.Uncompressed},{encoding=.Uncompressed,speed=.Fast}}) { + testing.expect_value(t,settings_hash_of(options,1),settings_hash_of(options,2)) + } + blocks:=Build_Options{srgb=true} + testing.expect(t,settings_hash_of(blocks,1)!=settings_hash_of(blocks,2),"blocks of another encoder are another texture") +} diff --git a/cook/textures/cache.odin b/cook/textures/cache.odin index 18feb24bf..0841730a5 100644 --- a/cook/textures/cache.odin +++ b/cook/textures/cache.odin @@ -1,21 +1,22 @@ package textures -import "core:hash/xxhash" -import "core:os" +import "core:mem" import "core:slice" import "mirai:asset" import "mirai:cook/cache" import "mirai:std/base" -// Built textures kept by what they are made from (cook/cache): an import that -// builds the same image with the same settings, and a cook that compresses -// the same RGBA8 texture, read the levels instead of making them again. Its -// files are textures built by BUILD_VERSION. +// Built textures kept by what they are made from in the cache CACHE +// (cook/cache): an import that builds the same image with the same settings, +// and a cook that compresses the same RGBA8 texture, read the levels instead +// of making them again. Its files are textures built by BUILD_VERSION. + +CACHE :: "textures" // An image built with options: its bytes, encoded, or the texels of made // (whose width is 0 for encoded bytes). cache_key_build :: proc(bytes:[]u8,made:Source,options:Build_Options) -> u64 { - key:=base.hash_combine(xxhash.XXH3_64_default(bytes),u64(u32(made.width))<<32|u64(u32(made.height))) + key:=base.hash_combine(base.hash64(bytes),u64(u32(made.width))<<32|u64(u32(made.height))) key=base.hash_combine(key,u64(u32(made.layers))|u64(made.format)<<32|u64(made.cube)<<40) return base.hash_combine(key,settings_hash(options))|1 } @@ -26,15 +27,20 @@ cache_key_blocks :: proc(content_hash:u64,rdo:=true) -> u64 { } // Whether the cache keeps a texture under key, without reading it. -cache_has :: proc(folder:string,key:u64) -> bool { - return folder!="" && os.exists(cache.file_path(folder,key,".tex")) +cache_has :: proc(c:cache.Cache,key:u64) -> bool { + return cache.has(c,CACHE,key,BUILD_VERSION) +} + +// Which of keys the cache keeps, into here, asking its source once for those it lacks. +cache_lookup :: proc(c:cache.Cache,keys:[]u64,here:[]bool) { + cache.lookup(c,CACHE,keys,BUILD_VERSION,here) } // The texture the cache keeps under key, its levels in allocator. False // without one, or with one that does not read back whole. -cache_read :: proc(folder:string,key:u64,types:^asset.Types,allocator:=context.allocator) -> (built:Built,ok:bool) { +cache_read :: proc(c:cache.Cache,key:u64,types:^asset.Types,allocator:=context.allocator) -> (built:Built,ok:bool) { file:asset.Asset - if !cache.load(folder,key,BUILD_VERSION,".tex",&file,context.temp_allocator) {return} + if !cache.load(c,CACHE,key,BUILD_VERSION,&file,context.temp_allocator) {return} defer asset.asset_close(&file) view,error:=asset.texture_view(&file,types) if error!=.None {return} @@ -45,11 +51,15 @@ cache_read :: proc(folder:string,key:u64,types:^asset.Types,allocator:=context.a return built,true } -// Keeps built under key; false when it cannot be written, which costs only -// the time of building it the next time. -cache_write :: proc(folder:string,key:u64,types:^asset.Types,built:^Built) -> bool { - if !cache.ready(folder) {return false} - if asset.texture_save(cache.file_path(folder,key,".tex"),types,built_desc(built,cache.id(key,BUILD_VERSION),nil))!=.None {return false} - cache.written(folder) - return true +// Keeps built under key; false when it cannot be kept, which costs only the +// time of building it the next time. +cache_write :: proc(c:cache.Cache,key:u64,types:^asset.Types,built:^Built) -> bool { + if c.store==nil {return false} + doc:asset.Document + if asset.texture_document(&doc,types,built_desc(built,cache.id(key,BUILD_VERSION),nil))!=.None {return false} + defer asset.document_destroy(&doc) + bytes,error:=asset.document_bytes(&doc) + if error!=.None {return false} + defer mem.free_bytes(bytes) + return cache.put(c,CACHE,key,BUILD_VERSION,bytes) } diff --git a/cook/textures/cache_test.odin b/cook/textures/cache_test.odin index e30b9fa08..13357d4a6 100644 --- a/cook/textures/cache_test.odin +++ b/cook/textures/cache_test.odin @@ -1,14 +1,12 @@ package textures -import "core:fmt" -import "core:os" +import "core:log" import "core:slice" import "core:testing" -import "core:time" import "mirai:asset" import "mirai:cook/cache" import "mirai:std/base" -import "mirai:std/serial" +import "mirai:store" @(test) built_textures_are_kept_by_image_and_settings_and_read_back_whole :: proc(t:^testing.T) { @@ -16,6 +14,9 @@ built_textures_are_kept_by_image_and_settings_and_read_back_whole :: proc(t:^tes types:asset.Types;asset.types_init(&types);defer asset.types_destroy(&types) folder:=base.test_folder(t,"textures-cache") if folder=="" {return} + kept,opened:=store.open(folder,context.temp_allocator) + if !testing.expect_value(t,opened,store.Error.None) {return} + c:=cache.Cache{store=&kept} pixels:=make([]u8,16*16*4,context.temp_allocator) for &value,at in pixels {value=at%4==3?255:u8(at*7)} options:=Build_Options{srgb=true} @@ -27,49 +28,78 @@ built_textures_are_kept_by_image_and_settings_and_read_back_whole :: proc(t:^tes testing.expect(t,key!=cache_key_build(other,rgba8(other,16,16),options),"and other bytes") testing.expect(t,cache_key_blocks(1)!=cache_key_blocks(2)) - _,missing:=cache_read(folder,key,&types,context.temp_allocator) + _,missing:=cache_read(c,key,&types,context.temp_allocator) testing.expect(t,!missing,"an empty cache has nothing") built,error:=build(rgba8(pixels,16,16),options,context.temp_allocator) testing.expect_value(t,error,Error.None) - testing.expect(t,cache_write(folder,key,&types,&built)) - read,found:=cache_read(folder,key,&types,context.temp_allocator) + testing.expect(t,cache_write(c,key,&types,&built)) + read,found:=cache_read(c,key,&types,context.temp_allocator) testing.expect(t,found && read.width==built.width && read.height==built.height && read.format==built.format && read.flags==built.flags && len(read.mips)==len(built.mips)) for level,at in built.mips { side:=max(16>>uint(at),1) testing.expect(t,same_level(built.format,read.mips[at],level,side,side)) } - _,other_key:=cache_read(folder,key+2,&types,context.temp_allocator) + _,other_key:=cache_read(c,key+2,&types,context.temp_allocator) testing.expect(t,!other_key) - _,no_folder:=cache_read("",key,&types,context.temp_allocator) - testing.expect(t,!no_folder && !cache_write("",key,&types,&built)) + _,no_store:=cache_read({},key,&types,context.temp_allocator) + testing.expect(t,!no_store && !cache_write({},key,&types,&built)) + testing.expect(t,cache_has(c,key) && !cache_has(c,key+2) && !cache_has({},key)) +} - // A file no build used for UNUSED_AGE goes, however small the folder; a read is a use. - trimmed:=base.test_folder(t,"textures-trim") - if trimmed=="" {return} - keys:[2]u64 - long_ago:=time.time_add(time.now(),-cache.UNUSED_AGE-time.Hour) - for &key_of,at in keys { - other[0]=u8(at) - key_of=cache_key_build(other,rgba8(other,16,16),options) - level_built,_:=build(rgba8(other,16,16),options,context.temp_allocator) - testing.expect(t,cache_write(trimmed,key_of,&types,&level_built)) - testing.expect(t,os.change_times(fmt.tprintf("%s/%016x.tex",trimmed,key_of),long_ago,long_ago)==nil) - } - _,first_read:=cache_read(trimmed,keys[0],&types,context.temp_allocator) - testing.expect(t,first_read) - testing.expect_value(t,cache.trim(trimmed),1) - _,kept_first:=cache_read(trimmed,keys[0],&types,context.temp_allocator) - _,kept_second:=cache_read(trimmed,keys[1],&types,context.temp_allocator) - testing.expect(t,kept_first && !kept_second,"the one read lately stays and the one unused for long goes") - // A file of the cache that does not read back whole is no texture. - path:=fmt.tprintf("%s/%016x.tex",folder,key) - whole,_:=serial.file_read(path,context.temp_allocator) - damaged:=slice.clone(whole,context.temp_allocator) - damaged[len(damaged)-1]~=0xff - for bytes in ([][]u8{whole[:len(whole)/2],damaged}) { - testing.expect_value(t,serial.file_write_atomic(path,bytes),serial.File_Error.None) - _,read_back:=cache_read(folder,key,&types,context.temp_allocator) - testing.expect(t,!read_back) +// One texel changed in the image of a texture changes a bounded run of the +// chunks of its object: the levels keep their cuts, so the rest are the same. +@(test) +a_texel_changed_changes_few_chunks_of_the_object :: proc(t:^testing.T) { + defer free_all(context.temp_allocator) + types:asset.Types;asset.types_init(&types);defer asset.types_destroy(&types) + folder:=base.test_folder(t,"textures-one-texel") + if folder=="" {return} + s,opened:=store.open(folder,context.temp_allocator) + if !testing.expect_value(t,opened,store.Error.None) {return} + defer store.close(&s) + SIDE :: 1024 + pixels:=make([]u8,SIDE*SIDE*4,context.temp_allocator) + state:=u64(11) + for &value in pixels { + state=state*6364136223846793005+1442695040888963407 + value=u8(state>>56) + } + object_of :: proc(t:^testing.T,s:^store.Store,types:^asset.Types,pixels:[]u8) -> store.Hash { + built,error:=build(rgba8(pixels,SIDE,SIDE),{srgb=true,speed=.Fast},context.temp_allocator) + if !testing.expect_value(t,error,Error.None) {return {}} + doc:asset.Document + if !testing.expect_value(t,asset.texture_document(&doc,types,built_desc(&built,{9,9},nil),context.temp_allocator),asset.Error.None) {return {}} + defer asset.document_destroy(&doc) + bytes,_:=asset.document_bytes(&doc,context.temp_allocator) + object,put_error:=asset.object_put(s,bytes) + testing.expect_value(t,put_error,asset.Error.None) + return object + } + leaves_of :: proc(s:^store.Store,root:store.Hash) -> map[store.Hash]bool { + out:=make(map[store.Hash]bool,context.temp_allocator) + walk :: proc(s:^store.Store,address:store.Hash,out:^map[store.Hash]bool) { + buffer:=make([]u8,store.CHUNK_MAX,context.temp_allocator) + bytes,_:=store.chunk_read(s,address,buffer) + n,error:=store.node_parse(bytes) + if error!=.None {return} + for e in n.entries { + if n.level>0 {walk(s,e.address,out)} else {out[e.address]=true} + } + } + walk(s,root,&out) + return out + } + before:=object_of(t,&s,&types,pixels) + pixels[(SIDE/3*SIDE+SIDE/2)*4]~=0x40 + after:=object_of(t,&s,&types,pixels) + store.commit(&s) + testing.expect(t,before!=after) + old,new:=leaves_of(&s,before),leaves_of(&s,after) + changed:=0 + for address in new { + if !(address in old) {changed+=1} } + log.infof("one texel changed %d of %d chunks",changed,len(new)) + testing.expectf(t,changed>0 && changed<=len(new)/2+1,"%d of %d chunks changed",changed,len(new)) } diff --git a/cook/textures/textures.odin b/cook/textures/textures.odin index c40c09c9f..76fcd1bce 100644 --- a/cook/textures/textures.odin +++ b/cook/textures/textures.odin @@ -23,7 +23,7 @@ import mm "mirai:std/math" MAX_SOURCE_SIDE :: 1<<15 MAX_SOURCE_PIXELS :: 1<<28 // Changes when build would give other bytes for the same image and options. -BUILD_VERSION :: 6 +BUILD_VERSION :: 7 RDO_LAMBDA_DEFAULT :: 30 RDO_LAMBDA_MAX :: 100 // Texels of a level from which build encodes its blocks on a job system of @@ -320,14 +320,19 @@ built_desc :: proc(built:^Built,id:base.Asset_Id,source:^asset.Import_Info,layou // What decides the bytes build gives for an image: the settings hash an // import records, so a texture is built again when one of them changes. Fast // is another hash; Final is the hash of the settings alone. -settings_hash :: proc(options:Build_Options) -> u64 { +settings_hash :: proc(options:Build_Options) -> u64 {return settings_hash_of(options,encoders_id())} + +// settings_hash with the encoders named encoders, which count only where they make the bytes: +// texels stored uncompressed are the same whatever block encoder the build has. +@(private) +settings_hash_of :: proc(options:Build_Options,encoders:u64) -> u64 { kind:=u64(options.srgb)|u64(options.normal_map)<<1|u64(options.encoding)<<2|u64(options.green_down)<<4|u64(options.keep_precision)<<5|u64(BUILD_VERSION)<<8 hash:=base.hash_combine(kind,u64(transmute(u32)options.alpha_coverage)) if options.alpha_fill>0 {hash=base.hash_combine(hash,u64(transmute(u32)options.alpha_fill)|1<<32)} final:=options final.speed=.Final hash=base.hash_combine(hash,u64(rdo_lambda(final))) - hash=base.hash_combine(hash,encoders_id()) + if options.encoding==.Blocks {hash=base.hash_combine(hash,encoders)} if options.channels!={} { c:=options.channels packed:=u64(c.from[0])|u64(c.from[1])<<8|u64(c.from[2])<<16|u64(c.from[3])<<24|u64(transmute(u8)c.invert)<<32 diff --git a/demo/assets/animation_gallery/animation_gallery_label.ui b/demo/assets/animation_gallery/animation_gallery_label.ui index 779ea7d99..4abd8ab0d 100644 Binary files a/demo/assets/animation_gallery/animation_gallery_label.ui and b/demo/assets/animation_gallery/animation_gallery_label.ui differ diff --git a/demo/assets/animation_gallery/animation_gallery_panel.ui b/demo/assets/animation_gallery/animation_gallery_panel.ui index 28ebd5fbb..a212e33ba 100644 Binary files a/demo/assets/animation_gallery/animation_gallery_panel.ui and b/demo/assets/animation_gallery/animation_gallery_panel.ui differ diff --git a/demo/assets/animation_gallery/animation_hud.ui b/demo/assets/animation_gallery/animation_hud.ui index 8b9b6a2d6..efd985234 100644 Binary files a/demo/assets/animation_gallery/animation_hud.ui and b/demo/assets/animation_gallery/animation_hud.ui differ diff --git a/demo/assets/animation_gallery/rig_playground/chain.mesh b/demo/assets/animation_gallery/rig_playground/chain.mesh index ec7f5291d..614766196 100644 Binary files a/demo/assets/animation_gallery/rig_playground/chain.mesh and b/demo/assets/animation_gallery/rig_playground/chain.mesh differ diff --git a/demo/assets/blockout/column.mesh b/demo/assets/blockout/column.mesh index 6bb3d682b..bf1bcc93e 100644 Binary files a/demo/assets/blockout/column.mesh and b/demo/assets/blockout/column.mesh differ diff --git a/demo/assets/blockout/tree.mesh b/demo/assets/blockout/tree.mesh index 6b1e50736..5f75e28e9 100644 Binary files a/demo/assets/blockout/tree.mesh and b/demo/assets/blockout/tree.mesh differ diff --git a/demo/assets/characters/jelly/jelly.gltf.import b/demo/assets/characters/jelly/jelly.gltf.import deleted file mode 100644 index e3bf74d2a..000000000 --- a/demo/assets/characters/jelly/jelly.gltf.import +++ /dev/null @@ -1,14 +0,0 @@ -{ - "version": 4, - "settings": {}, - "folder": "jelly", - "inputs": ["jelly.bin"], - "outputs": [ - {"kind": "SCEN", "name": "jelly", "index": 0, "id": "107c9b8f4bf9dfe8936235eae8e042a6", "written": "29d766ccf9f00d54"}, - {"kind": "MAT", "name": "Candy", "index": 0, "id": "a18adcad4fedf21125038a819d27acd2"}, - {"kind": "SKEL", "name": "Jelly bones", "index": 0, "id": "dae452ce18d59e780ff745f7d1b43697", "written": "c2aa6b07e094c55c"}, - {"kind": "MESH", "name": "Party Jelly", "index": 0, "id": "25a3e22ecb9c265a1614515f868efc52", "part": true}, - {"kind": "MESH", "name": "Party Jelly (1)", "index": 1, "id": "1c00bfb5bceb2619036b114104f42040", "part": true}, - {"kind": "ANIM", "name": "jelly", "index": 0, "id": "0d555b0da56949fb3cdbdf41410432f4", "written": "85b9467c73dda666"} - ] -} diff --git a/demo/assets/characters/jelly/jelly/Candy.mat b/demo/assets/characters/jelly/jelly/Candy.mat index d5b1a1ca2..af2101cfe 100644 Binary files a/demo/assets/characters/jelly/jelly/Candy.mat and b/demo/assets/characters/jelly/jelly/Candy.mat differ diff --git a/demo/assets/characters/jelly/jelly/Jelly bones.skeleton b/demo/assets/characters/jelly/jelly/Jelly bones.skeleton index d5c8476e6..9bdd31fe1 100644 Binary files a/demo/assets/characters/jelly/jelly/Jelly bones.skeleton and b/demo/assets/characters/jelly/jelly/Jelly bones.skeleton differ diff --git a/demo/assets/characters/jelly/jelly/jelly.anim b/demo/assets/characters/jelly/jelly/jelly.anim index d2d935c57..a9ebfcb5b 100644 Binary files a/demo/assets/characters/jelly/jelly/jelly.anim and b/demo/assets/characters/jelly/jelly/jelly.anim differ diff --git a/demo/assets/characters/jelly/jelly/jelly.scene b/demo/assets/characters/jelly/jelly/jelly.scene index d98328c63..60b261454 100644 Binary files a/demo/assets/characters/jelly/jelly/jelly.scene and b/demo/assets/characters/jelly/jelly/jelly.scene differ diff --git a/demo/assets/characters/mirai/Armature.skeleton b/demo/assets/characters/mirai/Armature.skeleton index a2090941d..9fd0147f3 100644 Binary files a/demo/assets/characters/mirai/Armature.skeleton and b/demo/assets/characters/mirai/Armature.skeleton differ diff --git a/demo/assets/characters/mirai/BakedMaterial.mat b/demo/assets/characters/mirai/BakedMaterial.mat index f8d00efce..2b034247a 100644 Binary files a/demo/assets/characters/mirai/BakedMaterial.mat and b/demo/assets/characters/mirai/BakedMaterial.mat differ diff --git a/demo/assets/characters/mirai/Mirai borde luminoso.mat b/demo/assets/characters/mirai/Mirai borde luminoso.mat index 18c48e543..86060626c 100644 Binary files a/demo/assets/characters/mirai/Mirai borde luminoso.mat and b/demo/assets/characters/mirai/Mirai borde luminoso.mat differ diff --git a/demo/assets/characters/mirai/Mirai holograma violeta.mat b/demo/assets/characters/mirai/Mirai holograma violeta.mat index c7d890f9b..fd5d5836c 100644 Binary files a/demo/assets/characters/mirai/Mirai holograma violeta.mat and b/demo/assets/characters/mirai/Mirai holograma violeta.mat differ diff --git a/demo/assets/characters/mirai/mirai.anim b/demo/assets/characters/mirai/mirai.anim index 9a99c23ab..6423863a0 100644 Binary files a/demo/assets/characters/mirai/mirai.anim and b/demo/assets/characters/mirai/mirai.anim differ diff --git a/demo/assets/characters/mirai/mirai.glb.import b/demo/assets/characters/mirai/mirai.glb.import deleted file mode 100644 index bcd237449..000000000 --- a/demo/assets/characters/mirai/mirai.glb.import +++ /dev/null @@ -1,17 +0,0 @@ -{ - "version": 4, - "settings": {}, - "outputs": [ - {"kind": "SCEN", "name": "mirai", "index": 0, "id": "94f640750cf5246f1c0728cc4cd2a07c", "written": "ebe25fa688c7e0a5"}, - {"kind": "TEX", "name": "texture_0", "index": 0, "id": "a3d324f2a5454495463d13c1ebad2c84", "part": true}, - {"kind": "TEX", "name": "texture_0_metallic_roughness", "index": 1, "id": "08abe809e118473bb530ac037696309e", "part": true}, - {"kind": "MAT", "name": "BakedMaterial", "index": 0, "id": "f1d4dfad902a16a7afc56bb3202b1d94"}, - {"kind": "MAT", "name": "Mirai holograma violeta", "index": 1, "id": "dbb521d6ca206ef049b42c4fad14b7d5"}, - {"kind": "MAT", "name": "Mirai borde luminoso", "index": 2, "id": "bbfe2f2406e2d1a5aaf297d5d2c8a990"}, - {"kind": "SKEL", "name": "Armature", "index": 0, "id": "0208471f893327bdcf8bbaa108a26121", "written": "053cbdb8eb5d0e21"}, - {"kind": "MESH", "name": "output_unwrapped", "index": 0, "id": "f0c947702bb8141dc1d27444a341d52c", "part": true}, - {"kind": "MESH", "name": "Cinta orbital Left", "index": 1, "id": "8d6c8aa43c816332bca9f38c1dd0d17f", "part": true}, - {"kind": "MESH", "name": "Cinta orbital Right", "index": 2, "id": "78744c9faee4ebd82e0f9ca443b72656", "part": true}, - {"kind": "ANIM", "name": "mirai", "index": 0, "id": "f3016e0fca41fca429b049307bde6f9f", "written": "feb4aff242f81c75"} - ] -} diff --git a/demo/assets/characters/mirai/mirai.scene b/demo/assets/characters/mirai/mirai.scene index e8f6b41c7..95b570288 100644 Binary files a/demo/assets/characters/mirai/mirai.scene and b/demo/assets/characters/mirai/mirai.scene differ diff --git a/demo/assets/characters/mirai/mirai_gallery.anim b/demo/assets/characters/mirai/mirai_gallery.anim index bae2b7944..0f19539e0 100644 Binary files a/demo/assets/characters/mirai/mirai_gallery.anim and b/demo/assets/characters/mirai/mirai_gallery.anim differ diff --git a/demo/assets/characters/robot/glb/13 joint robot rig.skeleton b/demo/assets/characters/robot/glb/13 joint robot rig.skeleton index 01d94b787..cb370066a 100644 Binary files a/demo/assets/characters/robot/glb/13 joint robot rig.skeleton and b/demo/assets/characters/robot/glb/13 joint robot rig.skeleton differ diff --git a/demo/assets/characters/robot/glb/robot blue.mat b/demo/assets/characters/robot/glb/robot blue.mat index fddcf4b72..ff8fe6639 100644 Binary files a/demo/assets/characters/robot/glb/robot blue.mat and b/demo/assets/characters/robot/glb/robot blue.mat differ diff --git a/demo/assets/characters/robot/glb/robot gold.mat b/demo/assets/characters/robot/glb/robot gold.mat index 7f974acc6..cf742e2a0 100644 Binary files a/demo/assets/characters/robot/glb/robot gold.mat and b/demo/assets/characters/robot/glb/robot gold.mat differ diff --git a/demo/assets/characters/robot/glb/robot joints.mat b/demo/assets/characters/robot/glb/robot joints.mat index 2d631e4c4..c30a6a51c 100644 Binary files a/demo/assets/characters/robot/glb/robot joints.mat and b/demo/assets/characters/robot/glb/robot joints.mat differ diff --git a/demo/assets/characters/robot/glb/robot visor.mat b/demo/assets/characters/robot/glb/robot visor.mat index b0b20ac1b..25ab5b8bc 100644 Binary files a/demo/assets/characters/robot/glb/robot visor.mat and b/demo/assets/characters/robot/glb/robot visor.mat differ diff --git a/demo/assets/characters/robot/glb/robot.scene b/demo/assets/characters/robot/glb/robot.scene index f250d9609..9870a38e4 100644 Binary files a/demo/assets/characters/robot/glb/robot.scene and b/demo/assets/characters/robot/glb/robot.scene differ diff --git a/demo/assets/characters/robot/gltf/13 joint robot rig.skeleton b/demo/assets/characters/robot/gltf/13 joint robot rig.skeleton index 010d1fffd..5030923a1 100644 Binary files a/demo/assets/characters/robot/gltf/13 joint robot rig.skeleton and b/demo/assets/characters/robot/gltf/13 joint robot rig.skeleton differ diff --git a/demo/assets/characters/robot/gltf/robot blue.mat b/demo/assets/characters/robot/gltf/robot blue.mat index 59fc93154..b0f24d4fc 100644 Binary files a/demo/assets/characters/robot/gltf/robot blue.mat and b/demo/assets/characters/robot/gltf/robot blue.mat differ diff --git a/demo/assets/characters/robot/gltf/robot gold.mat b/demo/assets/characters/robot/gltf/robot gold.mat index 0c705b0d5..f92bc8ad8 100644 Binary files a/demo/assets/characters/robot/gltf/robot gold.mat and b/demo/assets/characters/robot/gltf/robot gold.mat differ diff --git a/demo/assets/characters/robot/gltf/robot joints.mat b/demo/assets/characters/robot/gltf/robot joints.mat index ac41a164e..fecea0b35 100644 Binary files a/demo/assets/characters/robot/gltf/robot joints.mat and b/demo/assets/characters/robot/gltf/robot joints.mat differ diff --git a/demo/assets/characters/robot/gltf/robot visor.mat b/demo/assets/characters/robot/gltf/robot visor.mat index 7e6f6b4d2..d06d8bba2 100644 Binary files a/demo/assets/characters/robot/gltf/robot visor.mat and b/demo/assets/characters/robot/gltf/robot visor.mat differ diff --git a/demo/assets/characters/robot/gltf/robot.scene b/demo/assets/characters/robot/gltf/robot.scene index 8c23cd3a6..a1d6d9549 100644 Binary files a/demo/assets/characters/robot/gltf/robot.scene and b/demo/assets/characters/robot/gltf/robot.scene differ diff --git a/demo/assets/characters/robot/robot.glb.import b/demo/assets/characters/robot/robot.glb.import deleted file mode 100644 index 86e0badfc..000000000 --- a/demo/assets/characters/robot/robot.glb.import +++ /dev/null @@ -1,14 +0,0 @@ -{ - "version": 4, - "settings": {}, - "folder": "glb", - "outputs": [ - {"kind": "SCEN", "name": "robot", "index": 0, "id": "e431738105898dd368735a6ebb1d3c19", "written": "776c222659264649"}, - {"kind": "MAT", "name": "robot blue", "index": 0, "id": "1c5c7311b02f9c27a26158a5dfbbfe47"}, - {"kind": "MAT", "name": "robot gold", "index": 1, "id": "985941907b0208e6eaef9ca10b6db8ba"}, - {"kind": "MAT", "name": "robot joints", "index": 2, "id": "d9b762e720d363036a57efa7b616eaef"}, - {"kind": "MAT", "name": "robot visor", "index": 3, "id": "ae20beadede617fdd239a5a6ad23eb83"}, - {"kind": "SKEL", "name": "13 joint robot rig", "index": 0, "id": "83d297850b4eccf18f4225970eff51fb", "written": "44640dab74c565fc"}, - {"kind": "MESH", "name": "Mirai robot", "index": 0, "id": "941beb67913839d9f4d4e05528cb0c80", "part": true} - ] -} diff --git a/demo/assets/characters/robot/robot.gltf.import b/demo/assets/characters/robot/robot.gltf.import deleted file mode 100644 index 8a06a58ba..000000000 --- a/demo/assets/characters/robot/robot.gltf.import +++ /dev/null @@ -1,15 +0,0 @@ -{ - "version": 4, - "settings": {}, - "folder": "gltf", - "inputs": ["robot.bin"], - "outputs": [ - {"kind": "SCEN", "name": "robot", "index": 0, "id": "b0f0241df6e014392af304b56cf59eab", "written": "84ec3e2050ad1320"}, - {"kind": "MAT", "name": "robot blue", "index": 0, "id": "c67757fb8b041b0d6b6178fe958bd91e"}, - {"kind": "MAT", "name": "robot gold", "index": 1, "id": "feebc67eaeb9081ec3d1f1edac208b2a"}, - {"kind": "MAT", "name": "robot joints", "index": 2, "id": "a453277713f405f86fd3dc27c2506977"}, - {"kind": "MAT", "name": "robot visor", "index": 3, "id": "0bf1bd3d54bcb875b34957598640c69d"}, - {"kind": "SKEL", "name": "13 joint robot rig", "index": 0, "id": "cf5b1fe881d020461927fe9b85c42136", "written": "327b76c5e7297716"}, - {"kind": "MESH", "name": "Mirai robot", "index": 0, "id": "3a94f8ad990aede6566ad2af8010bb23", "part": true} - ] -} diff --git a/demo/assets/characters/spider/spider.mesh b/demo/assets/characters/spider/spider.mesh index 20bc2f56a..a3e63c2fc 100644 Binary files a/demo/assets/characters/spider/spider.mesh and b/demo/assets/characters/spider/spider.mesh differ diff --git a/demo/assets/characters/tall_mirai/tall_mirai.anim b/demo/assets/characters/tall_mirai/tall_mirai.anim index f4a038aee..fe4627c77 100644 Binary files a/demo/assets/characters/tall_mirai/tall_mirai.anim and b/demo/assets/characters/tall_mirai/tall_mirai.anim differ diff --git a/demo/assets/characters/tall_mirai/tall_mirai.mesh b/demo/assets/characters/tall_mirai/tall_mirai.mesh index 2fa368307..5ad3f7aa7 100644 Binary files a/demo/assets/characters/tall_mirai/tall_mirai.mesh and b/demo/assets/characters/tall_mirai/tall_mirai.mesh differ diff --git a/demo/assets/church/church_glass_color.png.import b/demo/assets/church/church_glass_color.png.import deleted file mode 100644 index 267a83f6c..000000000 --- a/demo/assets/church/church_glass_color.png.import +++ /dev/null @@ -1,9 +0,0 @@ -{ - "version": 4, - "settings": { - "image": "Color" - }, - "outputs": [ - {"kind": "TEX", "name": "church_glass_color", "index": 0, "id": "2767edae7841cea315b1e40d922c9950"} - ] -} diff --git a/demo/assets/deform/blob.mesh b/demo/assets/deform/blob.mesh index bcc2f975e..6f4d8fd35 100644 Binary files a/demo/assets/deform/blob.mesh and b/demo/assets/deform/blob.mesh differ diff --git a/demo/assets/deform/flag.mesh b/demo/assets/deform/flag.mesh index 36eead593..75734a034 100644 Binary files a/demo/assets/deform/flag.mesh and b/demo/assets/deform/flag.mesh differ diff --git a/demo/assets/deform/hose.mesh b/demo/assets/deform/hose.mesh index 3c8c3cd5d..2d980df71 100644 Binary files a/demo/assets/deform/hose.mesh and b/demo/assets/deform/hose.mesh differ diff --git a/demo/assets/deform/membrane.mesh b/demo/assets/deform/membrane.mesh index 8e33b35fc..3dd3496aa 100644 Binary files a/demo/assets/deform/membrane.mesh and b/demo/assets/deform/membrane.mesh differ diff --git a/demo/assets/deform/post.mesh b/demo/assets/deform/post.mesh index 09c07400e..fd5a711aa 100644 Binary files a/demo/assets/deform/post.mesh and b/demo/assets/deform/post.mesh differ diff --git a/demo/assets/deform/reed.mesh b/demo/assets/deform/reed.mesh index 3f7fce6f2..8bf948108 100644 Binary files a/demo/assets/deform/reed.mesh and b/demo/assets/deform/reed.mesh differ diff --git a/demo/assets/dig/dig_hud.ui b/demo/assets/dig/dig_hud.ui index 8efd73f34..25640ff5b 100644 Binary files a/demo/assets/dig/dig_hud.ui and b/demo/assets/dig/dig_hud.ui differ diff --git a/demo/assets/foliage_extreme/tree_broad.obj.import b/demo/assets/foliage_extreme/tree_broad.obj.import deleted file mode 100644 index b3a3c1f1d..000000000 --- a/demo/assets/foliage_extreme/tree_broad.obj.import +++ /dev/null @@ -1,12 +0,0 @@ -{ - "version": 4, - "settings": {}, - "folder": "tree_broad", - "inputs": ["trees.mtl"], - "outputs": [ - {"kind": "SCEN", "name": "tree_broad", "index": 0, "id": "73668be537ffad086162e2c98e1c79f6", "written": "133223e7d8fe9922"}, - {"kind": "MAT", "name": "bark", "index": 0, "id": "031b5c900785b6de16ce2ab9904c2288"}, - {"kind": "MAT", "name": "leaf", "index": 1, "id": "2012dad11d76c044f98348272649af04"}, - {"kind": "MESH", "name": "tree_broad", "index": 0, "id": "f745c4a330cd0cdd5a593bdf377d89ca", "part": true} - ] -} diff --git a/demo/assets/foliage_extreme/tree_broad/bark.mat b/demo/assets/foliage_extreme/tree_broad/bark.mat index a8007cf7a..902e49f17 100644 Binary files a/demo/assets/foliage_extreme/tree_broad/bark.mat and b/demo/assets/foliage_extreme/tree_broad/bark.mat differ diff --git a/demo/assets/foliage_extreme/tree_broad/leaf.mat b/demo/assets/foliage_extreme/tree_broad/leaf.mat index d36f14811..dae02bf03 100644 Binary files a/demo/assets/foliage_extreme/tree_broad/leaf.mat and b/demo/assets/foliage_extreme/tree_broad/leaf.mat differ diff --git a/demo/assets/foliage_extreme/tree_broad/tree_broad.scene b/demo/assets/foliage_extreme/tree_broad/tree_broad.scene index 81420ae8d..7dd1b9b8a 100644 Binary files a/demo/assets/foliage_extreme/tree_broad/tree_broad.scene and b/demo/assets/foliage_extreme/tree_broad/tree_broad.scene differ diff --git a/demo/assets/foliage_extreme/tree_bush.obj.import b/demo/assets/foliage_extreme/tree_bush.obj.import deleted file mode 100644 index 1d26322b6..000000000 --- a/demo/assets/foliage_extreme/tree_bush.obj.import +++ /dev/null @@ -1,12 +0,0 @@ -{ - "version": 4, - "settings": {}, - "folder": "tree_bush", - "inputs": ["trees.mtl"], - "outputs": [ - {"kind": "SCEN", "name": "tree_bush", "index": 0, "id": "317c6ad95fead4485707ae3231936f9b", "written": "4be2430d2ecb8870"}, - {"kind": "MAT", "name": "bark", "index": 0, "id": "b96df7055aceeb7248fcde10158da5cc"}, - {"kind": "MAT", "name": "leaf", "index": 1, "id": "fbcbc189b6096b232b1c8ff070f19043"}, - {"kind": "MESH", "name": "tree_bush", "index": 0, "id": "61539a5b43404159745748d7814cab3c", "part": true} - ] -} diff --git a/demo/assets/foliage_extreme/tree_bush/bark.mat b/demo/assets/foliage_extreme/tree_bush/bark.mat index 83120a9af..fb2a912e6 100644 Binary files a/demo/assets/foliage_extreme/tree_bush/bark.mat and b/demo/assets/foliage_extreme/tree_bush/bark.mat differ diff --git a/demo/assets/foliage_extreme/tree_bush/leaf.mat b/demo/assets/foliage_extreme/tree_bush/leaf.mat index b80c24066..ffcdc8483 100644 Binary files a/demo/assets/foliage_extreme/tree_bush/leaf.mat and b/demo/assets/foliage_extreme/tree_bush/leaf.mat differ diff --git a/demo/assets/foliage_extreme/tree_bush/tree_bush.scene b/demo/assets/foliage_extreme/tree_bush/tree_bush.scene index 45d4bd89d..bdd598125 100644 Binary files a/demo/assets/foliage_extreme/tree_bush/tree_bush.scene and b/demo/assets/foliage_extreme/tree_bush/tree_bush.scene differ diff --git a/demo/assets/foliage_extreme/tree_tall.obj.import b/demo/assets/foliage_extreme/tree_tall.obj.import deleted file mode 100644 index 15ed86ade..000000000 --- a/demo/assets/foliage_extreme/tree_tall.obj.import +++ /dev/null @@ -1,12 +0,0 @@ -{ - "version": 4, - "settings": {}, - "folder": "tree_tall", - "inputs": ["trees.mtl"], - "outputs": [ - {"kind": "SCEN", "name": "tree_tall", "index": 0, "id": "10f19381013cd286f491234605c5ca3a", "written": "bcedec98648a1309"}, - {"kind": "MAT", "name": "bark", "index": 0, "id": "1e0d094f208a1c4fd1832e6810416f45"}, - {"kind": "MAT", "name": "leaf", "index": 1, "id": "d447cd3c68bfe38f537484e2c968b98e"}, - {"kind": "MESH", "name": "tree_tall", "index": 0, "id": "08c37e88a5a8125dd0c6c67b2fb2cc40", "part": true} - ] -} diff --git a/demo/assets/foliage_extreme/tree_tall/bark.mat b/demo/assets/foliage_extreme/tree_tall/bark.mat index 72f649d85..f7264960e 100644 Binary files a/demo/assets/foliage_extreme/tree_tall/bark.mat and b/demo/assets/foliage_extreme/tree_tall/bark.mat differ diff --git a/demo/assets/foliage_extreme/tree_tall/leaf.mat b/demo/assets/foliage_extreme/tree_tall/leaf.mat index 52058ca2a..3b7550b32 100644 Binary files a/demo/assets/foliage_extreme/tree_tall/leaf.mat and b/demo/assets/foliage_extreme/tree_tall/leaf.mat differ diff --git a/demo/assets/foliage_extreme/tree_tall/tree_tall.scene b/demo/assets/foliage_extreme/tree_tall/tree_tall.scene index 8ba48cae7..27b13601d 100644 Binary files a/demo/assets/foliage_extreme/tree_tall/tree_tall.scene and b/demo/assets/foliage_extreme/tree_tall/tree_tall.scene differ diff --git a/demo/assets/forest/bush.gltf.import b/demo/assets/forest/bush.gltf.import deleted file mode 100644 index 8844d6e98..000000000 --- a/demo/assets/forest/bush.gltf.import +++ /dev/null @@ -1,13 +0,0 @@ -{ - "version": 4, - "settings": {}, - "folder": "bush", - "inputs": ["bush.bin", "leaves_color.png"], - "outputs": [ - {"kind": "SCEN", "name": "bush", "index": 0, "id": "48eed1e5335b875bef27025acf93953a", "written": "2c69b74538ea5d42"}, - {"kind": "TEX", "name": "leaves_color", "index": 0, "id": "33f31fe1b5c654da025e160204ebcd70"}, - {"kind": "MAT", "name": "Bark", "index": 0, "id": "f9bebb3af1fed4d0e8103c0e0387b11a"}, - {"kind": "MAT", "name": "Leaves", "index": 1, "id": "e26eb420ab436a838111b0626b6a8a79"}, - {"kind": "MESH", "name": "Bush", "index": 0, "id": "88ceb31480f8e2c656ca94fb40dbda3d", "part": true} - ] -} diff --git a/demo/assets/forest/bush/Bark.mat b/demo/assets/forest/bush/Bark.mat index b84c66515..f27004db1 100644 Binary files a/demo/assets/forest/bush/Bark.mat and b/demo/assets/forest/bush/Bark.mat differ diff --git a/demo/assets/forest/bush/Leaves.mat b/demo/assets/forest/bush/Leaves.mat index ff2ba3939..e69154147 100644 Binary files a/demo/assets/forest/bush/Leaves.mat and b/demo/assets/forest/bush/Leaves.mat differ diff --git a/demo/assets/forest/bush/bush.scene b/demo/assets/forest/bush/bush.scene index d38f76ece..6fb8b6419 100644 Binary files a/demo/assets/forest/bush/bush.scene and b/demo/assets/forest/bush/bush.scene differ diff --git a/demo/assets/gallery/leaves_color.png.import b/demo/assets/gallery/leaves_color.png.import deleted file mode 100644 index 16408fbd6..000000000 --- a/demo/assets/gallery/leaves_color.png.import +++ /dev/null @@ -1,10 +0,0 @@ -{ - "version": 4, - "settings": { - "image": "Color", - "alpha_coverage": 0.5 - }, - "outputs": [ - {"kind": "TEX", "name": "leaves_color", "index": 0, "id": "df49540f1f9dbcf6fe2e612ea427757e"} - ] -} diff --git a/demo/assets/gallery/plant.gltf.import b/demo/assets/gallery/plant.gltf.import deleted file mode 100644 index 936d40f58..000000000 --- a/demo/assets/gallery/plant.gltf.import +++ /dev/null @@ -1,14 +0,0 @@ -{ - "version": 4, - "settings": {}, - "folder": "plant", - "inputs": ["leaves_color.png"], - "outputs": [ - {"kind": "SCEN", "name": "plant", "index": 0, "id": "fc120101f1ddb302e135784184f05931", "written": "e2cbe15d5a77ca78"}, - {"kind": "TEX", "name": "leaves_color", "index": 0, "id": "dba1b6bd3e8cde629938d41dc8934d02"}, - {"kind": "MAT", "name": "Terracotta", "index": 0, "id": "e2a8b0fcb9401808520b370016df35c9"}, - {"kind": "MAT", "name": "Soil", "index": 1, "id": "9337601926ea70572a4442c984759e15"}, - {"kind": "MAT", "name": "Leaves", "index": 2, "id": "7e248f2df1c8c849ff972c9a1ce95fb5"}, - {"kind": "MESH", "name": "Plant", "index": 0, "id": "72d9220e763e42b1da3107e0e5256b90", "part": true} - ] -} diff --git a/demo/assets/gallery/plant/Leaves.mat b/demo/assets/gallery/plant/Leaves.mat index ba5db6287..50b94a6b2 100644 Binary files a/demo/assets/gallery/plant/Leaves.mat and b/demo/assets/gallery/plant/Leaves.mat differ diff --git a/demo/assets/gallery/plant/Soil.mat b/demo/assets/gallery/plant/Soil.mat index 7ffefabd4..cb95ca44e 100644 Binary files a/demo/assets/gallery/plant/Soil.mat and b/demo/assets/gallery/plant/Soil.mat differ diff --git a/demo/assets/gallery/plant/Terracotta.mat b/demo/assets/gallery/plant/Terracotta.mat index 83564368f..1b26ab537 100644 Binary files a/demo/assets/gallery/plant/Terracotta.mat and b/demo/assets/gallery/plant/Terracotta.mat differ diff --git a/demo/assets/gallery/plant/plant.scene b/demo/assets/gallery/plant/plant.scene index 6bf26f02a..4b6f1bf27 100644 Binary files a/demo/assets/gallery/plant/plant.scene and b/demo/assets/gallery/plant/plant.scene differ diff --git a/demo/assets/gallery/vase.obj.import b/demo/assets/gallery/vase.obj.import deleted file mode 100644 index 45b521ad0..000000000 --- a/demo/assets/gallery/vase.obj.import +++ /dev/null @@ -1,13 +0,0 @@ -{ - "version": 4, - "settings": {}, - "folder": "vase", - "inputs": ["vase.mtl", "vase_color.png"], - "outputs": [ - {"kind": "SCEN", "name": "vase", "index": 0, "id": "f430e11663e1ae229bf1511ac0c7f82d", "written": "ec533d99744379fe"}, - {"kind": "TEX", "name": "vase_color", "index": 0, "id": "f8b0959094ef4d6757a87f85ec92a493"}, - {"kind": "MAT", "name": "Ceramic", "index": 0, "id": "757da5a655983f642167f358f7add279"}, - {"kind": "MAT", "name": "Gilding", "index": 1, "id": "be0bbaf3f4f7c9fc87a15c7d08b0bb18"}, - {"kind": "MESH", "name": "Vase", "index": 0, "id": "0d6f52247150edd52ac17287d54e7bf9", "part": true} - ] -} diff --git a/demo/assets/gallery/vase/Ceramic.mat b/demo/assets/gallery/vase/Ceramic.mat index 4f72a3dad..cbfb9c58d 100644 Binary files a/demo/assets/gallery/vase/Ceramic.mat and b/demo/assets/gallery/vase/Ceramic.mat differ diff --git a/demo/assets/gallery/vase/Gilding.mat b/demo/assets/gallery/vase/Gilding.mat index 6f5884899..ce50bca63 100644 Binary files a/demo/assets/gallery/vase/Gilding.mat and b/demo/assets/gallery/vase/Gilding.mat differ diff --git a/demo/assets/gallery/vase/vase.scene b/demo/assets/gallery/vase/vase.scene index 73693aeea..e103c84c5 100644 Binary files a/demo/assets/gallery/vase/vase.scene and b/demo/assets/gallery/vase/vase.scene differ diff --git a/demo/assets/gallery/vase_color.png.import b/demo/assets/gallery/vase_color.png.import deleted file mode 100644 index 88211ba33..000000000 --- a/demo/assets/gallery/vase_color.png.import +++ /dev/null @@ -1,9 +0,0 @@ -{ - "version": 4, - "settings": { - "image": "Color" - }, - "outputs": [ - {"kind": "TEX", "name": "vase_color", "index": 0, "id": "c94299d672a2e4371f7a5374961b418c"} - ] -} diff --git a/demo/assets/interface/button.ui b/demo/assets/interface/button.ui index 95bbefdb5..696788c3d 100644 Binary files a/demo/assets/interface/button.ui and b/demo/assets/interface/button.ui differ diff --git a/demo/assets/interface/dialogue.ui b/demo/assets/interface/dialogue.ui index 9d57302b7..2f39ae27f 100644 Binary files a/demo/assets/interface/dialogue.ui and b/demo/assets/interface/dialogue.ui differ diff --git a/demo/assets/interface/hud.ui b/demo/assets/interface/hud.ui index fc82f583e..f6e96a15e 100644 Binary files a/demo/assets/interface/hud.ui and b/demo/assets/interface/hud.ui differ diff --git a/demo/assets/interface/inventory.ui b/demo/assets/interface/inventory.ui index 2a62f4d80..222bd65cc 100644 Binary files a/demo/assets/interface/inventory.ui and b/demo/assets/interface/inventory.ui differ diff --git a/demo/assets/interface/menu.ui b/demo/assets/interface/menu.ui index 680182fb3..f52bd5520 100644 Binary files a/demo/assets/interface/menu.ui and b/demo/assets/interface/menu.ui differ diff --git a/demo/assets/interface/player_hud.ui b/demo/assets/interface/player_hud.ui index 9990d12ed..da467a9f6 100644 Binary files a/demo/assets/interface/player_hud.ui and b/demo/assets/interface/player_hud.ui differ diff --git a/demo/assets/interface/tag.ui b/demo/assets/interface/tag.ui index 79b5ff244..7702002aa 100644 Binary files a/demo/assets/interface/tag.ui and b/demo/assets/interface/tag.ui differ diff --git a/demo/assets/physics_gallery/bricks.mesh b/demo/assets/physics_gallery/bricks.mesh index 28a1a60ff..5fb4ac10d 100644 Binary files a/demo/assets/physics_gallery/bricks.mesh and b/demo/assets/physics_gallery/bricks.mesh differ diff --git a/demo/assets/physics_gallery/hierarchy.mesh b/demo/assets/physics_gallery/hierarchy.mesh index 6255a02ad..5bed7be81 100644 Binary files a/demo/assets/physics_gallery/hierarchy.mesh and b/demo/assets/physics_gallery/hierarchy.mesh differ diff --git a/demo/assets/physics_gallery/mixed_bricks.mesh b/demo/assets/physics_gallery/mixed_bricks.mesh index c1652d020..398cce786 100644 Binary files a/demo/assets/physics_gallery/mixed_bricks.mesh and b/demo/assets/physics_gallery/mixed_bricks.mesh differ diff --git a/demo/assets/physics_gallery/physics_gallery_label.ui b/demo/assets/physics_gallery/physics_gallery_label.ui index 6ead30b5b..aefec2eb1 100644 Binary files a/demo/assets/physics_gallery/physics_gallery_label.ui and b/demo/assets/physics_gallery/physics_gallery_label.ui differ diff --git a/demo/assets/physics_gallery/physics_hud.ui b/demo/assets/physics_gallery/physics_hud.ui index 2ab2396af..e5d2569af 100644 Binary files a/demo/assets/physics_gallery/physics_hud.ui and b/demo/assets/physics_gallery/physics_hud.ui differ diff --git a/demo/assets/physics_gallery/ragdoll/rig.mesh b/demo/assets/physics_gallery/ragdoll/rig.mesh index b3bb4d995..87fe45482 100644 Binary files a/demo/assets/physics_gallery/ragdoll/rig.mesh and b/demo/assets/physics_gallery/ragdoll/rig.mesh differ diff --git a/demo/assets/physics_gallery/recorded.physics_cache b/demo/assets/physics_gallery/recorded.physics_cache index 87e18e976..9611e1834 100644 Binary files a/demo/assets/physics_gallery/recorded.physics_cache and b/demo/assets/physics_gallery/recorded.physics_cache differ diff --git a/demo/assets/physics_gallery/wall.mesh b/demo/assets/physics_gallery/wall.mesh index ca82211c9..d26c13fb4 100644 Binary files a/demo/assets/physics_gallery/wall.mesh and b/demo/assets/physics_gallery/wall.mesh differ diff --git a/demo/assets/portals/portal_hud.ui b/demo/assets/portals/portal_hud.ui index 646bf3eb3..cc1d7c4c3 100644 Binary files a/demo/assets/portals/portal_hud.ui and b/demo/assets/portals/portal_hud.ui differ diff --git a/demo/assets/raft/raft_hud.ui b/demo/assets/raft/raft_hud.ui index d94f2bc07..8707d56a1 100644 Binary files a/demo/assets/raft/raft_hud.ui and b/demo/assets/raft/raft_hud.ui differ diff --git a/demo/assets/showcase/hud.ui b/demo/assets/showcase/hud.ui index 613c1e4a8..9bb4029f0 100644 Binary files a/demo/assets/showcase/hud.ui and b/demo/assets/showcase/hud.ui differ diff --git a/demo/assets/showcase/hum.wav.import b/demo/assets/showcase/hum.wav.import deleted file mode 100644 index b4a7f0988..000000000 --- a/demo/assets/showcase/hum.wav.import +++ /dev/null @@ -1,7 +0,0 @@ -{ - "version": 4, - "settings": {}, - "outputs": [ - {"kind": "SND", "name": "hum", "index": 0, "id": "7f27bbcfa7a4b5fac01753762f745b26"} - ] -} diff --git a/demo/assets/showcase/sign.ui b/demo/assets/showcase/sign.ui index 4b3eafcd0..ff4b2c536 100644 Binary files a/demo/assets/showcase/sign.ui and b/demo/assets/showcase/sign.ui differ diff --git a/demo/assets/sky/sky_day.hdr.import b/demo/assets/sky/sky_day.hdr.import deleted file mode 100644 index 12015d660..000000000 --- a/demo/assets/sky/sky_day.hdr.import +++ /dev/null @@ -1,7 +0,0 @@ -{ - "version": 4, - "settings": {}, - "outputs": [ - {"kind": "TEX", "name": "sky_day", "index": 0, "id": "39808e0def899c47a24a89a3fab89e64"} - ] -} diff --git a/demo/assets/sky/sky_night.jpg.import b/demo/assets/sky/sky_night.jpg.import deleted file mode 100644 index e2ad2c255..000000000 --- a/demo/assets/sky/sky_night.jpg.import +++ /dev/null @@ -1,9 +0,0 @@ -{ - "version": 4, - "settings": { - "image": "Color" - }, - "outputs": [ - {"kind": "TEX", "name": "sky_night", "index": 0, "id": "8a6329018eb7b53a0215bd4bdbf5a2c9"} - ] -} diff --git a/demo/assets/sky/sky_overcast.jpg.import b/demo/assets/sky/sky_overcast.jpg.import deleted file mode 100644 index 7d1d15e4d..000000000 --- a/demo/assets/sky/sky_overcast.jpg.import +++ /dev/null @@ -1,9 +0,0 @@ -{ - "version": 4, - "settings": { - "image": "Color" - }, - "outputs": [ - {"kind": "TEX", "name": "sky_overcast", "index": 0, "id": "88a3a1a16a759ed20966d1c3ff54ada1"} - ] -} diff --git a/demo/assets/sky/sky_sunset.jpg.import b/demo/assets/sky/sky_sunset.jpg.import deleted file mode 100644 index 07433cbec..000000000 --- a/demo/assets/sky/sky_sunset.jpg.import +++ /dev/null @@ -1,9 +0,0 @@ -{ - "version": 4, - "settings": { - "image": "Color" - }, - "outputs": [ - {"kind": "TEX", "name": "sky_sunset", "index": 0, "id": "7b819d061e0487f04a17587b8ff0bce1"} - ] -} diff --git a/demo/assets/surfaces/bricks/bricks_color.png.import b/demo/assets/surfaces/bricks/bricks_color.png.import deleted file mode 100644 index 40ad27f62..000000000 --- a/demo/assets/surfaces/bricks/bricks_color.png.import +++ /dev/null @@ -1,9 +0,0 @@ -{ - "version": 4, - "settings": { - "image": "Color" - }, - "outputs": [ - {"kind": "TEX", "name": "bricks_color", "index": 0, "id": "4c24cf1fa5b4eeca8ac84ea3d8813cfc"} - ] -} diff --git a/demo/assets/surfaces/bricks/bricks_normal.png.import b/demo/assets/surfaces/bricks/bricks_normal.png.import deleted file mode 100644 index 794606cbd..000000000 --- a/demo/assets/surfaces/bricks/bricks_normal.png.import +++ /dev/null @@ -1,9 +0,0 @@ -{ - "version": 4, - "settings": { - "image": "Normal" - }, - "outputs": [ - {"kind": "TEX", "name": "bricks_normal", "index": 0, "id": "0873ef8dc2c7deca1ed9eb84f5a8d6b9"} - ] -} diff --git a/demo/assets/surfaces/bricks/bricks_orm.png.import b/demo/assets/surfaces/bricks/bricks_orm.png.import deleted file mode 100644 index d6640a312..000000000 --- a/demo/assets/surfaces/bricks/bricks_orm.png.import +++ /dev/null @@ -1,7 +0,0 @@ -{ - "version": 4, - "settings": {}, - "outputs": [ - {"kind": "TEX", "name": "bricks_orm", "index": 0, "id": "d526fc286cf3d0bb3c956200037149d5"} - ] -} diff --git a/demo/assets/surfaces/cobblestone/cobblestone_color.png.import b/demo/assets/surfaces/cobblestone/cobblestone_color.png.import deleted file mode 100644 index b4ba0c460..000000000 --- a/demo/assets/surfaces/cobblestone/cobblestone_color.png.import +++ /dev/null @@ -1,9 +0,0 @@ -{ - "version": 4, - "settings": { - "image": "Color" - }, - "outputs": [ - {"kind": "TEX", "name": "cobblestone_color", "index": 0, "id": "9a69320baec0a94ab9eb8d2010a8a8a4"} - ] -} diff --git a/demo/assets/surfaces/cobblestone/cobblestone_normal.png.import b/demo/assets/surfaces/cobblestone/cobblestone_normal.png.import deleted file mode 100644 index 45262b305..000000000 --- a/demo/assets/surfaces/cobblestone/cobblestone_normal.png.import +++ /dev/null @@ -1,9 +0,0 @@ -{ - "version": 4, - "settings": { - "image": "Normal" - }, - "outputs": [ - {"kind": "TEX", "name": "cobblestone_normal", "index": 0, "id": "1a08080be2a08b5fe300ec425923c195"} - ] -} diff --git a/demo/assets/surfaces/cobblestone/cobblestone_orm.png.import b/demo/assets/surfaces/cobblestone/cobblestone_orm.png.import deleted file mode 100644 index 991d12207..000000000 --- a/demo/assets/surfaces/cobblestone/cobblestone_orm.png.import +++ /dev/null @@ -1,7 +0,0 @@ -{ - "version": 4, - "settings": {}, - "outputs": [ - {"kind": "TEX", "name": "cobblestone_orm", "index": 0, "id": "94cef7c053d5afed918f83a3d6242853"} - ] -} diff --git a/demo/assets/surfaces/glass/stained_glass_color.png.import b/demo/assets/surfaces/glass/stained_glass_color.png.import deleted file mode 100644 index 6e1b89b79..000000000 --- a/demo/assets/surfaces/glass/stained_glass_color.png.import +++ /dev/null @@ -1,9 +0,0 @@ -{ - "version": 4, - "settings": { - "image": "Color" - }, - "outputs": [ - {"kind": "TEX", "name": "stained_glass_color", "index": 0, "id": "601bad68a6b7e349bc2655ac13c9196f"} - ] -} diff --git a/demo/assets/surfaces/marble/marble_color.png.import b/demo/assets/surfaces/marble/marble_color.png.import deleted file mode 100644 index 5d7f668f2..000000000 --- a/demo/assets/surfaces/marble/marble_color.png.import +++ /dev/null @@ -1,9 +0,0 @@ -{ - "version": 4, - "settings": { - "image": "Color" - }, - "outputs": [ - {"kind": "TEX", "name": "marble_color", "index": 0, "id": "f3ace87b27d210d20ba711efdf73ad3a"} - ] -} diff --git a/demo/assets/surfaces/marble/marble_normal.png.import b/demo/assets/surfaces/marble/marble_normal.png.import deleted file mode 100644 index e5dc4f758..000000000 --- a/demo/assets/surfaces/marble/marble_normal.png.import +++ /dev/null @@ -1,9 +0,0 @@ -{ - "version": 4, - "settings": { - "image": "Normal" - }, - "outputs": [ - {"kind": "TEX", "name": "marble_normal", "index": 0, "id": "6efbfe1939e3a2a35c0ccc8621cc6abb"} - ] -} diff --git a/demo/assets/surfaces/marble/marble_orm.png.import b/demo/assets/surfaces/marble/marble_orm.png.import deleted file mode 100644 index 2f55d2ecd..000000000 --- a/demo/assets/surfaces/marble/marble_orm.png.import +++ /dev/null @@ -1,7 +0,0 @@ -{ - "version": 4, - "settings": {}, - "outputs": [ - {"kind": "TEX", "name": "marble_orm", "index": 0, "id": "972fc9634ba6242d0b5f33e03e3a0741"} - ] -} diff --git a/demo/assets/surfaces/metal/brushed_normal.png.import b/demo/assets/surfaces/metal/brushed_normal.png.import deleted file mode 100644 index 33a41e8c9..000000000 --- a/demo/assets/surfaces/metal/brushed_normal.png.import +++ /dev/null @@ -1,9 +0,0 @@ -{ - "version": 4, - "settings": { - "image": "Normal" - }, - "outputs": [ - {"kind": "TEX", "name": "brushed_normal", "index": 0, "id": "b9466f1cd86f5f900d74e12980fdc702"} - ] -} diff --git a/demo/assets/surfaces/metal/brushed_orm.png.import b/demo/assets/surfaces/metal/brushed_orm.png.import deleted file mode 100644 index cdb45412f..000000000 --- a/demo/assets/surfaces/metal/brushed_orm.png.import +++ /dev/null @@ -1,7 +0,0 @@ -{ - "version": 4, - "settings": {}, - "outputs": [ - {"kind": "TEX", "name": "brushed_orm", "index": 0, "id": "ec4b617cf48145349c2c76f465e19ae6"} - ] -} diff --git a/demo/assets/surfaces/metal/grate_color.png.import b/demo/assets/surfaces/metal/grate_color.png.import deleted file mode 100644 index 58a3e7c7c..000000000 --- a/demo/assets/surfaces/metal/grate_color.png.import +++ /dev/null @@ -1,10 +0,0 @@ -{ - "version": 4, - "settings": { - "image": "Color", - "alpha_coverage": 0.5 - }, - "outputs": [ - {"kind": "TEX", "name": "grate_color", "index": 0, "id": "90b31208c9dfc6a6d00ab5517f084571"} - ] -} diff --git a/demo/assets/surfaces/metal/grate_normal.png.import b/demo/assets/surfaces/metal/grate_normal.png.import deleted file mode 100644 index 547724392..000000000 --- a/demo/assets/surfaces/metal/grate_normal.png.import +++ /dev/null @@ -1,9 +0,0 @@ -{ - "version": 4, - "settings": { - "image": "Normal" - }, - "outputs": [ - {"kind": "TEX", "name": "grate_normal", "index": 0, "id": "c9d9c9f0f1dea3ec0ceec0f9671187eb"} - ] -} diff --git a/demo/assets/surfaces/metal/grate_orm.png.import b/demo/assets/surfaces/metal/grate_orm.png.import deleted file mode 100644 index a46e120f5..000000000 --- a/demo/assets/surfaces/metal/grate_orm.png.import +++ /dev/null @@ -1,7 +0,0 @@ -{ - "version": 4, - "settings": {}, - "outputs": [ - {"kind": "TEX", "name": "grate_orm", "index": 0, "id": "71e4b6effc8f9e4ac6c7f1dc0f02612e"} - ] -} diff --git a/demo/assets/surfaces/neon/neon_emission.png.import b/demo/assets/surfaces/neon/neon_emission.png.import deleted file mode 100644 index db5eae895..000000000 --- a/demo/assets/surfaces/neon/neon_emission.png.import +++ /dev/null @@ -1,9 +0,0 @@ -{ - "version": 4, - "settings": { - "image": "Color" - }, - "outputs": [ - {"kind": "TEX", "name": "neon_emission", "index": 0, "id": "6d9362793631542f0b778005f9fd8dd0"} - ] -} diff --git a/demo/assets/surfaces/tiles/tiles_color.png.import b/demo/assets/surfaces/tiles/tiles_color.png.import deleted file mode 100644 index 1c217cf03..000000000 --- a/demo/assets/surfaces/tiles/tiles_color.png.import +++ /dev/null @@ -1,9 +0,0 @@ -{ - "version": 4, - "settings": { - "image": "Color" - }, - "outputs": [ - {"kind": "TEX", "name": "tiles_color", "index": 0, "id": "cd3d66765e834e0c508906257f888eba"} - ] -} diff --git a/demo/assets/surfaces/tiles/tiles_normal.png.import b/demo/assets/surfaces/tiles/tiles_normal.png.import deleted file mode 100644 index 08ea58e93..000000000 --- a/demo/assets/surfaces/tiles/tiles_normal.png.import +++ /dev/null @@ -1,9 +0,0 @@ -{ - "version": 4, - "settings": { - "image": "Normal" - }, - "outputs": [ - {"kind": "TEX", "name": "tiles_normal", "index": 0, "id": "bd7cde48d01cd173f904fe6b00fb72dc"} - ] -} diff --git a/demo/assets/surfaces/tiles/tiles_orm.png.import b/demo/assets/surfaces/tiles/tiles_orm.png.import deleted file mode 100644 index a17220ba3..000000000 --- a/demo/assets/surfaces/tiles/tiles_orm.png.import +++ /dev/null @@ -1,7 +0,0 @@ -{ - "version": 4, - "settings": {}, - "outputs": [ - {"kind": "TEX", "name": "tiles_orm", "index": 0, "id": "84dbfd2533a249a9716c78d548c6c654"} - ] -} diff --git a/demo/assets/surfaces/wood/planks_color.png.import b/demo/assets/surfaces/wood/planks_color.png.import deleted file mode 100644 index 8844b1b87..000000000 --- a/demo/assets/surfaces/wood/planks_color.png.import +++ /dev/null @@ -1,9 +0,0 @@ -{ - "version": 4, - "settings": { - "image": "Color" - }, - "outputs": [ - {"kind": "TEX", "name": "planks_color", "index": 0, "id": "79a53ef6d585bc9fd1a912a9e592f913"} - ] -} diff --git a/demo/assets/surfaces/wood/planks_normal.png.import b/demo/assets/surfaces/wood/planks_normal.png.import deleted file mode 100644 index d4b65ab27..000000000 --- a/demo/assets/surfaces/wood/planks_normal.png.import +++ /dev/null @@ -1,9 +0,0 @@ -{ - "version": 4, - "settings": { - "image": "Normal" - }, - "outputs": [ - {"kind": "TEX", "name": "planks_normal", "index": 0, "id": "0558bb6aa0ec012cac33f59d343e99ed"} - ] -} diff --git a/demo/assets/surfaces/wood/planks_orm.png.import b/demo/assets/surfaces/wood/planks_orm.png.import deleted file mode 100644 index 454ad998d..000000000 --- a/demo/assets/surfaces/wood/planks_orm.png.import +++ /dev/null @@ -1,7 +0,0 @@ -{ - "version": 4, - "settings": {}, - "outputs": [ - {"kind": "TEX", "name": "planks_orm", "index": 0, "id": "6646d15415504809a3da8752ce1d6ca6"} - ] -} diff --git a/demo/assets/vfx_gallery/vfx_hud.ui b/demo/assets/vfx_gallery/vfx_hud.ui index a6cebcf28..e0b167ff4 100644 Binary files a/demo/assets/vfx_gallery/vfx_hud.ui and b/demo/assets/vfx_gallery/vfx_hud.ui differ diff --git a/demo/game/animation_gallery_matching_test.odin b/demo/game/animation_gallery_matching_test.odin index 30ef35d53..736c9f658 100644 --- a/demo/game/animation_gallery_matching_test.odin +++ b/demo/game/animation_gallery_matching_test.odin @@ -1,7 +1,5 @@ package game -import "core:fmt" -import "core:os" import "core:testing" import "mirai:asset" import "mirai:core" @@ -22,8 +20,7 @@ gallery_mirai_matching_plays_idle_walk_and_jog_by_the_speed_it_is_asked :: proc( progress:core.Table(anim.Animation_Progress);defer core.table_destroy(&progress) panel:Gallery_UI core.entities_init(&f.entities) - root,found:=os.lookup_env("MIRAI_ROOT",context.temp_allocator) - if !testing.expect(t,found && host.assets_init(&f.assets,{root=fmt.tprintf("%s/demo/assets",root),imported=fmt.tprintf("%s/demo/.mirai/imported",root),capacity=512})) {return} + if !demo_assets_init(t,&f.assets) {return} f.app={assets=&f.assets,allocator=context.allocator} ids,ok:=mirai_assets(t,&f.assets,"characters/mirai/mirai_gallery.character") if !ok {return} diff --git a/demo/game/animation_motor_test.odin b/demo/game/animation_motor_test.odin index edfc3a639..92d8e75f0 100644 --- a/demo/game/animation_motor_test.odin +++ b/demo/game/animation_motor_test.odin @@ -30,7 +30,7 @@ animation_motor_preserves_world_root_direction_facing_and_physical_speed_limits_ } // The body of Mirai: the first mesh of the import of characters/mirai/mirai.glb, by the id its -// sidecar keeps. The sync of a build writes it to the imported folder. +// package keeps. The demo lists it from its packages in the machine store. @(private) MIRAI_BODY :: "f0c947702bb8141dc1d27444a341d52c" diff --git a/demo/game/mirai_matching_test.odin b/demo/game/mirai_matching_test.odin index b283cf6fa..bad40247c 100644 --- a/demo/game/mirai_matching_test.odin +++ b/demo/game/mirai_matching_test.odin @@ -8,6 +8,7 @@ import "core:testing" import "mirai:core" import render "mirai:engine/render" import "mirai:host" +import "mirai:project" import "mirai:std/base" import mm "mirai:std/math" import anim "plugins:animation/runtime" @@ -21,8 +22,7 @@ mirai_matching_selects_a_walking_clip :: proc(t:^testing.T) { graphs:anim.Character_Assets;defer anim.character_assets_free(&graphs) progress:core.Table(anim.Animation_Progress);defer core.table_destroy(&progress) core.entities_init(&f.entities) - root,found:=os.lookup_env("MIRAI_ROOT",context.temp_allocator) - if !testing.expect(t,found && host.assets_init(&f.assets,{root=fmt.tprintf("%s/demo/assets",root),imported=fmt.tprintf("%s/demo/.mirai/imported",root),capacity=512})) {return} + if !demo_assets_init(t,&f.assets) {return} f.app={assets=&f.assets,allocator=context.allocator} ids,ok:=mirai_assets(t,&f.assets,"characters/mirai/mirai_matching.character") if !ok {return} @@ -93,8 +93,7 @@ mirai_starting_to_its_side_turns_no_bone_faster_than_its_walk :: proc(t:^testing constraints:anim.Animation_Constraints;defer anim.constraints_free(&constraints) grounding:anim.Grounding_Runtime;defer anim.grounding_free(&grounding) core.entities_init(&f.entities) - root,found:=os.lookup_env("MIRAI_ROOT",context.temp_allocator) - if !testing.expect(t,found && host.assets_init(&f.assets,{root=fmt.tprintf("%s/demo/assets",root),imported=fmt.tprintf("%s/demo/.mirai/imported",root),capacity=512})) {return} + if !demo_assets_init(t,&f.assets) {return} f.app={assets=&f.assets,allocator=context.allocator} ids,ok:=mirai_assets(t,&f.assets,"characters/mirai/mirai_matching.character") if !ok {return} @@ -146,3 +145,15 @@ mirai_starting_to_its_side_turns_no_bone_faster_than_its_walk :: proc(t:^testing testing.expectf(t,start<=2*walk,"starting to its side no bone turns faster than twice its walk: %.2f against %.2f rad/s",start,walk) testing.expect_value(t,problems.count,0) } + +// The assets of the demo as its player sets them up: its own, those of its +// plugins and the outputs of the packages of both in the machine store. +demo_assets_init :: proc(t:^testing.T,a:^host.Assets) -> bool { + root,found:=os.lookup_env("MIRAI_ROOT",context.temp_allocator) + if !testing.expect(t,found,"MIRAI_ROOT names the engine") {return false} + demo:=fmt.tprintf("%s/demo",root) + p,error:=project.open(demo,root,context.temp_allocator) + if !testing.expect_value(t,error.code,project.Error_Code.None) {return false} + assets,listed:=host.project_assets(&p,"",context.temp_allocator) + return testing.expect(t,listed && host.assets_init(a,{root=fmt.tprintf("%s/assets",demo),listed=assets.listed,mounts=assets.mounts,package_problems=assets.problems,capacity=512})) +} diff --git a/demo/packages.mirai b/demo/packages.mirai new file mode 100644 index 000000000..882affd49 --- /dev/null +++ b/demo/packages.mirai @@ -0,0 +1,45 @@ +{ + "version": 2, + "packages": [ + {"id": "780bee6597fbb860e803d513bda2dee7", "manifest": "4a963eaca3eae9448050350443c6f8fd2bd347823bd95bcfbb5dcd1c5ee5370f", "folder": "characters/jelly/jelly", "assets": [], "written": [{"id": "0d555b0da56949fb3cdbdf41410432f4", "hash": "65dd63a360e19c1d"}, {"id": "107c9b8f4bf9dfe8936235eae8e042a6", "hash": "5b5880022fafcf27"}, {"id": "dae452ce18d59e780ff745f7d1b43697", "hash": "362f067fb5ff411f"}]}, + {"id": "0512ac6bb76639efa1aa35746972fedb", "manifest": "3917d5ddca017afdb78f7390198c987789bbf86852b07a846a96503fd3280e90", "folder": "characters/mirai", "assets": [], "written": [{"id": "0208471f893327bdcf8bbaa108a26121", "hash": "2d314f5bdf5a523e"}, {"id": "94f640750cf5246f1c0728cc4cd2a07c", "hash": "788f158048ce257b"}, {"id": "f3016e0fca41fca429b049307bde6f9f", "hash": "fe095a7f121c24c1"}]}, + {"id": "2d26a9a0d52c3edd23a71d2cc8d6e463", "manifest": "60cb4e9c196cdad90fbf5defbdc67a6fa641338b80f3e1166064c8573acf1a00", "folder": "characters/robot/glb", "assets": [], "written": [{"id": "83d297850b4eccf18f4225970eff51fb", "hash": "bf93bd458cff13fe"}, {"id": "e431738105898dd368735a6ebb1d3c19", "hash": "f37982ab320ae1f2"}]}, + {"id": "84060b2db086d874fb6a99a638a79a10", "manifest": "1c848c98e0f08a7645bc8e598dbbe72007ddbe945eaa665083f76780e08b6137", "folder": "characters/robot/gltf", "assets": [], "written": [{"id": "b0f0241df6e014392af304b56cf59eab", "hash": "8cda5d5db03a2b44"}, {"id": "cf5b1fe881d020461927fe9b85c42136", "hash": "f47e8b048a96688f"}]}, + {"id": "44342052da337fa065c60cf296a4466a", "manifest": "8537ae1e9dabe37f7b501e251fb39fa83d136ab825f85fd6faeeb7068f34a49a", "folder": "church", "assets": [], "written": []}, + {"id": "3df37c7e292cc0578d432dfdb3489314", "manifest": "12c923cc09ff66347e681708f18600ce0cfc64dc5a0876d19ac3fb6c31810b46", "folder": "forest/bush", "assets": [], "written": [{"id": "48eed1e5335b875bef27025acf93953a", "hash": "c8f83ce6424e34d4"}]}, + {"id": "caae575f45863b717aafbe7fe73862c0", "manifest": "7616716f5e8ef9957e7e8439c567854e015ac970d32ad5c6d61d5de288e21530", "folder": "gallery", "assets": [], "written": []}, + {"id": "6498e1a94e6c4621f925d22f8384243a", "manifest": "f943fbda4e5f3e2d21db427c8edb288b619695e94bcb7044cbca54f9083bc1a1", "folder": "gallery/plant", "assets": [], "written": [{"id": "fc120101f1ddb302e135784184f05931", "hash": "57cd300b21844cd6"}]}, + {"id": "a3799fb21f329ae819440d1726f2aa20", "manifest": "3e577e2889305d7efd56fb9e47d11e7420b76654c6096d27663d865b36b84d1e", "folder": "gallery/vase", "assets": [], "written": [{"id": "f430e11663e1ae229bf1511ac0c7f82d", "hash": "3b40a211e5bc6d93"}]}, + {"id": "ca27d206e666815335b68ee9c8900ab7", "manifest": "b3d7a22ae8f1d6a895f5398c22f91d521aeac6e34fc583f30d07f506623b7ef4", "folder": "gallery", "assets": [], "written": []}, + {"id": "7572bd9ffa0dd2354005eff39b25f1bb", "manifest": "d9067a4a5673870ded5622963e9937580372fd9c4be7c60cbf1e1d7884afa80c", "folder": "showcase", "assets": [], "written": []}, + {"id": "181b8168d78256877581cb3f1fda937e", "manifest": "1a337d3e6ab531f0559a93db34b3b9563d4be6ff1354a59789875c6bdd5af261", "folder": "sky", "assets": [], "written": []}, + {"id": "c1a66f64c05efe53cca206765ab98117", "manifest": "a39653aff76a573ab91ac1a7c4717be9770a053b86d153a1ca083208221712d0", "folder": "sky", "assets": [], "written": []}, + {"id": "4840a3f5a151cbe8c1b403b30fe8240d", "manifest": "623313b02fd3e64fec926caffcb1470c3fae02f48a29a94176b3aa42307766fd", "folder": "sky", "assets": [], "written": []}, + {"id": "e022a803a1c6fb9fe4b86ab65a48bc24", "manifest": "f83c31a2d6c7929d4d5fb5b921c0c3e52c7ee06807c0967d52f150a3720623bb", "folder": "sky", "assets": [], "written": []}, + {"id": "0d0874d674d00d43ab66a390641369df", "manifest": "8f31ccfb8cf203764bfa66d37fe5f6e3dbb36aa62806ca8099cf570b7902bcaf", "folder": "surfaces/bricks", "assets": [], "written": []}, + {"id": "0ab7eefb80a0bb80c8cd0065d5c53f11", "manifest": "6eccf80dcb05311a5d6951e666ac5085b87a6188fb1970872abd9059030018c4", "folder": "surfaces/bricks", "assets": [], "written": []}, + {"id": "26356114f7faffcfafff109308dd9a32", "manifest": "a167b042e04fa82c3cc0dc5a812b27ca4096f72ca6d54773f9dec952aa33ad9d", "folder": "surfaces/bricks", "assets": [], "written": []}, + {"id": "c0d7666d947e65fc7dd68a31013c0939", "manifest": "3f538b9e6f732075b4c363c29794378657126105df9e3ea9731e73dd4735d867", "folder": "surfaces/cobblestone", "assets": [], "written": []}, + {"id": "a278db7076e649a48933fc75b0dc1a4c", "manifest": "b666d4035290effaf589397c6f1db1280c115131aa95e18746990cc95c960835", "folder": "surfaces/cobblestone", "assets": [], "written": []}, + {"id": "ae40765be8d60ddad02e47bb21203f8f", "manifest": "bb75ff24c6019485ca30a18ca4ceec0a85ce35e72c76365bc5b231c01b6e9974", "folder": "surfaces/cobblestone", "assets": [], "written": []}, + {"id": "6cc37d40c23ded0fb9cc963a4dec2fbb", "manifest": "07fef0754a8ed415be607c429fea81eae4bc68e5d505084b838c0f903904ab4b", "folder": "surfaces/glass", "assets": [], "written": []}, + {"id": "09648c700552c36b5b4062fd1822de01", "manifest": "3e1aada9237d0076bc8d5c548c3ee8b782a6a2de721a6d0afd492934f6f4a5fd", "folder": "surfaces/marble", "assets": [], "written": []}, + {"id": "2d6b0454a36971219d185d940f836ac7", "manifest": "2b2027b23a16bbea30c4cb3ecbc2cdc8b2b16074700cd89a4780390bacaec932", "folder": "surfaces/marble", "assets": [], "written": []}, + {"id": "a68982d58a94bfa0fcb588bf93dfafec", "manifest": "7b1ae691cd8969afe5a313747f792e6d3c018b6cb60a94a21e997f75bc7ac79f", "folder": "surfaces/marble", "assets": [], "written": []}, + {"id": "641ffc8940c220c3b2460f7a9eaa7862", "manifest": "730eaab4050470f4db747e8eddd2a4879d8bc7e8a2a6a99b256b9df4136d28fa", "folder": "surfaces/metal", "assets": [], "written": []}, + {"id": "a03a7d06fb4c433e89ba1ecd314d91ed", "manifest": "4153cc5e753ba39602ffded1fb1b353c4ae7e07387ee9d5c1a800e58f9d0299a", "folder": "surfaces/metal", "assets": [], "written": []}, + {"id": "e0495081646e8954b731fc082bcd8884", "manifest": "81200390a7453dd2fbd4ee589dbd9fbc2a86da349299aa6083b7664947214e35", "folder": "surfaces/metal", "assets": [], "written": []}, + {"id": "65805c17b2454730221e3d63066ce7f5", "manifest": "1f304dc9e03252dda912d5d5381f6ffd02869a7c974f0328e0bb7303499be5bd", "folder": "surfaces/metal", "assets": [], "written": []}, + {"id": "81d18317bfbbf0851f30ebcb2de41b67", "manifest": "6fef9b0ccfddcf796583a20fa2837bf3a09235b4c262f28778e01d3370dc8ffd", "folder": "surfaces/metal", "assets": [], "written": []}, + {"id": "856074124b2e8924607a1e79488b6abd", "manifest": "1b3c97c929e2eeca39cbe75ca5ae2a8c04e3ad5abdc22b1d90f6b8731d61a464", "folder": "surfaces/neon", "assets": [], "written": []}, + {"id": "a270e82878cb164498ba98398060ccb8", "manifest": "990aa1e56c54010dcdbfb641b030bfe274a72281753e8f0a567bcadb53ec4028", "folder": "surfaces/tiles", "assets": [], "written": []}, + {"id": "0b5202b0a19887317347ddcbfdcad5f6", "manifest": "1d9f92898ed6e4775e013790a7ccb27755b33b881047c8eed6be2a41b6ec63c0", "folder": "surfaces/tiles", "assets": [], "written": []}, + {"id": "de7548cbbad3494b21280744cd16ed68", "manifest": "e6a7c04950a4f843283e188eeca9538d0f98035ff07ddf8e2216edc44f76ab90", "folder": "surfaces/tiles", "assets": [], "written": []}, + {"id": "e96915c067e6fe50d517b59bf3b17d61", "manifest": "79982691c9fd719b5f3e34331a1cbeb88676be5f5fd8c630453b5860d0fd59f9", "folder": "surfaces/wood", "assets": [], "written": []}, + {"id": "f4ccf188ab1bf19a62c53faaef829fba", "manifest": "a129e3f38b1008609560167f132b43076b087fce75805e78f41f0d2cd09623c2", "folder": "surfaces/wood", "assets": [], "written": []}, + {"id": "0964d5141215cea8462119f07a904e49", "manifest": "c5c4fb4a315b60117f7f6564c7f39b7e4efa7d2db6caa5afcf0cb48f30715cc9", "folder": "surfaces/wood", "assets": [], "written": []}, + {"id": "81dd55e2e057dae790850f4a006dce86", "manifest": "1c7577bc9dc9e0fb9098a9f76c5b544b0bb04c872b8bf8522dbec916b883cbfd", "folder": "foliage_extreme/tree_broad", "assets": [], "written": [{"id": "73668be537ffad086162e2c98e1c79f6", "hash": "f7d64916b7bf866a"}]}, + {"id": "9c80ffb4545f5d74f0a1ccf6acedba6b", "manifest": "177382bb6964ce9c6b3d15e12f6f50c25d7d5c68bb17d46d7a0b0e1b8bfe0445", "folder": "foliage_extreme/tree_bush", "assets": [], "written": [{"id": "317c6ad95fead4485707ae3231936f9b", "hash": "4570dc11b07a13a9"}]}, + {"id": "2dbac73486cbcaad3569e05ecd01bb4f", "manifest": "17a6ed50f1576aa65e2279d141ce04423e7ade5c7804d215099db38d9a35ddb0", "folder": "foliage_extreme/tree_tall", "assets": [], "written": [{"id": "10f19381013cd286f491234605c5ca3a", "hash": "0f43206836f2f3e5"}]} + ] +} diff --git a/demo/packages/0512ac6bb76639efa1aa35746972fedb.mirai b/demo/packages/0512ac6bb76639efa1aa35746972fedb.mirai new file mode 100644 index 000000000..3f6ca2aa4 --- /dev/null +++ b/demo/packages/0512ac6bb76639efa1aa35746972fedb.mirai @@ -0,0 +1,21 @@ +{ + "version": 1, + "id": "0512ac6bb76639efa1aa35746972fedb", + "name": "mirai", + "origin": {"path": "assets/characters/mirai/mirai.glb", "folder": false, "importer": "glTF"}, + "identity": "339c12d886baefcd14f96e1c9cb6501e6b85183d0b5edbba6b290d3853c2937c", + "inputs": [], + "assets": [ + {"kind": "SCEN", "name": "mirai", "index": 0, "id": "94f640750cf5246f1c0728cc4cd2a07c"}, + {"kind": "TEX", "name": "texture_0", "index": 0, "id": "a3d324f2a5454495463d13c1ebad2c84", "part": true, "owner": "94f640750cf5246f1c0728cc4cd2a07c"}, + {"kind": "TEX", "name": "texture_0_metallic_roughness", "index": 1, "id": "08abe809e118473bb530ac037696309e", "part": true, "owner": "94f640750cf5246f1c0728cc4cd2a07c"}, + {"kind": "MAT", "name": "BakedMaterial", "index": 0, "id": "f1d4dfad902a16a7afc56bb3202b1d94", "owner": "94f640750cf5246f1c0728cc4cd2a07c", "dependencies": ["08abe809e118473bb530ac037696309e", "a3d324f2a5454495463d13c1ebad2c84"]}, + {"kind": "MAT", "name": "Mirai holograma violeta", "index": 1, "id": "dbb521d6ca206ef049b42c4fad14b7d5"}, + {"kind": "MAT", "name": "Mirai borde luminoso", "index": 2, "id": "bbfe2f2406e2d1a5aaf297d5d2c8a990"}, + {"kind": "SKEL", "name": "Armature", "index": 0, "id": "0208471f893327bdcf8bbaa108a26121", "owner": "94f640750cf5246f1c0728cc4cd2a07c"}, + {"kind": "MESH", "name": "output_unwrapped", "index": 0, "id": "f0c947702bb8141dc1d27444a341d52c", "part": true, "owner": "94f640750cf5246f1c0728cc4cd2a07c", "dependencies": ["0208471f893327bdcf8bbaa108a26121", "f1d4dfad902a16a7afc56bb3202b1d94"]}, + {"kind": "MESH", "name": "Cinta orbital Left", "index": 1, "id": "8d6c8aa43c816332bca9f38c1dd0d17f", "part": true, "owner": "94f640750cf5246f1c0728cc4cd2a07c", "dependencies": ["0208471f893327bdcf8bbaa108a26121", "bbfe2f2406e2d1a5aaf297d5d2c8a990", "dbb521d6ca206ef049b42c4fad14b7d5"]}, + {"kind": "MESH", "name": "Cinta orbital Right", "index": 2, "id": "78744c9faee4ebd82e0f9ca443b72656", "part": true, "owner": "94f640750cf5246f1c0728cc4cd2a07c", "dependencies": ["0208471f893327bdcf8bbaa108a26121", "bbfe2f2406e2d1a5aaf297d5d2c8a990", "dbb521d6ca206ef049b42c4fad14b7d5"]}, + {"kind": "ANIM", "name": "mirai", "index": 0, "id": "f3016e0fca41fca429b049307bde6f9f", "dependencies": ["0208471f893327bdcf8bbaa108a26121"]} + ] +} diff --git a/demo/packages/09648c700552c36b5b4062fd1822de01.mirai b/demo/packages/09648c700552c36b5b4062fd1822de01.mirai new file mode 100644 index 000000000..908889266 --- /dev/null +++ b/demo/packages/09648c700552c36b5b4062fd1822de01.mirai @@ -0,0 +1,14 @@ +{ + "version": 1, + "id": "09648c700552c36b5b4062fd1822de01", + "name": "marble_color", + "origin": {"path": "assets/surfaces/marble/marble_color.png", "folder": false, "importer": "image"}, + "identity": "2bbad925aaaa5a3c690d3e9061e0cee2f6d89ed230c0a78735a74b97bd707b66", + "settings": { + "image": "Color" + }, + "inputs": [], + "assets": [ + {"kind": "TEX", "name": "marble_color", "index": 0, "id": "f3ace87b27d210d20ba711efdf73ad3a"} + ] +} diff --git a/demo/packages/0964d5141215cea8462119f07a904e49.mirai b/demo/packages/0964d5141215cea8462119f07a904e49.mirai new file mode 100644 index 000000000..467ab477d --- /dev/null +++ b/demo/packages/0964d5141215cea8462119f07a904e49.mirai @@ -0,0 +1,11 @@ +{ + "version": 1, + "id": "0964d5141215cea8462119f07a904e49", + "name": "planks_orm", + "origin": {"path": "assets/surfaces/wood/planks_orm.png", "folder": false, "importer": "image"}, + "identity": "a2dcb1c391aa037e1f7886bec3d3e17360d588365e162544f1aac4c6cb4532c1", + "inputs": [], + "assets": [ + {"kind": "TEX", "name": "planks_orm", "index": 0, "id": "6646d15415504809a3da8752ce1d6ca6"} + ] +} diff --git a/demo/packages/0ab7eefb80a0bb80c8cd0065d5c53f11.mirai b/demo/packages/0ab7eefb80a0bb80c8cd0065d5c53f11.mirai new file mode 100644 index 000000000..d47109d81 --- /dev/null +++ b/demo/packages/0ab7eefb80a0bb80c8cd0065d5c53f11.mirai @@ -0,0 +1,14 @@ +{ + "version": 1, + "id": "0ab7eefb80a0bb80c8cd0065d5c53f11", + "name": "bricks_normal", + "origin": {"path": "assets/surfaces/bricks/bricks_normal.png", "folder": false, "importer": "image"}, + "identity": "368a6ec83712603a49d698bec592fc764a4c289b9ffacc76273700cb4eaca77b", + "settings": { + "image": "Normal" + }, + "inputs": [], + "assets": [ + {"kind": "TEX", "name": "bricks_normal", "index": 0, "id": "0873ef8dc2c7deca1ed9eb84f5a8d6b9"} + ] +} diff --git a/demo/packages/0b5202b0a19887317347ddcbfdcad5f6.mirai b/demo/packages/0b5202b0a19887317347ddcbfdcad5f6.mirai new file mode 100644 index 000000000..1bda6238d --- /dev/null +++ b/demo/packages/0b5202b0a19887317347ddcbfdcad5f6.mirai @@ -0,0 +1,14 @@ +{ + "version": 1, + "id": "0b5202b0a19887317347ddcbfdcad5f6", + "name": "tiles_normal", + "origin": {"path": "assets/surfaces/tiles/tiles_normal.png", "folder": false, "importer": "image"}, + "identity": "fab044b54421b971441925ac3cd28572d405557ff481d49e448e1604c2842e2c", + "settings": { + "image": "Normal" + }, + "inputs": [], + "assets": [ + {"kind": "TEX", "name": "tiles_normal", "index": 0, "id": "bd7cde48d01cd173f904fe6b00fb72dc"} + ] +} diff --git a/demo/packages/0d0874d674d00d43ab66a390641369df.mirai b/demo/packages/0d0874d674d00d43ab66a390641369df.mirai new file mode 100644 index 000000000..83d3a5406 --- /dev/null +++ b/demo/packages/0d0874d674d00d43ab66a390641369df.mirai @@ -0,0 +1,14 @@ +{ + "version": 1, + "id": "0d0874d674d00d43ab66a390641369df", + "name": "bricks_color", + "origin": {"path": "assets/surfaces/bricks/bricks_color.png", "folder": false, "importer": "image"}, + "identity": "5161142e9629e908d0b4834890974b3c71806bdc8c2ef3f4dac651326c87609b", + "settings": { + "image": "Color" + }, + "inputs": [], + "assets": [ + {"kind": "TEX", "name": "bricks_color", "index": 0, "id": "4c24cf1fa5b4eeca8ac84ea3d8813cfc"} + ] +} diff --git a/demo/packages/181b8168d78256877581cb3f1fda937e.mirai b/demo/packages/181b8168d78256877581cb3f1fda937e.mirai new file mode 100644 index 000000000..d38b1f80a --- /dev/null +++ b/demo/packages/181b8168d78256877581cb3f1fda937e.mirai @@ -0,0 +1,11 @@ +{ + "version": 1, + "id": "181b8168d78256877581cb3f1fda937e", + "name": "sky_day", + "origin": {"path": "assets/sky/sky_day.hdr", "folder": false, "importer": "image"}, + "identity": "69fe28de049e33d362a88e29fb8cd207eb86ece14f820cd2495f4ce6ab5e1a63", + "inputs": [], + "assets": [ + {"kind": "TEX", "name": "sky_day", "index": 0, "id": "39808e0def899c47a24a89a3fab89e64"} + ] +} diff --git a/demo/packages/26356114f7faffcfafff109308dd9a32.mirai b/demo/packages/26356114f7faffcfafff109308dd9a32.mirai new file mode 100644 index 000000000..d4873aa32 --- /dev/null +++ b/demo/packages/26356114f7faffcfafff109308dd9a32.mirai @@ -0,0 +1,11 @@ +{ + "version": 1, + "id": "26356114f7faffcfafff109308dd9a32", + "name": "bricks_orm", + "origin": {"path": "assets/surfaces/bricks/bricks_orm.png", "folder": false, "importer": "image"}, + "identity": "b5b53e22299c99493478359f1e2936970ca5191915b380c995ea296bd95458fe", + "inputs": [], + "assets": [ + {"kind": "TEX", "name": "bricks_orm", "index": 0, "id": "d526fc286cf3d0bb3c956200037149d5"} + ] +} diff --git a/demo/packages/2d26a9a0d52c3edd23a71d2cc8d6e463.mirai b/demo/packages/2d26a9a0d52c3edd23a71d2cc8d6e463.mirai new file mode 100644 index 000000000..5b9c86c74 --- /dev/null +++ b/demo/packages/2d26a9a0d52c3edd23a71d2cc8d6e463.mirai @@ -0,0 +1,17 @@ +{ + "version": 1, + "id": "2d26a9a0d52c3edd23a71d2cc8d6e463", + "name": "robot", + "origin": {"path": "assets/characters/robot/robot.glb", "folder": false, "importer": "glTF"}, + "identity": "7afe8a1a9c863a77b6a70bb570e9e678e59d01af0c432e3c93d4d922f5425235", + "inputs": [], + "assets": [ + {"kind": "SCEN", "name": "robot", "index": 0, "id": "e431738105898dd368735a6ebb1d3c19"}, + {"kind": "MAT", "name": "robot blue", "index": 0, "id": "1c5c7311b02f9c27a26158a5dfbbfe47"}, + {"kind": "MAT", "name": "robot gold", "index": 1, "id": "985941907b0208e6eaef9ca10b6db8ba"}, + {"kind": "MAT", "name": "robot joints", "index": 2, "id": "d9b762e720d363036a57efa7b616eaef"}, + {"kind": "MAT", "name": "robot visor", "index": 3, "id": "ae20beadede617fdd239a5a6ad23eb83"}, + {"kind": "SKEL", "name": "13 joint robot rig", "index": 0, "id": "83d297850b4eccf18f4225970eff51fb", "owner": "e431738105898dd368735a6ebb1d3c19"}, + {"kind": "MESH", "name": "Mirai robot", "index": 0, "id": "941beb67913839d9f4d4e05528cb0c80", "part": true, "owner": "e431738105898dd368735a6ebb1d3c19", "dependencies": ["1c5c7311b02f9c27a26158a5dfbbfe47", "83d297850b4eccf18f4225970eff51fb", "985941907b0208e6eaef9ca10b6db8ba", "ae20beadede617fdd239a5a6ad23eb83", "d9b762e720d363036a57efa7b616eaef"]} + ] +} diff --git a/demo/packages/2d6b0454a36971219d185d940f836ac7.mirai b/demo/packages/2d6b0454a36971219d185d940f836ac7.mirai new file mode 100644 index 000000000..de397c2ba --- /dev/null +++ b/demo/packages/2d6b0454a36971219d185d940f836ac7.mirai @@ -0,0 +1,14 @@ +{ + "version": 1, + "id": "2d6b0454a36971219d185d940f836ac7", + "name": "marble_normal", + "origin": {"path": "assets/surfaces/marble/marble_normal.png", "folder": false, "importer": "image"}, + "identity": "bc6f7c7783c0abc4aa24aa5951e9358ffcce1a75eabeba4a844305b51a82fecd", + "settings": { + "image": "Normal" + }, + "inputs": [], + "assets": [ + {"kind": "TEX", "name": "marble_normal", "index": 0, "id": "6efbfe1939e3a2a35c0ccc8621cc6abb"} + ] +} diff --git a/demo/packages/2dbac73486cbcaad3569e05ecd01bb4f.mirai b/demo/packages/2dbac73486cbcaad3569e05ecd01bb4f.mirai new file mode 100644 index 000000000..3cb3dd789 --- /dev/null +++ b/demo/packages/2dbac73486cbcaad3569e05ecd01bb4f.mirai @@ -0,0 +1,14 @@ +{ + "version": 1, + "id": "2dbac73486cbcaad3569e05ecd01bb4f", + "name": "tree_tall", + "origin": {"path": "assets/foliage_extreme/tree_tall.obj", "folder": false, "importer": "OBJ"}, + "identity": "3259aa4d9b011a2cd76a9d6f297fe105a7ce5ad76f1a3591db932171c43a4c80", + "inputs": ["trees.mtl"], + "assets": [ + {"kind": "SCEN", "name": "tree_tall", "index": 0, "id": "10f19381013cd286f491234605c5ca3a"}, + {"kind": "MAT", "name": "bark", "index": 0, "id": "1e0d094f208a1c4fd1832e6810416f45"}, + {"kind": "MAT", "name": "leaf", "index": 1, "id": "d447cd3c68bfe38f537484e2c968b98e"}, + {"kind": "MESH", "name": "tree_tall", "index": 0, "id": "08c37e88a5a8125dd0c6c67b2fb2cc40", "part": true, "owner": "10f19381013cd286f491234605c5ca3a", "dependencies": ["1e0d094f208a1c4fd1832e6810416f45", "d447cd3c68bfe38f537484e2c968b98e"]} + ] +} diff --git a/demo/packages/3df37c7e292cc0578d432dfdb3489314.mirai b/demo/packages/3df37c7e292cc0578d432dfdb3489314.mirai new file mode 100644 index 000000000..ccd6a98d7 --- /dev/null +++ b/demo/packages/3df37c7e292cc0578d432dfdb3489314.mirai @@ -0,0 +1,15 @@ +{ + "version": 1, + "id": "3df37c7e292cc0578d432dfdb3489314", + "name": "bush", + "origin": {"path": "assets/forest/bush.gltf", "folder": false, "importer": "glTF"}, + "identity": "28e19acd32a28538e3db9c40143f10c02e7e8a08e3074331c48e40485f284a47", + "inputs": ["bush.bin", "leaves_color.png"], + "assets": [ + {"kind": "SCEN", "name": "bush", "index": 0, "id": "48eed1e5335b875bef27025acf93953a"}, + {"kind": "TEX", "name": "leaves_color", "index": 0, "id": "33f31fe1b5c654da025e160204ebcd70"}, + {"kind": "MAT", "name": "Bark", "index": 0, "id": "f9bebb3af1fed4d0e8103c0e0387b11a"}, + {"kind": "MAT", "name": "Leaves", "index": 1, "id": "e26eb420ab436a838111b0626b6a8a79", "dependencies": ["33f31fe1b5c654da025e160204ebcd70"]}, + {"kind": "MESH", "name": "Bush", "index": 0, "id": "88ceb31480f8e2c656ca94fb40dbda3d", "part": true, "owner": "48eed1e5335b875bef27025acf93953a", "dependencies": ["e26eb420ab436a838111b0626b6a8a79", "f9bebb3af1fed4d0e8103c0e0387b11a"]} + ] +} diff --git a/demo/packages/44342052da337fa065c60cf296a4466a.mirai b/demo/packages/44342052da337fa065c60cf296a4466a.mirai new file mode 100644 index 000000000..d6565cc96 --- /dev/null +++ b/demo/packages/44342052da337fa065c60cf296a4466a.mirai @@ -0,0 +1,14 @@ +{ + "version": 1, + "id": "44342052da337fa065c60cf296a4466a", + "name": "church_glass_color", + "origin": {"path": "assets/church/church_glass_color.png", "folder": false, "importer": "image"}, + "identity": "7a992dadb99729c9515c80e3b85b47c43e13b44bdfbedd0ff0815211b10596b7", + "settings": { + "image": "Color" + }, + "inputs": [], + "assets": [ + {"kind": "TEX", "name": "church_glass_color", "index": 0, "id": "2767edae7841cea315b1e40d922c9950"} + ] +} diff --git a/demo/packages/4840a3f5a151cbe8c1b403b30fe8240d.mirai b/demo/packages/4840a3f5a151cbe8c1b403b30fe8240d.mirai new file mode 100644 index 000000000..22666e6dc --- /dev/null +++ b/demo/packages/4840a3f5a151cbe8c1b403b30fe8240d.mirai @@ -0,0 +1,14 @@ +{ + "version": 1, + "id": "4840a3f5a151cbe8c1b403b30fe8240d", + "name": "sky_overcast", + "origin": {"path": "assets/sky/sky_overcast.jpg", "folder": false, "importer": "image"}, + "identity": "badb56de863cc885d6f6554fa75d6f109b40416b073e2a3ae1e798674bfa030d", + "settings": { + "image": "Color" + }, + "inputs": [], + "assets": [ + {"kind": "TEX", "name": "sky_overcast", "index": 0, "id": "88a3a1a16a759ed20966d1c3ff54ada1"} + ] +} diff --git a/demo/packages/641ffc8940c220c3b2460f7a9eaa7862.mirai b/demo/packages/641ffc8940c220c3b2460f7a9eaa7862.mirai new file mode 100644 index 000000000..8cd88cc53 --- /dev/null +++ b/demo/packages/641ffc8940c220c3b2460f7a9eaa7862.mirai @@ -0,0 +1,14 @@ +{ + "version": 1, + "id": "641ffc8940c220c3b2460f7a9eaa7862", + "name": "brushed_normal", + "origin": {"path": "assets/surfaces/metal/brushed_normal.png", "folder": false, "importer": "image"}, + "identity": "ccde0d801f6b101dc0a1ee36b07dad836fe33412f22d3ac32723076196627c90", + "settings": { + "image": "Normal" + }, + "inputs": [], + "assets": [ + {"kind": "TEX", "name": "brushed_normal", "index": 0, "id": "b9466f1cd86f5f900d74e12980fdc702"} + ] +} diff --git a/demo/packages/6498e1a94e6c4621f925d22f8384243a.mirai b/demo/packages/6498e1a94e6c4621f925d22f8384243a.mirai new file mode 100644 index 000000000..5b41c9903 --- /dev/null +++ b/demo/packages/6498e1a94e6c4621f925d22f8384243a.mirai @@ -0,0 +1,16 @@ +{ + "version": 1, + "id": "6498e1a94e6c4621f925d22f8384243a", + "name": "plant", + "origin": {"path": "assets/gallery/plant.gltf", "folder": false, "importer": "glTF"}, + "identity": "5d76a337b138857552efbbc0ff81c70d8e38cf597e18c05eaa8f03ac8b63b278", + "inputs": ["leaves_color.png"], + "assets": [ + {"kind": "SCEN", "name": "plant", "index": 0, "id": "fc120101f1ddb302e135784184f05931"}, + {"kind": "TEX", "name": "leaves_color", "index": 0, "id": "dba1b6bd3e8cde629938d41dc8934d02"}, + {"kind": "MAT", "name": "Terracotta", "index": 0, "id": "e2a8b0fcb9401808520b370016df35c9"}, + {"kind": "MAT", "name": "Soil", "index": 1, "id": "9337601926ea70572a4442c984759e15"}, + {"kind": "MAT", "name": "Leaves", "index": 2, "id": "7e248f2df1c8c849ff972c9a1ce95fb5", "dependencies": ["dba1b6bd3e8cde629938d41dc8934d02"]}, + {"kind": "MESH", "name": "Plant", "index": 0, "id": "72d9220e763e42b1da3107e0e5256b90", "part": true, "owner": "fc120101f1ddb302e135784184f05931", "dependencies": ["7e248f2df1c8c849ff972c9a1ce95fb5", "9337601926ea70572a4442c984759e15", "e2a8b0fcb9401808520b370016df35c9"]} + ] +} diff --git a/demo/packages/65805c17b2454730221e3d63066ce7f5.mirai b/demo/packages/65805c17b2454730221e3d63066ce7f5.mirai new file mode 100644 index 000000000..39cd3441b --- /dev/null +++ b/demo/packages/65805c17b2454730221e3d63066ce7f5.mirai @@ -0,0 +1,14 @@ +{ + "version": 1, + "id": "65805c17b2454730221e3d63066ce7f5", + "name": "grate_normal", + "origin": {"path": "assets/surfaces/metal/grate_normal.png", "folder": false, "importer": "image"}, + "identity": "84317f85bc995fe3ab6ae9962f1cd05bfd423bd05d87a2183232ecbececa02ca", + "settings": { + "image": "Normal" + }, + "inputs": [], + "assets": [ + {"kind": "TEX", "name": "grate_normal", "index": 0, "id": "c9d9c9f0f1dea3ec0ceec0f9671187eb"} + ] +} diff --git a/demo/packages/6cc37d40c23ded0fb9cc963a4dec2fbb.mirai b/demo/packages/6cc37d40c23ded0fb9cc963a4dec2fbb.mirai new file mode 100644 index 000000000..0580d0f42 --- /dev/null +++ b/demo/packages/6cc37d40c23ded0fb9cc963a4dec2fbb.mirai @@ -0,0 +1,14 @@ +{ + "version": 1, + "id": "6cc37d40c23ded0fb9cc963a4dec2fbb", + "name": "stained_glass_color", + "origin": {"path": "assets/surfaces/glass/stained_glass_color.png", "folder": false, "importer": "image"}, + "identity": "d5232abb2f9a961f16fd3e896f55ae94418df3baa00ced6f3ed776b29c4e34cc", + "settings": { + "image": "Color" + }, + "inputs": [], + "assets": [ + {"kind": "TEX", "name": "stained_glass_color", "index": 0, "id": "601bad68a6b7e349bc2655ac13c9196f"} + ] +} diff --git a/demo/packages/7572bd9ffa0dd2354005eff39b25f1bb.mirai b/demo/packages/7572bd9ffa0dd2354005eff39b25f1bb.mirai new file mode 100644 index 000000000..045fc563a --- /dev/null +++ b/demo/packages/7572bd9ffa0dd2354005eff39b25f1bb.mirai @@ -0,0 +1,11 @@ +{ + "version": 1, + "id": "7572bd9ffa0dd2354005eff39b25f1bb", + "name": "hum", + "origin": {"path": "assets/showcase/hum.wav", "folder": false, "importer": "sound"}, + "identity": "a136b93fa9f126831d67ac5f21d3453fd3c404372406ad7cbbca848e5f8cf9db", + "inputs": [], + "assets": [ + {"kind": "SND", "name": "hum", "index": 0, "id": "7f27bbcfa7a4b5fac01753762f745b26"} + ] +} diff --git a/demo/packages/780bee6597fbb860e803d513bda2dee7.mirai b/demo/packages/780bee6597fbb860e803d513bda2dee7.mirai new file mode 100644 index 000000000..288895207 --- /dev/null +++ b/demo/packages/780bee6597fbb860e803d513bda2dee7.mirai @@ -0,0 +1,16 @@ +{ + "version": 1, + "id": "780bee6597fbb860e803d513bda2dee7", + "name": "jelly", + "origin": {"path": "assets/characters/jelly/jelly.gltf", "folder": false, "importer": "glTF"}, + "identity": "a8aa4182d7481e139c7925b475ece930212b1ddf640d0b4da7342ca25c2d26af", + "inputs": ["jelly.bin"], + "assets": [ + {"kind": "SCEN", "name": "jelly", "index": 0, "id": "107c9b8f4bf9dfe8936235eae8e042a6"}, + {"kind": "MAT", "name": "Candy", "index": 0, "id": "a18adcad4fedf21125038a819d27acd2"}, + {"kind": "SKEL", "name": "Jelly bones", "index": 0, "id": "dae452ce18d59e780ff745f7d1b43697", "owner": "107c9b8f4bf9dfe8936235eae8e042a6"}, + {"kind": "MESH", "name": "Party Jelly", "index": 0, "id": "25a3e22ecb9c265a1614515f868efc52", "part": true, "owner": "107c9b8f4bf9dfe8936235eae8e042a6", "dependencies": ["a18adcad4fedf21125038a819d27acd2"]}, + {"kind": "MESH", "name": "Party Jelly (1)", "index": 1, "id": "1c00bfb5bceb2619036b114104f42040", "part": true, "owner": "107c9b8f4bf9dfe8936235eae8e042a6", "dependencies": ["a18adcad4fedf21125038a819d27acd2", "dae452ce18d59e780ff745f7d1b43697"]}, + {"kind": "ANIM", "name": "jelly", "index": 0, "id": "0d555b0da56949fb3cdbdf41410432f4", "dependencies": ["dae452ce18d59e780ff745f7d1b43697"]} + ] +} diff --git a/demo/packages/81d18317bfbbf0851f30ebcb2de41b67.mirai b/demo/packages/81d18317bfbbf0851f30ebcb2de41b67.mirai new file mode 100644 index 000000000..4ba5c9d0e --- /dev/null +++ b/demo/packages/81d18317bfbbf0851f30ebcb2de41b67.mirai @@ -0,0 +1,11 @@ +{ + "version": 1, + "id": "81d18317bfbbf0851f30ebcb2de41b67", + "name": "grate_orm", + "origin": {"path": "assets/surfaces/metal/grate_orm.png", "folder": false, "importer": "image"}, + "identity": "a135326e4f00d4d96a7c4e6b64637be8940ab66db13f2d4747db018ce7c753f3", + "inputs": [], + "assets": [ + {"kind": "TEX", "name": "grate_orm", "index": 0, "id": "71e4b6effc8f9e4ac6c7f1dc0f02612e"} + ] +} diff --git a/demo/packages/81dd55e2e057dae790850f4a006dce86.mirai b/demo/packages/81dd55e2e057dae790850f4a006dce86.mirai new file mode 100644 index 000000000..43f02ae55 --- /dev/null +++ b/demo/packages/81dd55e2e057dae790850f4a006dce86.mirai @@ -0,0 +1,14 @@ +{ + "version": 1, + "id": "81dd55e2e057dae790850f4a006dce86", + "name": "tree_broad", + "origin": {"path": "assets/foliage_extreme/tree_broad.obj", "folder": false, "importer": "OBJ"}, + "identity": "6d2d4abbfd3f449acaf45b90767efd7adebd784081516a03ba0eff0df68583ff", + "inputs": ["trees.mtl"], + "assets": [ + {"kind": "SCEN", "name": "tree_broad", "index": 0, "id": "73668be537ffad086162e2c98e1c79f6"}, + {"kind": "MAT", "name": "bark", "index": 0, "id": "031b5c900785b6de16ce2ab9904c2288"}, + {"kind": "MAT", "name": "leaf", "index": 1, "id": "2012dad11d76c044f98348272649af04"}, + {"kind": "MESH", "name": "tree_broad", "index": 0, "id": "f745c4a330cd0cdd5a593bdf377d89ca", "part": true, "owner": "73668be537ffad086162e2c98e1c79f6", "dependencies": ["031b5c900785b6de16ce2ab9904c2288", "2012dad11d76c044f98348272649af04"]} + ] +} diff --git a/demo/packages/84060b2db086d874fb6a99a638a79a10.mirai b/demo/packages/84060b2db086d874fb6a99a638a79a10.mirai new file mode 100644 index 000000000..dae073294 --- /dev/null +++ b/demo/packages/84060b2db086d874fb6a99a638a79a10.mirai @@ -0,0 +1,17 @@ +{ + "version": 1, + "id": "84060b2db086d874fb6a99a638a79a10", + "name": "robot", + "origin": {"path": "assets/characters/robot/robot.gltf", "folder": false, "importer": "glTF"}, + "identity": "4e42171c4b4f4eeb1e541f249c84ef5c3c9e2d15c0224e9dc3371ebbe5d37c53", + "inputs": ["robot.bin"], + "assets": [ + {"kind": "SCEN", "name": "robot", "index": 0, "id": "b0f0241df6e014392af304b56cf59eab"}, + {"kind": "MAT", "name": "robot blue", "index": 0, "id": "c67757fb8b041b0d6b6178fe958bd91e"}, + {"kind": "MAT", "name": "robot gold", "index": 1, "id": "feebc67eaeb9081ec3d1f1edac208b2a"}, + {"kind": "MAT", "name": "robot joints", "index": 2, "id": "a453277713f405f86fd3dc27c2506977"}, + {"kind": "MAT", "name": "robot visor", "index": 3, "id": "0bf1bd3d54bcb875b34957598640c69d"}, + {"kind": "SKEL", "name": "13 joint robot rig", "index": 0, "id": "cf5b1fe881d020461927fe9b85c42136", "owner": "b0f0241df6e014392af304b56cf59eab"}, + {"kind": "MESH", "name": "Mirai robot", "index": 0, "id": "3a94f8ad990aede6566ad2af8010bb23", "part": true, "owner": "b0f0241df6e014392af304b56cf59eab", "dependencies": ["0bf1bd3d54bcb875b34957598640c69d", "a453277713f405f86fd3dc27c2506977", "c67757fb8b041b0d6b6178fe958bd91e", "cf5b1fe881d020461927fe9b85c42136", "feebc67eaeb9081ec3d1f1edac208b2a"]} + ] +} diff --git a/demo/packages/856074124b2e8924607a1e79488b6abd.mirai b/demo/packages/856074124b2e8924607a1e79488b6abd.mirai new file mode 100644 index 000000000..0e67bb0e6 --- /dev/null +++ b/demo/packages/856074124b2e8924607a1e79488b6abd.mirai @@ -0,0 +1,14 @@ +{ + "version": 1, + "id": "856074124b2e8924607a1e79488b6abd", + "name": "neon_emission", + "origin": {"path": "assets/surfaces/neon/neon_emission.png", "folder": false, "importer": "image"}, + "identity": "e7133ec38468f5414b4e8f8a65e57e4e065b176b21049819ef50ca0a852eed76", + "settings": { + "image": "Color" + }, + "inputs": [], + "assets": [ + {"kind": "TEX", "name": "neon_emission", "index": 0, "id": "6d9362793631542f0b778005f9fd8dd0"} + ] +} diff --git a/demo/packages/9c80ffb4545f5d74f0a1ccf6acedba6b.mirai b/demo/packages/9c80ffb4545f5d74f0a1ccf6acedba6b.mirai new file mode 100644 index 000000000..9bcde715b --- /dev/null +++ b/demo/packages/9c80ffb4545f5d74f0a1ccf6acedba6b.mirai @@ -0,0 +1,14 @@ +{ + "version": 1, + "id": "9c80ffb4545f5d74f0a1ccf6acedba6b", + "name": "tree_bush", + "origin": {"path": "assets/foliage_extreme/tree_bush.obj", "folder": false, "importer": "OBJ"}, + "identity": "ade0468efbb5b7d565235873a7fb3b8f09186d1e1c8ade8e4b8eaa226147f748", + "inputs": ["trees.mtl"], + "assets": [ + {"kind": "SCEN", "name": "tree_bush", "index": 0, "id": "317c6ad95fead4485707ae3231936f9b"}, + {"kind": "MAT", "name": "bark", "index": 0, "id": "b96df7055aceeb7248fcde10158da5cc"}, + {"kind": "MAT", "name": "leaf", "index": 1, "id": "fbcbc189b6096b232b1c8ff070f19043"}, + {"kind": "MESH", "name": "tree_bush", "index": 0, "id": "61539a5b43404159745748d7814cab3c", "part": true, "owner": "317c6ad95fead4485707ae3231936f9b", "dependencies": ["b96df7055aceeb7248fcde10158da5cc", "fbcbc189b6096b232b1c8ff070f19043"]} + ] +} diff --git a/demo/packages/a03a7d06fb4c433e89ba1ecd314d91ed.mirai b/demo/packages/a03a7d06fb4c433e89ba1ecd314d91ed.mirai new file mode 100644 index 000000000..ac0bdfa3b --- /dev/null +++ b/demo/packages/a03a7d06fb4c433e89ba1ecd314d91ed.mirai @@ -0,0 +1,11 @@ +{ + "version": 1, + "id": "a03a7d06fb4c433e89ba1ecd314d91ed", + "name": "brushed_orm", + "origin": {"path": "assets/surfaces/metal/brushed_orm.png", "folder": false, "importer": "image"}, + "identity": "1ec644f5675f0f31c435f1531200733cd7404a1dfc65c6ff16e37b9ac6040feb", + "inputs": [], + "assets": [ + {"kind": "TEX", "name": "brushed_orm", "index": 0, "id": "ec4b617cf48145349c2c76f465e19ae6"} + ] +} diff --git a/demo/packages/a270e82878cb164498ba98398060ccb8.mirai b/demo/packages/a270e82878cb164498ba98398060ccb8.mirai new file mode 100644 index 000000000..45812d8f0 --- /dev/null +++ b/demo/packages/a270e82878cb164498ba98398060ccb8.mirai @@ -0,0 +1,14 @@ +{ + "version": 1, + "id": "a270e82878cb164498ba98398060ccb8", + "name": "tiles_color", + "origin": {"path": "assets/surfaces/tiles/tiles_color.png", "folder": false, "importer": "image"}, + "identity": "3915769e8b5d9cc9af39584990a4fb69ac1f58764adc66d3fb4094005497f149", + "settings": { + "image": "Color" + }, + "inputs": [], + "assets": [ + {"kind": "TEX", "name": "tiles_color", "index": 0, "id": "cd3d66765e834e0c508906257f888eba"} + ] +} diff --git a/demo/packages/a278db7076e649a48933fc75b0dc1a4c.mirai b/demo/packages/a278db7076e649a48933fc75b0dc1a4c.mirai new file mode 100644 index 000000000..1cf0235e5 --- /dev/null +++ b/demo/packages/a278db7076e649a48933fc75b0dc1a4c.mirai @@ -0,0 +1,14 @@ +{ + "version": 1, + "id": "a278db7076e649a48933fc75b0dc1a4c", + "name": "cobblestone_normal", + "origin": {"path": "assets/surfaces/cobblestone/cobblestone_normal.png", "folder": false, "importer": "image"}, + "identity": "c96c505d447b06db5058734fe8e034334fba141d57eaeca7f85ae25802a45f68", + "settings": { + "image": "Normal" + }, + "inputs": [], + "assets": [ + {"kind": "TEX", "name": "cobblestone_normal", "index": 0, "id": "1a08080be2a08b5fe300ec425923c195"} + ] +} diff --git a/demo/packages/a3799fb21f329ae819440d1726f2aa20.mirai b/demo/packages/a3799fb21f329ae819440d1726f2aa20.mirai new file mode 100644 index 000000000..8926a2be4 --- /dev/null +++ b/demo/packages/a3799fb21f329ae819440d1726f2aa20.mirai @@ -0,0 +1,15 @@ +{ + "version": 1, + "id": "a3799fb21f329ae819440d1726f2aa20", + "name": "vase", + "origin": {"path": "assets/gallery/vase.obj", "folder": false, "importer": "OBJ"}, + "identity": "5875aea8a04578f3158c7f275449d9b33f78c1b058906552335a81afd01f7fe4", + "inputs": ["vase.mtl", "vase_color.png"], + "assets": [ + {"kind": "SCEN", "name": "vase", "index": 0, "id": "f430e11663e1ae229bf1511ac0c7f82d"}, + {"kind": "TEX", "name": "vase_color", "index": 0, "id": "f8b0959094ef4d6757a87f85ec92a493"}, + {"kind": "MAT", "name": "Ceramic", "index": 0, "id": "757da5a655983f642167f358f7add279", "dependencies": ["f8b0959094ef4d6757a87f85ec92a493"]}, + {"kind": "MAT", "name": "Gilding", "index": 1, "id": "be0bbaf3f4f7c9fc87a15c7d08b0bb18"}, + {"kind": "MESH", "name": "Vase", "index": 0, "id": "0d6f52247150edd52ac17287d54e7bf9", "part": true, "owner": "f430e11663e1ae229bf1511ac0c7f82d", "dependencies": ["757da5a655983f642167f358f7add279", "be0bbaf3f4f7c9fc87a15c7d08b0bb18"]} + ] +} diff --git a/demo/packages/a68982d58a94bfa0fcb588bf93dfafec.mirai b/demo/packages/a68982d58a94bfa0fcb588bf93dfafec.mirai new file mode 100644 index 000000000..4c5f5539f --- /dev/null +++ b/demo/packages/a68982d58a94bfa0fcb588bf93dfafec.mirai @@ -0,0 +1,11 @@ +{ + "version": 1, + "id": "a68982d58a94bfa0fcb588bf93dfafec", + "name": "marble_orm", + "origin": {"path": "assets/surfaces/marble/marble_orm.png", "folder": false, "importer": "image"}, + "identity": "318157572d9864452082d14787cc9ac8eb7cf3ce56cf81609d20996ef34a273b", + "inputs": [], + "assets": [ + {"kind": "TEX", "name": "marble_orm", "index": 0, "id": "972fc9634ba6242d0b5f33e03e3a0741"} + ] +} diff --git a/demo/packages/ae40765be8d60ddad02e47bb21203f8f.mirai b/demo/packages/ae40765be8d60ddad02e47bb21203f8f.mirai new file mode 100644 index 000000000..d748f76a4 --- /dev/null +++ b/demo/packages/ae40765be8d60ddad02e47bb21203f8f.mirai @@ -0,0 +1,11 @@ +{ + "version": 1, + "id": "ae40765be8d60ddad02e47bb21203f8f", + "name": "cobblestone_orm", + "origin": {"path": "assets/surfaces/cobblestone/cobblestone_orm.png", "folder": false, "importer": "image"}, + "identity": "2a95d7c8e5402ec5bdc2f7e01aed3c0ed96d37d9ed066c08c7c1e944ac88a630", + "inputs": [], + "assets": [ + {"kind": "TEX", "name": "cobblestone_orm", "index": 0, "id": "94cef7c053d5afed918f83a3d6242853"} + ] +} diff --git a/demo/packages/c0d7666d947e65fc7dd68a31013c0939.mirai b/demo/packages/c0d7666d947e65fc7dd68a31013c0939.mirai new file mode 100644 index 000000000..7069a3086 --- /dev/null +++ b/demo/packages/c0d7666d947e65fc7dd68a31013c0939.mirai @@ -0,0 +1,14 @@ +{ + "version": 1, + "id": "c0d7666d947e65fc7dd68a31013c0939", + "name": "cobblestone_color", + "origin": {"path": "assets/surfaces/cobblestone/cobblestone_color.png", "folder": false, "importer": "image"}, + "identity": "4bf20aa0c9be138d32c781598a709118b64335c81d7d9b86aec7b3ddf9efbf07", + "settings": { + "image": "Color" + }, + "inputs": [], + "assets": [ + {"kind": "TEX", "name": "cobblestone_color", "index": 0, "id": "9a69320baec0a94ab9eb8d2010a8a8a4"} + ] +} diff --git a/demo/packages/c1a66f64c05efe53cca206765ab98117.mirai b/demo/packages/c1a66f64c05efe53cca206765ab98117.mirai new file mode 100644 index 000000000..f5ab52504 --- /dev/null +++ b/demo/packages/c1a66f64c05efe53cca206765ab98117.mirai @@ -0,0 +1,14 @@ +{ + "version": 1, + "id": "c1a66f64c05efe53cca206765ab98117", + "name": "sky_night", + "origin": {"path": "assets/sky/sky_night.jpg", "folder": false, "importer": "image"}, + "identity": "a6bc49bf6d7489954dafb9d55d279df060250cc8f6d8366cbd48b1501a6ee048", + "settings": { + "image": "Color" + }, + "inputs": [], + "assets": [ + {"kind": "TEX", "name": "sky_night", "index": 0, "id": "8a6329018eb7b53a0215bd4bdbf5a2c9"} + ] +} diff --git a/demo/packages/ca27d206e666815335b68ee9c8900ab7.mirai b/demo/packages/ca27d206e666815335b68ee9c8900ab7.mirai new file mode 100644 index 000000000..0e1970ea2 --- /dev/null +++ b/demo/packages/ca27d206e666815335b68ee9c8900ab7.mirai @@ -0,0 +1,14 @@ +{ + "version": 1, + "id": "ca27d206e666815335b68ee9c8900ab7", + "name": "vase_color", + "origin": {"path": "assets/gallery/vase_color.png", "folder": false, "importer": "image"}, + "identity": "f2fb83e9ce3a4a2aeba751a76a796edb4ee13ec9450b9011befd02e0348d1c30", + "settings": { + "image": "Color" + }, + "inputs": [], + "assets": [ + {"kind": "TEX", "name": "vase_color", "index": 0, "id": "c94299d672a2e4371f7a5374961b418c"} + ] +} diff --git a/demo/packages/caae575f45863b717aafbe7fe73862c0.mirai b/demo/packages/caae575f45863b717aafbe7fe73862c0.mirai new file mode 100644 index 000000000..ea539fcd0 --- /dev/null +++ b/demo/packages/caae575f45863b717aafbe7fe73862c0.mirai @@ -0,0 +1,15 @@ +{ + "version": 1, + "id": "caae575f45863b717aafbe7fe73862c0", + "name": "leaves_color", + "origin": {"path": "assets/gallery/leaves_color.png", "folder": false, "importer": "image"}, + "identity": "30b62e5a3a6b818103845cb1db9d509622a31605dd8f4cdc740053311340fa93", + "settings": { + "image": "Color", + "alpha_coverage": 0.5 + }, + "inputs": [], + "assets": [ + {"kind": "TEX", "name": "leaves_color", "index": 0, "id": "df49540f1f9dbcf6fe2e612ea427757e"} + ] +} diff --git a/demo/packages/de7548cbbad3494b21280744cd16ed68.mirai b/demo/packages/de7548cbbad3494b21280744cd16ed68.mirai new file mode 100644 index 000000000..2099e9dad --- /dev/null +++ b/demo/packages/de7548cbbad3494b21280744cd16ed68.mirai @@ -0,0 +1,11 @@ +{ + "version": 1, + "id": "de7548cbbad3494b21280744cd16ed68", + "name": "tiles_orm", + "origin": {"path": "assets/surfaces/tiles/tiles_orm.png", "folder": false, "importer": "image"}, + "identity": "032ec4ed33e3897d8890d072eaf44ae99ea73024adfe597e062e583e61c06b88", + "inputs": [], + "assets": [ + {"kind": "TEX", "name": "tiles_orm", "index": 0, "id": "84dbfd2533a249a9716c78d548c6c654"} + ] +} diff --git a/demo/packages/e022a803a1c6fb9fe4b86ab65a48bc24.mirai b/demo/packages/e022a803a1c6fb9fe4b86ab65a48bc24.mirai new file mode 100644 index 000000000..1973c3a07 --- /dev/null +++ b/demo/packages/e022a803a1c6fb9fe4b86ab65a48bc24.mirai @@ -0,0 +1,14 @@ +{ + "version": 1, + "id": "e022a803a1c6fb9fe4b86ab65a48bc24", + "name": "sky_sunset", + "origin": {"path": "assets/sky/sky_sunset.jpg", "folder": false, "importer": "image"}, + "identity": "0b4e80818fedc88b32a0b6550eec6158e2727bfd1d99253c232e50cb828668cb", + "settings": { + "image": "Color" + }, + "inputs": [], + "assets": [ + {"kind": "TEX", "name": "sky_sunset", "index": 0, "id": "7b819d061e0487f04a17587b8ff0bce1"} + ] +} diff --git a/demo/packages/e0495081646e8954b731fc082bcd8884.mirai b/demo/packages/e0495081646e8954b731fc082bcd8884.mirai new file mode 100644 index 000000000..3b211455b --- /dev/null +++ b/demo/packages/e0495081646e8954b731fc082bcd8884.mirai @@ -0,0 +1,15 @@ +{ + "version": 1, + "id": "e0495081646e8954b731fc082bcd8884", + "name": "grate_color", + "origin": {"path": "assets/surfaces/metal/grate_color.png", "folder": false, "importer": "image"}, + "identity": "f65a1b168299988cb3046b4003612437577bb217f7afbbeae8dada3c085b60c2", + "settings": { + "image": "Color", + "alpha_coverage": 0.5 + }, + "inputs": [], + "assets": [ + {"kind": "TEX", "name": "grate_color", "index": 0, "id": "90b31208c9dfc6a6d00ab5517f084571"} + ] +} diff --git a/demo/packages/e96915c067e6fe50d517b59bf3b17d61.mirai b/demo/packages/e96915c067e6fe50d517b59bf3b17d61.mirai new file mode 100644 index 000000000..5e25a58e7 --- /dev/null +++ b/demo/packages/e96915c067e6fe50d517b59bf3b17d61.mirai @@ -0,0 +1,14 @@ +{ + "version": 1, + "id": "e96915c067e6fe50d517b59bf3b17d61", + "name": "planks_color", + "origin": {"path": "assets/surfaces/wood/planks_color.png", "folder": false, "importer": "image"}, + "identity": "49264e2dfeefc900bf103a7d93f6c80ec28d75a409eb86d103cdf0c1313bf953", + "settings": { + "image": "Color" + }, + "inputs": [], + "assets": [ + {"kind": "TEX", "name": "planks_color", "index": 0, "id": "79a53ef6d585bc9fd1a912a9e592f913"} + ] +} diff --git a/demo/packages/f4ccf188ab1bf19a62c53faaef829fba.mirai b/demo/packages/f4ccf188ab1bf19a62c53faaef829fba.mirai new file mode 100644 index 000000000..a9e7b0588 --- /dev/null +++ b/demo/packages/f4ccf188ab1bf19a62c53faaef829fba.mirai @@ -0,0 +1,14 @@ +{ + "version": 1, + "id": "f4ccf188ab1bf19a62c53faaef829fba", + "name": "planks_normal", + "origin": {"path": "assets/surfaces/wood/planks_normal.png", "folder": false, "importer": "image"}, + "identity": "be4569af461f13cc92440f370b2e097d6c0e4fd8119656525b343b7695c98dcc", + "settings": { + "image": "Normal" + }, + "inputs": [], + "assets": [ + {"kind": "TEX", "name": "planks_normal", "index": 0, "id": "0558bb6aa0ec012cac33f59d343e99ed"} + ] +} diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index fcdb32964..bc7970440 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -322,7 +322,7 @@ project's world through Numbered layers; a package only imports lower layers. -- **0** `std/*`: standard library (base, math, serial, jobs, spatial, geom, anim, ai, input, +- **0** `std/*`: standard library (base, blake3, math, serial, jobs, spatial, geom, anim, ai, input, subprocess, ui_format). It does not depend on the engine. `std/compressor` is not imported as `mirai:`: it is the open implementation of the collection `compressor` (`data`, `texture`, `blocks`, the last engine-time), which every compile names and which the private @@ -330,10 +330,14 @@ Numbered layers; a package only imports lower layers. `std/ui_format` is what the structure of a `.ui` is judged by (node kinds, override fields, limits), which `ui` builds and `asset` checks without importing each other. - **1** `project`: `project.mirai` and `plugin.mirai` manifests, the layout of a project. +- **1** `store`: the machine-wide content-addressed store of objects and refs under one root. - **2** `net`, `asset`, `ui`, `audio`, `rhi`: network transport, asset formats, UI, mixer and GPU; - independent of each other; `asset` reads from `project` what a project's assets compress with. + independent of each other; `asset` reads from `project` what a project's assets compress with + and keeps in `store` the records of imports (package and build manifests, `packages.mirai`). `rhi/shared_device` is the device the GPU tests share; only tests import it, also the GPU tests of `rhi`, in `rhi/tests`. +- **3** `exchange`: objects and refs of `store` between peers over the connections of `net` hosts; + a headless peer and the editor link it. - **3** `render`, `platform`: hybrid renderer and SDL3. `render/abi` declares the constants the renderer shares with its shaders, on top of `asset` alone: `cook/shaders` imports it to write them for the shaders, since the tool cannot import `render`. @@ -357,7 +361,9 @@ Numbered layers; a package only imports lower layers. `cook/install` (a project's plugins that come from git, and the catalog). The importers are `cook/import` (core, no formats), one package per format (`cook/import/obj`, `gltf`, `usd`, `fbx`, which reads with the C library of `third_party/ufbx`) and `cook/import/sources` (the table of - formats and `import_source`, on top of `cook/prefab`). `cook/author` is what an interface or a + formats and `import_source`, on top of `cook/prefab`). `cook/share` shares a project peer to + peer: its snapshot, and its packages refreshed through the library from `exchange` peers. + `cook/author` is what an interface or a scene is authored with as a nested document, between its text and the file of the asset, on top of `asset`, `ui`, `core`, `host` and `engine/*`: the editor's `ui.*` methods and `mirai ui` and `mirai scene` go through it, and the presets an entity is made from are there. diff --git a/docs/ODIN_WORKAROUNDS.md b/docs/ODIN_WORKAROUNDS.md index 474ba357e..9154e84cc 100644 --- a/docs/ODIN_WORKAROUNDS.md +++ b/docs/ODIN_WORKAROUNDS.md @@ -30,6 +30,7 @@ Entries are for Odin `dev-2026-11`. | `base.remove_all` (`std/base/remove_all.odin`): every folder of the engine and its tests is removed with it instead of `os.remove_all`, whose Windows version calls `SHFileOperationW`; the shell raises first-chance C++ exceptions inside it, and the vectored handler of `core:testing` stops the test for any of them ("Signal caught: Unknown"), or kills the runner when the cleanup of a stopped test runs on the main thread | none | `os.remove_all` on Windows walks the folder with `DeleteFileW` and `RemoveDirectoryW`, and `core:testing` on Windows lets an exception that a handler of the program catches go by; go back to `os.remove_all` and run the editor tests of `ai_subagents_test.odin` and `sections_test.odin` together | | `base.make_directory_all` (`std/base/folders.odin`): takes `.Exist` for a folder that is already there as made, where a caller checks the result; the other callers accept `.Exist` with `os.is_dir` | odin-lang/Odin#8473 | on macOS and Linux, `os.make_directory_all` of a folder that is already there returns nil, as on Windows; go back to `os.make_directory_all` and run `mirai test std/base` | | `base.copy_directory_all` (`std/base/folders.odin`): every folder the engine and its tests copy; on macOS `os.copy_directory_all` hands the folder to `copyfile` with `COPYFILE_RECURSIVE`, which copies the source itself into a destination that is already there | odin-lang/Odin#8474 | on macOS, `os.copy_directory_all(dst, src)` with `dst` already a folder leaves `src/a.txt` at `dst/a.txt`; go back to it and run `mirai test cook/author` | +| `file_share_delete` (`store/system_windows.odin`): the store opens its files with its own `CreateFileW` sharing read, write and delete, and replaces and removes them with POSIX semantics; `os.open` on Windows shares only read and write, so a file someone holds open can be neither renamed over nor removed | none (patch and repro in `bin/store/odin-share-delete.md`, not published) | `os.open` of a file on Windows passes `FILE_SHARE_DELETE`, and `os.rename` over it while another handle from `os.open` holds it succeeds; go back to `os.open` in `file_open` and run `mirai test store` on Windows | | `constant_field_compare` (`engine/render/graphics_test.odin`): compares a field of a constant through `testing.expect_value` | [odin-lang/Odin#8321](https://github.com/odin-lang/Odin/issues/8321) | with `A :: struct {x:f32, e:E}` and `C :: A{x=1}`, `C.e == .Off` (a field the literal leaves out compared with `==`) is true; today it folds to false | ## Slang and SPIRV diff --git a/docs/game/assets.md b/docs/game/assets.md index 1dfa57976..93dbde806 100644 --- a/docs/game/assets.md +++ b/docs/game/assets.md @@ -32,10 +32,10 @@ mirai import assets/player/step.wav # assets/player/step.sound - A model writes meshes, materials, textures, skeletons, clips and a prefab: instantiate that. - An image is color unless its name says otherwise (`normal`, `_n`, `rough`, `metal`, `orm`, `_ao`, `height`, `mask`); force it with `--srgb`, `--data`, `--normal`, `--normal-green-down`. -- `.import` beside the source keeps its flags, folder and asset ids, so importing again - updates in place and keeps material edits; another edit stops it, which names the file, until - `--discard-changes`. Commit it and move it with its source; a renamed part is a new asset. -- Meshes, textures, sounds and fonts are built into `.mirai/imported/`, never committed: `mirai +- An import leaves the source where it is and pins its package in `packages.mirai`, manifest in + `packages/` (flags, folder, asset ids): importing again keeps ids and material edits, another + edit stops it until `--discard-changes`; a renamed part is a new asset. Commit both. +- Meshes, textures, sounds and fonts are built into the machine's store, never committed: `mirai build` makes them again on a fresh clone. Edit their source, not them. - A mesh keeps only the colliders its source brings; placed, it collides only with them. An `Auto` `Collider` without them is its triangles when still, convex pieces of them when it moves. diff --git a/docs/game/assets/content-tool.md b/docs/game/assets/content-tool.md index 9b4dc1114..60f2d38d9 100644 --- a/docs/game/assets/content-tool.md +++ b/docs/game/assets/content-tool.md @@ -55,9 +55,9 @@ hum:=report.assets[0].id // random the first time, kept after - Write the WAV only when its bytes differ from the file on disk, so an unchanged source is imported as the same file. -- The id an import returns is random the first time and the one `.import` keeps from then - on: read it with `asset.list`; never in a constant. Commit the WAV and its `.import`; the - `.sound` is built again from them. +- The id an import returns is random the first time and the one the package of the source keeps + from then on: read it with `asset.list`; never in a constant. Commit the WAV, `packages.mirai` + and `packages/`; the `.sound` is built again from them. ## Sound set @@ -79,8 +79,8 @@ A tool can import art too, with the function `mirai import` calls: report,error:=sources.import_source("assets/arena/crate.glb") ``` -`sources` is `mirai:cook/import/sources`. It keeps the settings and the ids in `.import` -beside the source, so a tool and `mirai import` give the same assets. +`sources` is `mirai:cook/import/sources`. It keeps the settings and the ids in the package of the +source, so a tool and `mirai import` give the same assets. ## Pitfalls that fail silently diff --git a/docs/game/networking.md b/docs/game/networking.md index b0418c3cc..3703c6244 100644 --- a/docs/game/networking.md +++ b/docs/game/networking.md @@ -26,7 +26,7 @@ mirai play --server # serves and plays mirai play --connect 127.0.0.1 --new-player # a client; --new-player: own player mirai play --connect 127.0.0.1 --latency 60 --loss 2 # --jitter too mirai play --server --invite # serves only who brings its code -mirai play --connect R2M0-253B-DK5G-0W87-M0MA-3CP3-TKJZ-C1RR-EJT +mirai play --connect # the code its server prints mirai build server # dedicated build, run with --dedicated ``` diff --git a/docs/game/ship.md b/docs/game/ship.md index 2d1fe6596..6b2f1264c 100644 --- a/docs/game/ship.md +++ b/docs/game/ship.md @@ -53,6 +53,6 @@ says how to fix each finding. - An asset only code names works in `mirai play` and is missing from `build game` unless it is a [root](assets.md#reference). Build before delivering. -- Art sources and their sidecars never ship, and a game package that imports an editor package - fails the build with `Engine_Time_Code`. +- Art sources and the manifests of their packages never ship, and a game package that imports + an editor package fails the build with `Engine_Time_Code`. - The shipped game on Windows is a console program. diff --git a/editor/ai_chat.odin b/editor/ai_chat.odin index 8c2ae5eb9..4692b5706 100644 --- a/editor/ai_chat.odin +++ b/editor/ai_chat.odin @@ -612,7 +612,7 @@ ai_chat_content_build :: proc(s:^Session,width,height:f32,size:f32=0) { switch g.drag.kind { case .Asset: ai_attachment_add(s,g.drag.asset_id) case .Entity: if g.drag.entity!=core.NONE {history.selection_add(s.selection,g.drag.entity);m.focus=true} - case .None,.Preset: + case .None,.Preset,.Package: } g.drag={} } diff --git a/editor/api_assets.odin b/editor/api_assets.odin index ffff0bae0..409a99011 100644 --- a/editor/api_assets.odin +++ b/editor/api_assets.odin @@ -83,7 +83,7 @@ Api_Detach_Result :: struct { // What api_asset_delete (asset_delete.odin) writes. Api_Asset_Delete_Result :: struct { - moved:[]string `mirai:"label=The paths moved to the trash: the asset and its parts"`, + moved:[]string `mirai:"label=The paths that left the project: the asset and its parts, moved to the trash, and what the package it is the root of listed"`, used_by:[]string `mirai:"label=The files that use what moved; they keep the reference"`, error:string `mirai:"optional,label=Why the rest could not be moved after some were"`, } @@ -91,13 +91,17 @@ Api_Asset_Delete_Result :: struct { @(private) api_assets_register :: proc(s:^Session,st:^Api_State) { assets:=Api_Asset_Params{limit=256} + add_default:=Api_Packages_Add_Params{seconds=API_IMPORT_SECONDS} api_register(s,st,{name="component.list",summary="The components a scene can hold, with their defaults and their fields as scene.write takes them.",params=type_info_of(Api_Component_Params),result=type_info_of(Api_Component_List),flags={.Core,.Read_Only},run=api_component_list}) api_register(s,st,{name="asset.list",summary="The assets of the registry by kind and folder, with a count of its broken files, duplicated ids and missing dependencies.",params=type_info_of(Api_Asset_Params),result=type_info_of(Api_Asset_List),defaults=&assets,flags={.Core,.Read_Only},run=api_asset_list}) api_register(s,st,{name="asset.detach",summary="Takes a part of an import out of the asset that owns it so it shows on its own; the import still writes it and leaves it out of the owner.",params=type_info_of(Api_Detach_Params),result=type_info_of(Api_Detach_Result),run=api_asset_detach}) api_register(s,st,{name="asset.create",summary="Makes an empty asset in a folder of the assets as Create of the content browser does; material.set, particle.set and the like fill it.",params=type_info_of(Api_Asset_Create_Params),result=type_info_of(Api_Asset_Create_Result),run=api_asset_create}) api_register(s,st,{name="asset.users",summary="The files that use an asset or a part it owns, as the Delete modal of the content browser lists them.",params=type_info_of(Api_Asset_Users_Params),result=type_info_of(Api_Asset_Users_Result),flags={.Read_Only},run=api_asset_users}) - api_register(s,st,{name="asset.delete",summary="Moves an asset and the parts it owns to the trash of the system at once and without undo. Fails for files of plugins, the engine or the editor, open tabs and outside Edit.",params=type_info_of(Api_Asset_Delete_Params),result=type_info_of(Api_Asset_Delete_Result),flags={.Destructive},run=api_asset_delete}) - api_register(s,st,{name="asset.move",summary="Moves, renames or copies an asset with the parts it owns, or a folder with all it holds, at once and without undo; a move keeps ids and references, an asset its extension. A move fails for files of plugins, the engine or the editor, open tabs, files being edited, a folder into itself and outside Edit.",params=type_info_of(Api_Asset_Move_Params),result=type_info_of(Api_Asset_Moved_Result),run=api_asset_move}) + api_register(s,st,{name="asset.delete",summary="Moves an asset and the parts it owns to the trash of the system at once and without undo; the root of a package of the library also takes the package out of the project, and the store keeps it. Fails for another asset a package lists, files of plugins, the engine or the editor, open tabs and outside Edit.",params=type_info_of(Api_Asset_Delete_Params),result=type_info_of(Api_Asset_Delete_Result),flags={.Destructive},run=api_asset_delete}) + api_register(s,st,{name="asset.move",summary="Moves, renames or copies an asset with the parts it owns, or a folder with all it holds, at once and without undo; a move keeps ids and references, an asset its extension, and moving the root of a package of the library moves where the project lists the package. A move fails for another asset a package lists, files of plugins, the engine or the editor, open tabs, files being edited, a folder into itself and outside Edit.",params=type_info_of(Api_Asset_Move_Params),result=type_info_of(Api_Asset_Moved_Result),run=api_asset_move}) + api_register(s,st,{name="packages.list",summary="The packages of the library of this machine, what every import ingests into its store, with their assets: name, kind, bytes, whether built, and whether the project pins the package.",params=type_info_of(Api_Packages_List_Params),result=type_info_of(Api_Packages_List_Result),flags={.Read_Only},run=api_packages_list}) + api_register(s,st,{name="packages.add",summary="Pins a package of the library, or some of its assets, in the project, listed in a folder of the assets, as carrying it from the library in the content browser does; its documents are written there. Answers as asset.import does when it ends.",params=type_info_of(Api_Packages_Add_Params),result=type_info_of(Import_Status_Result),defaults=&add_default,run=api_packages_add}) + collab_library_api_register(s,st) } // ---------------------------------------------------------------- component.list diff --git a/editor/api_diagnostics.odin b/editor/api_diagnostics.odin index e4e2d0a8e..c0182080b 100644 --- a/editor/api_diagnostics.odin +++ b/editor/api_diagnostics.odin @@ -129,6 +129,7 @@ api_diagnostics :: proc(s:^Session,call:^api.Call) { append(&errors,entry) } api_duplicates_list(&app.assets.registry,&errors) + for problem in host.assets_package_problems(app.assets) {append(&errors,Api_Engine_Diagnostics_Result_Asset_Errors{kind="Package",problem="Not_Listed",message=problem})} } r.asset_errors=errors[:] r.build.phase="Idle" diff --git a/editor/api_server.odin b/editor/api_server.odin index 28c8bbb39..a3c4cf6bb 100644 --- a/editor/api_server.odin +++ b/editor/api_server.odin @@ -312,6 +312,13 @@ api_idle :: proc(s:^Session) -> bool { restart(s,{"--built"}) return true } + // Nor the end that starts the editor of another project, which a join into its folder asks + // for: whoever drives this editor asked for it, so its own turn working is no question. + if .Quit_Check in s.requests && s.projects!=nil && s.projects.next { + s.requests-={.Quit_Check} + documents_quit(s,agents_asked=true) + if s.quit {return true} + } api_jobs(s) api.server_wait(&st.server,API_IDLE_WAIT) return true diff --git a/editor/asset_delete.odin b/editor/asset_delete.odin index 6cb7b2816..97797ec4e 100644 --- a/editor/asset_delete.odin +++ b/editor/asset_delete.odin @@ -5,6 +5,7 @@ import "core:path/filepath" import "core:slice" import "core:strings" import "mirai:asset" +import "mirai:cook/import/sources" import "mirai:core" import "mirai:editor/api" import "mirai:host" @@ -16,7 +17,11 @@ import "mirai:ui" // trash of the system, from which the user brings it back, at once and not as // an edit, as an import writes. The parts of an import go with the asset that // owns them; what uses it, saved or in an open document, is named before and -// keeps the reference. +// keeps the reference. The root of a package (library.odin) takes the package +// out of the packages.mirai of the project, which version control brings +// back: what it listed leaves the project, and its blobs stay in the store. A +// source of it inside the project goes to the trash with it; one outside +// stays. Another asset a package lists is refused. Api_Asset_Delete_Params :: struct { asset:[API_PATH_BYTES]u8 `mirai:"text,label=The asset to delete: its path under the asset folder or its id"`, @@ -46,19 +51,19 @@ asset_delete_files :: proc(r:^asset.Registry,id:base.Asset_Id) -> []u32 { return files[:] } -// As asset_delete_files, with what goes when the asset is the root of an -// import: what that import keeps in the imported folder, which nothing builds -// once its source and its sidecar are gone. +// As asset_delete_files, with the package the asset is the root of, which +// deleting it unpins, and what that package lists. @(private) -asset_delete_all :: proc(s:^Session,r:^asset.Registry,id:base.Asset_Id) -> (files:[]u32,sidecar:string) { - listed:=make([dynamic]u32,context.temp_allocator) - append(&listed,..asset_delete_files(r,id)) - cached:[]u32 - sidecar,cached=import_root(s,r,id) - for at in cached { - if !slice.contains(listed[:],at) {append(&listed,at)} +asset_delete_all :: proc(s:^Session,r:^asset.Registry,id:base.Asset_Id) -> (files:[]u32,pin:Library_Pin,unpins:bool) { + out:=make([dynamic]u32,context.temp_allocator) + append(&out,..asset_delete_files(r,id)) + file,found:=asset.registry_find(r,id) + if !found {return out[:],{},false} + if pin,unpins=library_pin_of(s,r,file);!unpins || asset.package_root(&pin.p)!=id {return out[:],{},false} + for &a in pin.p.assets { + if at,known:=r.by_id[a.id];known && r.files[at].listed && !slice.contains(out[:],at) {append(&out,at)} } - return listed[:],sidecar + return out[:],pin,true } // What uses what deleting moves and is not moved with it, by path: the files @@ -78,8 +83,14 @@ asset_delete_users :: proc(s:^Session,r:^asset.Registry,files:[]u32) -> []string @(private) asset_delete_refusal :: proc(s:^Session,r:^asset.Registry,file:^asset.Registry_File) -> string { if refusal:=asset_owned_refusal(r,file);refusal!="" {return refusal} - if file.imported!=0 { - if sidecar,_:=import_root(s,r,file.id);sidecar=="" {return "It is built by the import of its source, which makes it again: delete or move the asset that source gives (its prefab), change the source, or duplicate it to have a copy of its own"} + if file.listed { + pin,found:=library_pin_of(s,r,file) + switch { + case !found: + return "It is listed from a package the project pins and the store at hand does not have: refresh the project, or duplicate it to have a copy of its own" + case asset.package_root(&pin.p)!=file.id: + return fmt.tprintf("It is listed from the package %s: delete the asset that package gives, %s, which takes the package out of the project, or duplicate it to have a copy of its own",pin.p.name,library_root_named(r,&pin.p)) + } } if watching(s) {return "The run of an agent is being watched; stop watching it first"} if busy:=import_busy(s);busy!="" {return busy} @@ -106,23 +117,34 @@ asset_delete :: proc(s:^Session,id:base.Asset_Id) -> (moved:[]string,message:str file,found:=asset.registry_find(r,id) if !found {return nil,"The asset is not in the registry"} if refusal:=asset_delete_refusal(s,r,file);refusal!="" {return nil,refusal} - files,sidecar:=asset_delete_all(s,r,id) + files,pin,unpins:=asset_delete_all(s,r,id) paths:=make([]string,len(files),context.temp_allocator) - for at,i in files {paths[i]=strings.clone(r.files[at].path,context.temp_allocator)} + listed:=make([]bool,len(files),context.temp_allocator) + for at,i in files {paths[i],listed[i]=strings.clone(r.files[at].path,context.temp_allocator),r.files[at].listed} // What the registry holds of them goes first, or it would keep answering for them. for path in paths {_,_=asset.registry_memory_drop(r,path)} done:=make([dynamic]string,context.temp_allocator) - for path in paths { + for path,i in paths { + // What a package lists is a blob of the store, which stays: it leaves with the package. + if listed[i] {continue} if !platform.trash(asset.registry_file_path(r,path,context.temp_allocator)) { message=fmt.tprintf("%s could not be moved to the trash",path) break } append(&done,path) } - // The import is given up with its root: its source stays, and importing it again is a new import. - if message=="" && sidecar!="" { - _=platform.trash(sidecar) - _=platform.trash(strings.trim_suffix(sidecar,asset.SIDECAR_EXTENSION)) + // The package is given up with its root: importing its source again is a new package. + if message=="" && unpins { + if _,error:=sources.project_unpin(pin.root.root,pin.entry.id);error!=.None { + message=fmt.tprintf("%s/%s could not be written (%v): check that it is not read-only, then delete again",pin.root.root,asset.PACKAGES_FILE,error) + } else { + for path,i in paths { + if listed[i] {append(&done,path)} + } + origin:=pin.p.origin.path + if origin!="" && asset_path_inside(origin,pin.root.root) {_=platform.trash(origin)} + library_relist(s,pin.root) + } } if len(done)>0 { host.assets_refresh(s.app.assets,done[:]) @@ -153,7 +175,7 @@ asset_delete_modal :: proc(s:^Session) { closed:=false defer if closed {g.modal,g.delete_asset=.None,{}} if ui.modal(ctx,shell_id(g,"delete_asset"),{width=420,padding=16,gap=14,close=&closed}) { - files,_:=asset_delete_all(s,r,file.id) + files,_,_:=asset_delete_all(s,r,file.id) users:=asset_delete_users(s,r,files) refusal:=asset_delete_refusal(s,r,file) ui.label(ctx,"Delete asset",ui.metric(ctx,"font_heading"),ui.tone(ctx,"text"),font=ui.FONT_SEMIBOLD) diff --git a/editor/asset_move.odin b/editor/asset_move.odin index 07c5ec323..4f93cae56 100644 --- a/editor/asset_move.odin +++ b/editor/asset_move.odin @@ -7,7 +7,7 @@ import "core:os" import "core:path/filepath" import "core:strings" import "mirai:asset" -import importer "mirai:cook/import" +import "mirai:cook/import/sources" import "mirai:editor/api" import "mirai:editor/history" import "mirai:host" @@ -21,9 +21,12 @@ import "mirai:std/serial" // asset names its id, which moving keeps: references, Recent, Favorites and // open documents follow it. An asset moves with the parts it owns, which keep // where they are beside it; an import keeps its source, which Import_Info -// names from the project. What is being edited does not move until its step -// ends. A duplicate is a new asset with ids of its own, and no -// output of an import. +// names from its package. Moving the root of a package of the library +// (library.odin) moves where the project lists the package, and what it +// lists goes there; a folder takes the packages listed in it. Another asset +// a package lists does not move on its own. What is being edited does not +// move until its step ends. A duplicate is a new asset with ids of its own, +// and no output of an import. Api_Asset_Move_Params :: struct { asset:[API_PATH_BYTES]u8 `mirai:"text,label=The asset or folder: its path under the asset folder or the id of an asset"`, @@ -84,7 +87,6 @@ asset_held_refusal :: proc(r:^asset.Registry,paths:[]string,folder:="") -> strin // Why an asset cannot be renamed or moved now; "" when it can. @(private) asset_move_refusal :: proc(s:^Session,r:^asset.Registry,file:^asset.Registry_File) -> string { - if file.imported!=0 {return "It is a part of what its source gives, listed in the folder of its source: move that folder, or duplicate it to have a copy that moves"} if refusal:=asset_create_refusal(s);refusal!="" {return refusal} if refusal:=asset_delete_refusal(s,r,file);refusal!="" {return refusal} files:=asset_delete_files(r,file.id) @@ -102,25 +104,20 @@ asset_move :: proc(s:^Session,id:base.Asset_Id,folder,name:string) -> (path,mess if r==nil {return "","The editor has no asset folder it can change"} file,found:=asset.registry_find(r,id) if !found {return "","The asset is not in the registry: asset.list lists them"} - // What a source gives is listed where the source is: its source moves, with its sidecar, and nothing is imported. - if sidecar,_:=import_root(s,r,id);file.imported!=0 && sidecar!="" { - into,refused:=asset_folder_clean(r,folder) - if refused!="" {return "",fmt.tprintf("folder: %s",refused)} - if busy:=import_busy(s);busy!="" {return "",busy} - source:=strings.trim_suffix(sidecar,importer.SIDECAR_EXTENSION) - target:=fmt.tprintf("%s/%s",asset.registry_file_path(r,into,context.temp_allocator),filepath.base(source)) - if dir:=filepath.dir(target);!os.is_dir(dir) {_=os.make_directory_all(dir)} - if os.exists(target) || os.rename(source,target)!=nil || os.rename(sidecar,fmt.tprintf("%s%s",target,importer.SIDECAR_EXTENSION))!=nil { - return "",fmt.tprintf("%s could not be moved to %s: a file is there, or another program has it open",source,target) - } - host.assets_scan(s.app.assets) - if moved,listed:=asset.registry_find(r,id);listed {path=strings.clone(moved.path,context.temp_allocator)} - return - } + if file.listed {return asset_move_listed(s,r,file,folder,name)} if refusal:=asset_move_refusal(s,r,file);refusal!="" {return "",refusal} + pin,pinned:=library_pin_of(s,r,file) + pinned&&=asset.package_root(&pin.p)==id into,refused:=asset_folder_clean(r,folder) if refused!="" {return "",fmt.tprintf("folder: %s",refused)} into=strings.clone(into,context.temp_allocator) + // The listing of its package goes where it goes, in the root it is pinned in. + listing:string + if pinned { + root,rest,rooted:=import_root_of(s,r,into) + if !rooted || root.root!=pin.root.root {return "",fmt.tprintf("folder: the package %s is pinned in %s: give a folder of its assets",pin.p.name,pin.root.root)} + listing=strings.clone(rest,context.temp_allocator) + } extension:=asset_path_extension(file.path) named:=strings.trim_space(name) if named=="" {named=asset_name(file.path)} @@ -129,8 +126,8 @@ asset_move :: proc(s:^Session,id:base.Asset_Id,folder,name:string) -> (path,mess source:=content_folder_of(file.path) moves:=make([dynamic]Asset_Move,context.temp_allocator) for at,i in asset_delete_files(r,id) { - // A part of the imported folder stays at the path its sidecar lists. - if i>0 && r.files[at].imported!=0 {continue} + // A part a package lists goes with the listing of its package. + if i>0 && r.files[at].listed {continue} from:=strings.clone(r.files[at].path,context.temp_allocator) to:=asset_create_join(into,named,extension) if i>0 {to=asset_create_join(into,from[len(source)+1:] if source!="" && asset_path_inside(from,source) else asset_path_base(from),"")} @@ -142,7 +139,7 @@ asset_move :: proc(s:^Session,id:base.Asset_Id,folder,name:string) -> (path,mess } append(&moves,Asset_Move{from,to}) } - if len(moves)==0 {return} + if len(moves)==0 && !(pinned && listing!=pin.entry.folder) {return} done:=0 for move in moves { from:=asset.registry_file_path(r,move.from,context.temp_allocator) @@ -165,6 +162,11 @@ asset_move :: proc(s:^Session,id:base.Asset_Id,folder,name:string) -> (path,mess asset_history_moved(s,asset.registry_file_path(r,move.from,context.temp_allocator),asset.registry_file_path(r,move.to,context.temp_allocator),false) asset_start_scene_moved(s,move.from,move.to,false) } + if pinned && listing!=pin.entry.folder { + entry:=pin.entry + entry.folder=listing + if problem:=library_repin(s,pin.root,entry);problem!="" {message=problem} + } host.assets_scan(s.app.assets) if c:=s.content;c!=nil { c.reveal=id @@ -173,6 +175,64 @@ asset_move :: proc(s:^Session,id:base.Asset_Id,folder,name:string) -> (path,mess return } +// Moves what a package lists: its root moves where the project lists the +// package, keeping the name its source gives it; any other is refused. +@(private) +asset_move_listed :: proc(s:^Session,r:^asset.Registry,file:^asset.Registry_File,folder,name:string) -> (path,message:string) { + id:=file.id + pin,found:=library_pin_of(s,r,file) + if !found {return "","It is listed from a package the project pins and the store at hand does not have: refresh the project, or duplicate it to have a copy that moves"} + if asset.package_root(&pin.p)!=id {return "",fmt.tprintf("It is listed from the package %s: move the asset that package gives, %s, which moves where the project lists the package, or duplicate it to have a copy that moves",pin.p.name,library_root_named(r,&pin.p))} + if refusal:=asset_create_refusal(s);refusal!="" {return "",refusal} + into,refused:=asset_folder_clean(r,folder) + if refused!="" {return "",fmt.tprintf("folder: %s",refused)} + extension:=asset_path_extension(file.path) + named:=strings.trim_space(name) + if extension!="" && strings.has_suffix(named,extension) {named=named[:len(named)-len(extension)]} + if named!="" && named!=asset_name(file.path) {return "",fmt.tprintf("name: %s keeps the name its source gives it: move it without a name, or duplicate it to name a copy",file.path)} + root,rest,rooted:=import_root_of(s,r,into) + if !rooted || root.root!=pin.root.root {return "",fmt.tprintf("folder: the package %s is pinned in %s: give a folder of its assets",pin.p.name,pin.root.root)} + entry:=pin.entry + entry.folder=rest + if entry.folder!=pin.entry.folder { + if problem:=library_repin(s,pin.root,entry);problem!="" {return "",problem} + } + moved,there:=asset.registry_find(r,id) + if !there {return "",fmt.tprintf("The package %s was listed in %s and its asset is not there: refresh the project",pin.p.name,into)} + path=strings.clone(moved.path,context.temp_allocator) + if c:=s.content;c!=nil { + c.reveal=id + content_place_set(c,into) + } + return +} + +// The entries of the packages root pins that are listed in the folder from +// of its assets or under it, each with the folder it is listed in once from +// is to; in the temp allocator. +@(private) +asset_folder_pins :: proc(root:Library_Root,from,to:string) -> []asset.Project_Package { + list,_,error:=asset.packages_load(root.root,context.temp_allocator) + if error!=.None {return nil} + out:=make([dynamic]asset.Project_Package,context.temp_allocator) + for entry in list.packages { + if entry.folder!=from && !asset_path_inside(entry.folder,from) {continue} + moved:=entry + moved.folder=strings.concatenate({to,entry.folder[len(from):]},context.temp_allocator) + append(&out,moved) + } + return out[:] +} + +// Whether the folder of the registry holds what a package lists. +@(private) +asset_folder_listed :: proc(r:^asset.Registry,folder:string) -> bool { + for &file in r.files { + if file.listed && asset_path_inside(file.path,folder) {return true} + } + return false +} + // Moves the folder from, with all it holds, to folder under name ("" keeps // the one it has). Returns where it is, or what failed as how to correct it. asset_folder_move :: proc(s:^Session,from,folder,name:string) -> (path,message:string) { @@ -191,15 +251,26 @@ asset_folder_move :: proc(s:^Session,from,folder,name:string) -> (path,message:s for &file in r.files { if file.state==.Asset && asset_path_inside(file.path,source) && scene_tab(s,file.id)>=0 {return "",fmt.tprintf("%s is open in a tab: close the tab first",file.path)} } + // The packages listed in it go with it, in the root they are pinned in. + root,from_rest,_:=import_root_of(s,r,source) + to_root,to_rest,rooted:=import_root_of(s,r,path) + pins:=asset_folder_pins(root,from_rest,to_rest) + if len(pins)>0 && (!rooted || to_root.root!=root.root) {return "",fmt.tprintf("%s lists packages pinned in %s: give a folder of its assets",source,root.root)} full:=asset.registry_file_path(r,source,context.temp_allocator) target:=asset.registry_file_path(r,path,context.temp_allocator) - // The sources of the folder go with it, with their sidecars: what they give is listed where they are, and nothing is imported. if os.is_dir(full) { if dir:=filepath.dir(target);!os.is_dir(dir) {_=os.make_directory_all(dir)} if os.rename(full,target)!=nil {return "",fmt.tprintf("%s could not be moved to %s; another program may have a file of it open",source,path)} } asset_history_moved(s,full,target,true) asset_start_scene_moved(s,source,path,true) + for entry in pins { + if error:=sources.project_pin(root.root,&s.store,entry);error!=.None { + message=fmt.tprintf("%s/%s could not be written (%v): the folder moved and some of its packages are listed where it was; check that the file is not read-only and move them",root.root,asset.PACKAGES_FILE,error) + break + } + } + if len(pins)>0 {library_relist(s,root)} host.assets_scan(s.app.assets) if c:=s.content;c!=nil {content_place_set(c,path)} return @@ -322,7 +393,7 @@ asset_folder_subject :: proc(s:^Session,from:string) -> (r:^asset.Registry,folde switch { case refusal!="": return case folder=="": refusal="The assets folder itself stays: give one of its folders" - case !os.is_dir(asset.registry_file_path(r,folder,context.temp_allocator)): refusal=fmt.tprintf("%s is not a folder of the assets: asset.list lists what they hold",folder) + case !os.is_dir(asset.registry_file_path(r,folder,context.temp_allocator)) && !asset_folder_listed(r,folder): refusal=fmt.tprintf("%s is not a folder of the assets: asset.list lists what they hold",folder) } return } @@ -423,7 +494,7 @@ api_asset_subject :: proc(call:^api.Call,r:^asset.Registry,name:string) -> (id:b } if file,found:=asset.registry_find_path(r,name);found && file.state==.Asset {return file.id,"",true} clean,refusal:=asset_folder_clean(r,name) - if refusal=="" && clean!="" && os.is_dir(asset.registry_file_path(r,clean,context.temp_allocator)) {return {},clean,true} + if refusal=="" && clean!="" && (os.is_dir(asset.registry_file_path(r,clean,context.temp_allocator)) || asset_folder_listed(r,clean)) {return {},clean,true} api.call_failf(call,api.INVALID_PARAMS,"%q is no asset nor folder: give a path under the asset folder or the id of an asset as asset.list lists them",name) return } diff --git a/editor/assets/presence/source/pup.glb.import b/editor/assets/presence/source/pup.glb.import deleted file mode 100644 index 04cae8dbd..000000000 --- a/editor/assets/presence/source/pup.glb.import +++ /dev/null @@ -1,12 +0,0 @@ -{ - "textures": "Blocks", - "rdo_lambda": 0, - "image": "Data", - "alpha_coverage": 0, - "lods": "Auto", - "parts": "All", - "front": "Auto", - "rig": "Source", - "rig_crotch": 0, - "rig_neck": 0 -} diff --git a/editor/assets_parts.odin b/editor/assets_parts.odin index b3ccce939..df5af1a5b 100644 --- a/editor/assets_parts.odin +++ b/editor/assets_parts.odin @@ -30,7 +30,7 @@ asset_detach :: proc(s:^Session,id:base.Asset_Id) -> (owner:base.Asset_Id,messag if !found {return {},fmt.tprintf("No asset has the id %s; asset.list lists them",base.asset_id_format(id,buffer[:]))} path:=file.path if file.owner=={} {return {},fmt.tprintf("%s is a part of no asset: it already is one of its own",path)} - if file.imported!=0 {return {},fmt.tprintf("%s is built by the import of its source, whose sidecar says it is a part: import the source with parts none, or duplicate the asset to have one of its own",path)} + if file.listed {return {},fmt.tprintf("%s is listed from a package whose manifest says it is a part: import its source again with parts none, or duplicate the asset to have one of its own",path)} owner=file.owner bytes,error:=asset.registry_read(r,path,context.temp_allocator) if error==.None {error=asset.header_owner_set(bytes,{})} diff --git a/editor/build.odin b/editor/build.odin index 028a41563..04643e125 100644 --- a/editor/build.odin +++ b/editor/build.odin @@ -90,6 +90,7 @@ Build_Run :: struct { thread:^thread.Thread, project_root,engine_root,folder,test_package:string, runs:string, // the folder of the binaries of the engine this editor runs + store_root:string, // of the store of the session names:[]string, action:build.Action, target:build.Target, @@ -118,6 +119,7 @@ build_run_start :: proc(s:^Session,action:build.Action,target:build.Target,folde r:=new(Build_Run,a) r.project_root,r.engine_root,r.runs=strings.clone(s.project_root,a),strings.clone(s.engine_root,a),strings.clone(engine_folder_here(s),a) r.folder,r.test_package=strings.clone(folder,a),strings.clone(test_package,a) + r.store_root=strings.clone(s.store.root,a) r.names=make([]string,len(names),a) for name,i in names {r.names[i]=strings.clone(name,a)} r.action,r.target,r.seconds,r.started=action,target,seconds,time.tick_now() @@ -161,7 +163,7 @@ build_run :: proc(r:^Build_Run) { if r.shaders.result.status==.Failed {return} } } - r.job=build.job_start({project=&p,target=r.target,action=r.action,output=r.folder,test_package=r.test_package,test={names=r.names,quiet=true},dev=r.target==.Editor && build.ENGINE_MODE_HERE==.Dev,engine=build.ENGINE_MODE_HERE,runs=r.runs}) + r.job=build.job_start({project=&p,target=r.target,action=r.action,output=r.folder,test_package=r.test_package,test={names=r.names,quiet=true},dev=r.target==.Editor && build.ENGINE_MODE_HERE==.Dev,engine=build.ENGINE_MODE_HERE,runs=r.runs,store_root=r.store_root}) for { ended:=build.job_poll(r.job) if r.shaders!=nil {ended=build.shaders_poll(r.shaders) && ended} diff --git a/editor/collab.odin b/editor/collab.odin index 1891dc2e3..883d96b8e 100644 --- a/editor/collab.odin +++ b/editor/collab.odin @@ -8,15 +8,19 @@ import "core:slice" import "core:strings" import "core:time" import "mirai:asset" +import "mirai:cook/share" import "mirai:core" import "mirai:editor/history" import scene "mirai:engine/scene" import "mirai:host" +import "mirai:platform" import wire "mirai:net" import gpu "mirai:render" import "mirai:std/base" +import "mirai:std/jobs" import mm "mirai:std/math" import "mirai:std/serial" +import "mirai:store" import "mirai:ui" // Collaboration: several editors share the scene of one of them. The host @@ -35,10 +39,10 @@ import "mirai:ui" // draw as avatars and outlines: selecting holds nothing. // // A guest joins with the code of the invitation of the host (net/invite.odin): -// its address on its network and on its router, and the token of the session. +// its address on its network and on its router, and the key of the session. COLLAB_PORT :: 7950 -COLLAB_VERSION :: 7 // of the messages: an editor of another one has another version of the project +COLLAB_VERSION :: 8 // of the messages: an editor of another one has another version of the project COLLAB_PEERS :: 8 // the host and seven guests COLLAB_NAME_BYTES :: 32 COLLAB_SELECTION :: 16 @@ -46,16 +50,19 @@ COLLAB_PRESENCE_INTERVAL :: 1.0/12 COLLAB_EDIT_INTERVAL :: 1.0/30 COLLAB_PRESENCE_SEEN :: 3.0 // seconds COLLAB_MESSAGE_BYTES :: 1<<30 // what one message may hold: a peer that sends more is cut off -COLLAB_CHANNELS :: []wire.Channel_Desc{{kind=.Reliable,queue_bytes=256<<10,window=512},{kind=.Unreliable}} +// Of the hosts of a session, whatever the project: a guest of another one still gets its files, +// and its Hello says whether it can enter the scene. +COLLAB_WIRE :: u32(COLLAB_VERSION)<<8 | u32(share.VERSION) COLLAB_HELLO_TIMEOUT :: 10.0 -COLLAB_WORKER_INTERVAL :: 2*time.Millisecond // between updates of the network half +COLLAB_LEAVE_WAIT :: 8*time.Millisecond // for the worker, which host_poll wakes, to send the goodbye COLLAB_PLAYING :: u8(1) // presence: inside a Play or a Simulate of its own COLLAB_SERVING :: u8(2) // presence: serves the game of a plugin, whose offer follows COLLAB_JOINED :: u8(4) // presence: a guest that plays the game the host serves -COLLAB_OFFER_BYTES :: 32 // what the presence of the host carries of the game it serves (Play_Provider.offer) +COLLAB_OFFER_BYTES :: 64 // what the presence of the host carries of the game it serves (Play_Provider.offer) Collab_Role :: enum u8 {None,Host,Guest} -Collab_Phase :: enum u8 {Idle,Connecting,Syncing,Joined,Ended} +// Updating: a guest brings its folder to the host's snapshot before it enters the scene. +Collab_Phase :: enum u8 {Idle,Connecting,Updating,Syncing,Joined,Ended} Collab_Error :: enum u8 { None, @@ -65,6 +72,9 @@ Collab_Error :: enum u8 { Transport, // the port could not be opened No_Assets, No_Random, // the system gives no secure random bytes, which the session is sealed with + No_Store, // the editor has no store, where the project travels through + No_Project, // the editor has no project to share or to join into + Project, // the project could not be offered: Collab.problem says why } collab_error_text :: proc(error:Collab_Error) -> string { @@ -72,10 +82,13 @@ collab_error_text :: proc(error:Collab_Error) -> string { case .None: return "" case .Busy: return "a session is open; leave it first" case .Playing: return "a game is playing; stop it first" - case .Invite: return "not the code of an invitation: like R2M0-253B-DK5G-0W87-M0MA-3CP3-TKJZ-C1RR-EJT; copy it again" + case .Invite: return "not the code of an invitation of this Mirai: 87 characters by groups of four; copy it again" case .Transport: return "the port could not be opened" case .No_Assets: return "the project has no assets folder" case .No_Random: return "this system gives no secure random bytes to seal the session with" + case .No_Store: return "the editor has no store, which the project travels through: start it with --store " + case .No_Project: return "the editor has no project: open one, or join into a folder" + case .Project: return "the project could not be offered" } return "" } @@ -88,6 +101,7 @@ Collab_Message :: enum u8 { Ping, // a point of the scene an editor pinged (collab_ping.odin) File, // an asset file an editor changed (collab_files.odin) File_Unsent, // an asset file an editor changed that is too large to travel + Project, // the host offers a new snapshot of its project (collab_project.odin) } // Set: the whole row, for whoever has none. Fields: runs of bytes of a row, @@ -162,7 +176,7 @@ Collab_State :: struct { mapping:wire.Port_Mapping, mapping_on:bool, port:int, - invite:u64, // the token of the session + invite:wire.Invite_Key, // of the session: the host's, with its private key invitation:wire.Invitation, // a guest: of the host self:u8, peers:[COLLAB_PEERS]Collab_Peer, @@ -175,7 +189,7 @@ Collab_State :: struct { by_hash:map[u64]int, epoch:u64, // of the history of the document being shared presence_at,edit_at:f64, - code:[48]u8, // the host: its invitation, as others reach it + code:[wire.INVITE_TEXT_BYTES]u8, // the host: its invitation, as others reach it code_length:int, sent_edits,received_edits,sent_ops,received_ops:u64, guest_name:[COLLAB_NAME_BYTES]u8, @@ -191,6 +205,25 @@ Collab_State :: struct { applying_file:bool, // a file of another end is being put: its change is not this end's unsent:[COLLAB_UNSENT]Collab_Unsent, // the files that did not travel; the first unsent_count unsent_count:int, + virtual:bool, // on the network of a test, whose clock is now + now:f64, + node_jobs:^jobs.Job_System, // of the exchange of node + node:^share.Node, // the project and the library over the connections of the session (collab_project.odin) + join:^Collab_Join, // a guest: what brings its folder to the host's snapshot + // The host: its project offered, and what changed since (collab_host_changes). + offered:store.Hash, + offers:int, // new snapshots offered during the session + watch:platform.Watch, + watching:bool, + changed_at,offered_at:f64, + libraries:^Collab_Libraries, // of the other editors, as last read (collab_library.odin) + reading:^Collab_Libraries, // and as being read again + take:^Collab_Take, // a package of one of them being taken + thumbs:[COLLAB_THUMB_FETCHES]Collab_Thumb, // thumbnails asked of them + thumbs_absent:map[u64]struct{}, // keys a peer had no thumbnail of + // A guest: an update asked for, and those done. + update_due:bool, + updates:int, } @(storage) @@ -199,10 +232,11 @@ Collab :: struct { // The fields of the Share popup. name:[COLLAB_NAME_BYTES]u8, name_length:int, - join:[64]u8, + join:[2*wire.INVITE_TEXT_BYTES]u8, join_length:int, joining:bool, error:Collab_Error, + problem:[256]u8, // of the error Project client:u64, // who this editor is in every session: one that joins again takes its place back pings:[COLLAB_PEERS]Collab_Ping, // the last of each editor, by its place in the session; this one alone is 0 } @@ -228,6 +262,10 @@ collab_state :: proc(s:^Session) -> ^Collab_State { return c==nil?nil:c.state } +// The clock of the session, in seconds. +@(private) +collab_now :: proc(st:^Collab_State) -> f64 {return st.virtual?st.now:wire.host_time(&st.host)} + collab_active :: proc(s:^Session) -> bool { st:=collab_state(s) return st!=nil && st.role!=.None && st.phase!=.Ended @@ -255,6 +293,7 @@ collab_summary :: proc(s:^Session) -> string { switch st.phase { case .Idle: return "guest" case .Connecting: return "guest connecting" + case .Updating: return "guest updating" case .Syncing: return "guest syncing" case .Joined: return "guest joined" case .Ended: return "ended" @@ -315,10 +354,13 @@ collab_name_set :: proc(c:^Collab,name:string) { } @(private) -collab_start :: proc(s:^Session,role:Collab_Role,port:int,name:string,invitation:wire.Invitation) -> (st:^Collab_State,error:Collab_Error) { +// transport: a network of a test, on its clock (Collab_State.now); none: UDP on port. +collab_start :: proc(s:^Session,role:Collab_Role,port:int,name:string,invitation:wire.Invitation,assets:=true,transport:=wire.Transport{}) -> (st:^Collab_State,error:Collab_Error) { c:=collab_of(s) if c.state!=nil && c.state.role!=.None && c.state.phase!=.Ended {return nil,.Busy} - if s.app.assets==nil {return nil,.No_Assets} + if assets && !s.has_project {return nil,.No_Project} + if assets && s.app.assets==nil {return nil,.No_Assets} + if s.store.root=="" {return nil,.No_Store} if s.mode!=.Edit {return nil,.Playing} if c.state!=nil {collab_destroy(s)} collab_name_set(c,name) @@ -326,37 +368,50 @@ collab_start :: proc(s:^Session,role:Collab_Role,port:int,name:string,invitation st.role=role st.layout=collab_layout_hash(s.info) // The addresses of an invitation are IPv4. + st.virtual=transport.send!=nil bind:=role==.Guest?wire.address_ip4(0,0,0,0,0):wire.address_any6(port) - if !wire.udp_open(&st.udp,bind) { + if !st.virtual && !wire.udp_open(&st.udp,bind) { free(st) return nil,.Transport } + channels:=collab_channels() desc:=wire.Host_Desc{ - mode=role==.Host?.Server:.Client,transport=wire.udp_transport(&st.udp),protocol_id=0x4d495241,version=st.layout,invite=invitation.token, - max_connections=COLLAB_PEERS-1,channels=COLLAB_CHANNELS, + mode=role==.Host?.Server:.Client,transport=st.virtual?transport:wire.udp_transport(&st.udp),protocol_id=0x4d495241,version=COLLAB_WIRE,invite=invitation.key, + max_connections=COLLAB_PEERS-1,channels=channels[:], } // Its network half runs on a thread of its own: an editor busy for a while // (generating the world of a game) keeps answering, and the session lives. - if wire.host_init(&st.host,desc)!=.None || !wire.worker_start(&st.host,COLLAB_WORKER_INTERVAL) { + if wire.host_init(&st.host,desc)!=.None || !st.virtual && !wire.worker_start(&st.host) { wire.host_destroy(&st.host) - wire.udp_close(&st.udp) + if !st.virtual {wire.udp_close(&st.udp)} free(st) return nil,.Transport } collab_tables_build(s,st) st.epoch=s.edits.epoch st.port=int(st.udp.address.port) - st.invite=invitation.token + st.invite=invitation.key c.state=st + if !collab_node_start(s,st) { + collab_destroy(s) + return nil,.Transport + } return st,.None } // Shares the scene: guests connect to this editor. -collab_host :: proc(s:^Session,port:int,name:string) -> Collab_Error { - invite,made:=wire.invite_token() +collab_host :: proc(s:^Session,port:int,name:string,transport:=wire.Transport{}) -> Collab_Error { + invite,made:=wire.invite_key() if !made {return .No_Random} - st,error:=collab_start(s,.Host,port>0?port:COLLAB_PORT,name,{token=invite}) + st,error:=collab_start(s,.Host,port>0?port:COLLAB_PORT,name,{key=invite},transport=transport) if error!=.None {return error} + if problem:=collab_project_offer(s,st);problem!="" { + collab_destroy(s) + c:=collab_of(s) + base.text_set(c.problem[:],problem) + return .Project + } + collab_watch_start(s,st) st.phase=.Joined st.self=0 c:=collab_of(s) @@ -364,25 +419,32 @@ collab_host :: proc(s:^Session,port:int,name:string) -> Collab_Error { st.peers[0].name_length=copy(st.peers[0].name[:],string(c.name[:c.name_length])) st.peers[0].orbit=s.viewport.orbit collab_journal_reset(s,st) - st.mapping_on=wire.mapping_start(&st.mapping,u16(st.port)) + st.mapping_on=!st.virtual && wire.mapping_start(&st.mapping,u16(st.port)) collab_codes_update(st) return .None } -// Enters the session of another editor with the code of its invitation: the scene is replaced by the one of the host. -collab_join :: proc(s:^Session,code,name:string) -> Collab_Error { - invitation,valid:=wire.invite_decode(code) - if !valid {return .Invite} - st,error:=collab_start(s,.Guest,0,name,invitation) +// Enters the session of another editor with the code of its invitation: +// first folder is brought to the host's snapshot (collab_project.odin), then +// the scene is replaced by the one of the host. folder: "" or the open +// project; another is opened once it has the host's files, and joins again. +// at: the host at that address, with the key of the invitation, in place of +// the addresses it carries. +collab_join :: proc(s:^Session,code,name:string,folder:="",at:=wire.Address{},transport:=wire.Transport{}) -> Collab_Error { + invitation,refused:=wire.invite_decode(code) + if refused!=.None {return .Invite} + target,reopen:=collab_join_folder(s,folder) + if target=="" {return .No_Project} + st,error:=collab_start(s,.Guest,0,name,invitation,assets=!reopen,transport=transport) if error!=.None {return error} - if wire.host_connect(&st.host,invitation.lan,invitation.public)!=.None { + connected:=at.kind!=.None?wire.host_connect(&st.host,at):wire.host_connect(&st.host,invitation.lan,invitation.public) + if connected!=.None || !collab_join_start(s,st,code,target,reopen) { collab_destroy(s) return .Transport } st.phase=.Connecting st.self=0xff st.invitation=invitation - st.peers[0].connected_at=wire.host_time(&st.host) return .None } @@ -396,7 +458,7 @@ collab_leave :: proc(s:^Session) { wire.host_disconnect(&st.host,0,.Closed) wire.host_poll(&st.host) // The worker sends them before the host goes. - time.sleep(4*COLLAB_WORKER_INTERVAL) + time.sleep(COLLAB_LEAVE_WAIT) } collab_destroy(s) } @@ -417,9 +479,10 @@ collab_destroy :: proc(s:^Session) { c:=collab_of(s) st:=c.state if st==nil {return} + collab_project_destroy(s,st) if st.mapping_on {wire.mapping_close(&st.mapping)} wire.host_destroy(&st.host) - wire.udp_close(&st.udp) + if !st.virtual {wire.udp_close(&st.udp)} for &peer in st.peers {collab_peer_free(&peer)} for path in st.files_out {delete(path)} delete(st.files_out) @@ -436,6 +499,8 @@ collab_destroy :: proc(s:^Session) { collab_end :: proc(st:^Collab_State,reason:string) { st.phase=.Ended st.reason=reason + // A join that runs ends at its next wait. + if st.node!=nil {share.node_stop(st.node)} } // The close of the editor. @@ -536,12 +601,15 @@ collab_peer_name :: proc(s:^Session,id:int) -> string { collab_update :: proc(s:^Session) { st:=collab_state(s) if st==nil || st.role==.None || st.phase==.Ended {return} - now:=wire.host_time(&st.host) + now:=collab_now(st) if st.role==.Host && st.mapping_on && int(now)%2==0 {collab_codes_update(st)} for command in wire.host_poll(&st.host) { + if share.node_command(st.node,&st.host,command) {continue} collab_command(s,st,command,now) if st.phase==.Ended {return} } + collab_project_update(s,st,now) + if st.phase==.Ended {return} // The scene of the document was replaced: the host shares the new one. if st.epoch!=s.edits.epoch { st.epoch=s.edits.epoch @@ -565,6 +633,8 @@ collab_update :: proc(s:^Session) { collab_presence_send(s,st) collab_agents_send(s,st,now) } + // From the first update: the clock of a test's network is the test's from then on. + if st.role==.Guest && st.phase==.Connecting && st.peers[0].connected_at==0 {st.peers[0].connected_at=now} if st.role==.Guest && st.phase==.Connecting && now-st.peers[0].connected_at>COLLAB_HELLO_TIMEOUT { collab_end(st,"the host did not answer") return @@ -585,16 +655,10 @@ collab_command :: proc(s:^Session,st:^Collab_State,command:wire.Command,now:f64) switch command.kind { case .Connected: if st.role==.Guest { - st.phase=.Syncing + // The Hello goes once the folder has the host's snapshot. + st.phase=.Updating st.peers[0].used=true st.peers[0].connection=command.connection - c:=collab_of(s) - w:serial.Writer - collab_begin(&w,.Hello) - serial.write_scalar(&w,st.layout) - serial.write_string(&w,string(c.name[:c.name_length])) - serial.write_scalar(&w,collab_client(c)) - collab_queue(&st.peers[0],serial.writer_bytes(&w)) } else { slot:=-1 for &peer,i in st.peers { @@ -610,7 +674,7 @@ collab_command :: proc(s:^Session,st:^Collab_State,command:wire.Command,now:f64) if st.role==.Guest { reason:="the host left" #partial switch command.reason { - case .Denied_Version: reason="the host has another version of the project" + case .Denied_Version: reason="this editor was built from other code than the host's project, whose files are here now: build the editor (Build) and join again" case .Denied_Full: reason="the session is full" case .Timeout: reason="the host stopped answering" case .Reliable_Overflow: reason="the connection could not keep up with what the session sends" @@ -804,11 +868,26 @@ collab_receive :: proc(s:^Session,st:^Collab_State,from:int,data:[]u8,now:f64) { if st.phase==.Joined && (st.role==.Guest || from>=0 && st.peers[from].joined) {collab_ping_receive(s,st,from,data)} case .File: if st.phase==.Joined && (st.role==.Guest || from>=0 && st.peers[from].joined) {collab_file_receive(s,st,from,data,&r)} + case .Project: + if st.role==.Guest {collab_update_request(st)} case .File_Unsent: if st.phase==.Joined && (st.role==.Guest || from>=0 && st.peers[from].joined) {collab_unsent_receive(s,st,from,&r)} } } +// A guest whose folder has the host's snapshot asks for the scene. +@(private) +collab_hello :: proc(s:^Session,st:^Collab_State) { + st.phase=.Syncing + c:=collab_of(s) + w:serial.Writer + collab_begin(&w,.Hello) + serial.write_scalar(&w,st.layout) + serial.write_string(&w,string(c.name[:c.name_length])) + serial.write_scalar(&w,collab_client(c)) + collab_queue(&st.peers[0],serial.writer_bytes(&w)) +} + @(private) collab_roster_send :: proc(st:^Collab_State) { if st.role!=.Host {return} @@ -1326,7 +1405,7 @@ collab_in_game :: proc(s:^Session) -> bool { collab_draw :: proc(s:^Session) { st:=collab_state(s) if st==nil || st.role==.None || st.phase!=.Joined || !presence_in_scene(s) {return} - now:=wire.host_time(&st.host) + now:=collab_now(st) lines:=&s.viewport.lines w:=edited_world(s) _,low,high,loaded:=presence_mesh(s.app.assets) @@ -1376,7 +1455,7 @@ collab_highlights :: proc(s:^Session,w:core.World_Ref,levels:[]u8) { st:=collab_state(s) if st==nil || st.phase!=.Joined || w.data!=s.edit_world || s.mode!=.Edit {return} es:=core.world_entities(w) - now:=wire.host_time(&st.host) + now:=collab_now(st) for &peer,i in st.peers { if !collab_peer_live(st,i,now) {continue} for id in peer.selection[:peer.selection_count] { @@ -1394,7 +1473,7 @@ collab_highlights :: proc(s:^Session,w:core.World_Ref,levels:[]u8) { collab_highlights_key :: proc(s:^Session) -> u64 { st:=collab_state(s) if st==nil || st.phase!=.Joined {return 0} - now:=wire.host_time(&st.host) + now:=collab_now(st) key:=u64(1) for &peer,i in st.peers { if !collab_peer_live(st,i,now) {continue} diff --git a/editor/collab_agents.odin b/editor/collab_agents.odin index c25e137c2..559f2612d 100644 --- a/editor/collab_agents.odin +++ b/editor/collab_agents.odin @@ -177,7 +177,7 @@ collab_agent_at :: proc(s:^Session,point:[2]f32) -> (peer,agent:int,rect:ui.Rect if st==nil || st.phase!=.Joined || s.mode!=.Edit {return} _,low,high,loaded:=presence_mesh(s.app.assets) if !loaded {return} - now:=wire.host_time(&st.host) + now:=collab_now(st) for i in 0.. string { + if error==.Project { + c:=collab_of(s) + return fmt.tprintf("%s: %s",collab_error_text(error),base.text_get(c.problem[:])) + } + return collab_error_text(error) } @(private) api_collab_host :: proc(s:^Session,call:^api.Call) { params:=api.call_params(call,Api_Collab_Host_Params) if error:=collab_host(s,params.port,base.text_get(params.name[:]));error!=.None { - api.call_fail(call,api.SERVER_ERROR,collab_error_text(error)) + api.call_fail(call,api.SERVER_ERROR,collab_error_say(s,error)) return } api_collab_result(s,call) @@ -95,8 +169,8 @@ api_collab_host :: proc(s:^Session,call:^api.Call) { @(private) api_collab_join :: proc(s:^Session,call:^api.Call) { params:=api.call_params(call,Api_Collab_Join_Params) - if error:=collab_join(s,base.text_get(params.invite[:]),base.text_get(params.name[:]));error!=.None { - api.call_fail(call,error==.Invite?api.INVALID_PARAMS:api.SERVER_ERROR,collab_error_text(error)) + if error:=collab_join(s,base.text_get(params.invite[:]),base.text_get(params.name[:]),base.text_get(params.folder[:]));error!=.None { + api.call_fail(call,error==.Invite?api.INVALID_PARAMS:api.SERVER_ERROR,collab_error_say(s,error)) return } api_collab_result(s,call) @@ -170,8 +244,12 @@ share_code :: proc(s:^Session,id,name,code,missing:string) { notify(s,"Copied the code") } } - if ui.box(ctx,{width=ui.GROW,height=ui.fixed(24),padding={6,0,6,0},align_y=.Center,fill=ui.tone(ctx,"well"),stroke=ui.tone(ctx,"well_stroke"),radius=ui.metric(ctx,"radius_control"),clip_x=true}) { - ui.label(ctx,code!=""?code:missing,code!=""?10.5:ui.metric(ctx,"font_label"),code!=""?ui.tone(ctx,"text"):ui.tone(ctx,"text_muted"),0) + // A code is wider than the popup: six of its groups a line. + LINE :: 6*5 + lines:=code!=""?(len(code)+LINE-1)/LINE:1 + if ui.box(ctx,{width=ui.GROW,height=ui.fixed(f32(18*lines+6)),direction=.Column,padding={6,3,6,3},fill=ui.tone(ctx,"well"),stroke=ui.tone(ctx,"well_stroke"),radius=ui.metric(ctx,"radius_control"),clip_x=true}) { + if code=="" {ui.label(ctx,missing,ui.metric(ctx,"font_label"),ui.tone(ctx,"text_muted"),0)} + for at:=0;atCOLLAB_LINES {ui.text(ctx,fmt.tprintf("%d more kept",len(j.report.kept)-COLLAB_LINES),muted)} + for line in j.report.failed[:min(len(j.report.failed),COLLAB_LINES)] {ui.text(ctx,line,danger)} + for line in j.report.packages.missing[:min(len(j.report.packages.missing),COLLAB_LINES)] {ui.text(ctx,line,danger)} +} diff --git a/editor/collab_edit_test.odin b/editor/collab_edit_test.odin index 2da29fa90..b17ec7af6 100644 --- a/editor/collab_edit_test.odin +++ b/editor/collab_edit_test.odin @@ -1,5 +1,6 @@ package editor +import "base:intrinsics" import "core:fmt" import "core:mem" import "core:strings" @@ -88,19 +89,26 @@ a_scene_of_the_host_that_fails_halfway_leaves_the_world_as_it_was :: proc(t:^tes } good,error:=asset.scene_bytes(&host_scene,context.temp_allocator) if !testing.expect_value(t,error,asset.Error.None) {return} - // The transforms of the broken copy do not decompress. + // The transforms of the broken copy are a stream that does not decode: its directory says so + // of the bytes, stored as they are, of their chunk. The load meets it when it reads the + // components, as it met a stream a compressor wrote. broken:=make([]u8,len(good),context.temp_allocator) copy(broken,good) file:asset.Asset if !testing.expect_value(t,asset.asset_open(&file,broken),asset.Error.None) {return} view,_:=asset.scene_view(&file,&assets.types) - entry:asset.Chunk_Entry + chunk:=-1 for &component in view.components { - if component.name=="Transform" {entry=asset.blob_chunk(&file.blob,component.table.columns[1].chunk)} + if component.name=="Transform" {chunk=component.table.columns[1].chunk} } + entry:asset.Chunk_Entry + if chunk>=0 {entry=asset.blob_chunk(&file.blob,chunk)} asset.asset_close(&file) - if !testing.expect_value(t,entry.compression,asset.Compression.Stream) {return} + if !testing.expect(t,chunk>=0) {return} + if !testing.expect(t,entry.compression==.None && entry.size>=64) {return} mem.set(&broken[entry.offset],0xff,int(entry.size)) + entry.compression=.Stream + intrinsics.unaligned_store((^asset.Chunk_Entry)(&broken[asset.HEADER_SIZE+chunk*asset.CHUNK_ENTRY_SIZE]),entry) world:=cast(^Play_Test_World)f.data testing.expect(t,!collab_snapshot_load(&f.s,st,broken)) @@ -286,8 +294,9 @@ a_file_as_large_as_a_session_carries_crosses_as_one_message :: proc(t:^testing.T host,guest:=new(Collab_State),new(Collab_State) defer free(host);defer free(guest) host.role,guest.role=.Host,.Guest + channels:=collab_channels() for st,i in ([2]^Collab_State{host,guest}) { - desc:=wire.Host_Desc{mode=i==0?.Server:.Client,transport=wire.sim_endpoint_transport(&network,i),protocol_id=0x4d495241,seed=u64(7+i),channels=COLLAB_CHANNELS,send_rate=1<<30,packets_per_update=64} + desc:=wire.Host_Desc{mode=i==0?.Server:.Client,transport=wire.sim_endpoint_transport(&network,i),protocol_id=0x4d495241,seed=u64(7+i),channels=channels[:]} testing.expect_value(t,wire.host_init(&st.host,desc),wire.Error.None) } defer {wire.host_destroy(&host.host);wire.host_destroy(&guest.host)} @@ -323,3 +332,15 @@ a_file_as_large_as_a_session_carries_crosses_as_one_message :: proc(t:^testing.T } testing.expect(t,arrived,"the file arrives") } + +// The code collab.status returns is the whole code: it decodes to the invitation of the host. +@(test) +the_code_collab_status_returns_reads_as_the_invitation :: proc(t:^testing.T) { + key,made:=wire.invite_key() + if !testing.expect(t,made) {return} + invitation:=wire.Invitation{lan=wire.address_ip4(192,168,1,2,COLLAB_PORT),public=wire.address_ip4(203,0,113,9,COLLAB_PORT),key={public=key.public,secret=key.secret}} + r:Api_Collab_Status + testing.expect(t,base.text_set(r.invite[:],wire.invite_encode(invitation,context.temp_allocator))) + got,refused:=wire.invite_decode(base.text_get(r.invite[:])) + testing.expect(t,refused==.None && got==invitation) +} diff --git a/editor/collab_library.odin b/editor/collab_library.odin new file mode 100644 index 000000000..edb1c7088 --- /dev/null +++ b/editor/collab_library.odin @@ -0,0 +1,408 @@ +package editor + +import "base:runtime" +import "core:fmt" +import "core:mem" +import "core:os" +import "core:strings" +import "core:sync" +import "core:thread" +import "mirai:asset" +import "mirai:cook/share" +import "mirai:editor/api" +import "mirai:std/base" +import "mirai:std/jobs" +import "mirai:std/serial" +import "mirai:store" +import "mirai:ui" + +// The libraries of the other editors of a session (cook/share/library.odin): +// the packages each one's machine exposes over the session, beside this +// machine's library (library.odin). Listing them and taking one run on threads +// of their own that wait on the node; a package taken is in this machine's +// library from then on, and is then added as one of it is. + +COLLAB_LIBRARY_SECONDS :: 10.0 // a listing older than this is read again when it is shown + +@(private) +Collab_Library :: struct { + peer:int, // in Collab_State.peers + node:share.Peer, + list:[]share.Peer_Package, + error:string, +} + +@(private) +Collab_Libraries :: struct { + thread:^thread.Thread, + done:bool, + node:^share.Node, + arena:mem.Dynamic_Arena, + allocator:runtime.Allocator, + libraries:[COLLAB_PEERS]Collab_Library, + count:int, + at:f64, +} + +@(private) +Collab_Take :: struct { + thread:^thread.Thread, + done:bool, + node:^share.Node, + manifest:store.Hash, + id:base.Asset_Id, + name:[64]u8, + root:string, + folder:string, + error:string, + received:u64, // of the node when it began + arena:mem.Dynamic_Arena, + allocator:runtime.Allocator, +} + +// The peers of the node, each with the place in the session of its editor. +@(private) +collab_library_peers :: proc(st:^Collab_State,into:[]Collab_Library) -> int { + buffer:[share.PEERS]share.Peer + count:=0 + for p in share.node_peers(st.node,buffer[:]) { + for &peer,i in st.peers { + if i==int(st.self) || !peer.used || peer.connection!=p.connection {continue} + if count (p:share.Peer_Package,problem:string) { + collab_libraries_promote(st) + l:=st.libraries + if l==nil {return {},"the libraries of the peers are being read: call packages.peers until listing is false"} + wanted:=strings.trim_space(named) + id,is_id:=serial.text_parse_asset_id(wanted) + for &library in l.libraries[:l.count] { + if library.peer!=peer {continue} + found:=-1 + for &q,at in library.list { + if !(is_id && q.id==id || !is_id && q.name==wanted) {continue} + if found>=0 {return {},fmt.tprintf("name: more than one package of that library is named %q: give its id, as packages.peers lists them",wanted)} + found=at + } + if found<0 {return {},fmt.tprintf("name: the library of %s has no package %q: packages.peers lists it",collab_peer_name(s,peer),wanted)} + return library.list[found],"" + } + return {},fmt.tprintf("peer: %d is not an editor of the session whose library is listed: packages.peers lists them",peer) +} + +// Takes a package of a peer's library into this machine's, then adds it to +// folder as one of this machine's library is added. What failed, as how to fix it. +@(private) +collab_library_add :: proc(s:^Session,peer:int,named,folder:string) -> string { + st:=collab_state(s) + if st==nil || st.node==nil || st.phase!=.Joined {return "this editor is in no session: the libraries of others are those of the editors of a session (collab.host or collab.join)"} + if st.take!=nil && !sync.atomic_load(&st.take.done) {return "a package of a peer is being added: wait for it, as packages.peers says"} + if busy:=import_busy(s);busy!="" {return busy} + p,problem:=collab_library_find(s,st,peer,named) + if problem!="" {return problem} + if st.take!=nil {collab_take_free(st.take)} + t:=new(Collab_Take) + mem.dynamic_arena_init(&t.arena,runtime.heap_allocator(),runtime.heap_allocator()) + t.allocator=base.dynamic_arena_allocator(&t.arena) + t.node,t.manifest,t.id=st.node,p.manifest,p.id + base.text_set(t.name[:],p.name) + t.root=strings.clone(s.project_root,t.allocator) + t.folder=strings.clone(folder,t.allocator) + t.received=share.node_totals(st.node).received + st.take=t + t.thread=thread.create_and_start_with_poly_data(t,proc(t:^Collab_Take) { + defer runtime.default_temp_allocator_destroy(&runtime.global_default_temp_allocator_data) + js:jobs.Job_System + threaded:=jobs.system_init(&js,{threads=max(os.get_processor_core_count()-2,0),allocator=runtime.heap_allocator()}) + worker:=jobs.Worker_Ref{} + if threaded {worker={system=&js,worker=jobs.OWNER}} + t.error=strings.clone(share.package_take(t.node,t.manifest,t.root,import_library,worker),t.allocator) + if threaded {jobs.system_destroy(&js)} + sync.atomic_store(&t.done,true) + }) + if t.thread==nil { + sync.atomic_store(&t.done,true) + t.error="no thread could be started to fetch it" + } + return "" +} + +// The end of a take, on the editor's thread: added as a package of this machine. +@(private) +collab_take_update :: proc(s:^Session,st:^Collab_State) { + t:=st.take + if t==nil || t.thread==nil || !sync.atomic_load(&t.done) {return} + thread.join(t.thread) + thread.destroy(t.thread) + t.thread=nil + name:=base.text_get(t.name[:]) + if t.error!="" { + notify(s,fmt.tprintf("%s was not added: %s",name,t.error)) + return + } + buffer:[base.ASSET_ID_TEXT]u8 + if problem:=library_add(s,base.asset_id_format(t.id,buffer[:]),nil,t.folder);problem!="" { + t.error=strings.clone(problem,t.allocator) + notify(s,fmt.tprintf("%s is in this machine's library now but was not added: %s",name,problem)) + } + // Read again: it is here now. + if st.libraries!=nil {st.libraries.at=-COLLAB_LIBRARY_SECONDS} +} + +COLLAB_THUMB_FETCHES :: 4 // thumbnails asked of peers at once + +// A thumbnail a visible card lacks, asked of the peer whose root shows it. +@(private) +Collab_Thumb :: struct { + used:bool, + key:u64, + name:[16]u8, + fetch:share.Fetch, +} + +// The thumbnails the visible cards of peers' roots lack, asked a few at a +// time of those peers by their keys; one a peer has not is not asked again. +@(private) +collab_thumbs_update :: proc(s:^Session,st:^Collab_State) { + for &t in st.thumbs { + if !t.used {continue} + #partial switch share.fetch_poll(st.node,&t.fetch) { + case .Done: + t.used=false + // The card reads it from the store in its next frame. + if c:=s.content;c!=nil { + for &held in c.thumbs { + if held.key==t.key {held.frame=0} + } + } + case .Absent: + st.thumbs_absent[t.key]={} + t.used=false + } + } + c:=s.content + if c==nil || st.phase!=.Joined {return} + peers:[COLLAB_PEERS]Collab_Library + count:=collab_library_peers(st,peers[:]) + wants: for want in c.thumb_wants { + if want.key in st.thumbs_absent {continue} + free:=-1 + for &t,at in st.thumbs { + if t.used && t.key==want.key {continue wants} + if !t.used && free<0 {free=at} + } + if free<0 {break} + for &peer in peers[:count] { + if peer.peer!=want.peer {continue} + t:=&st.thumbs[free] + t^={used=true,key=want.key} + t.fetch=share.fetch_start(st.node,peer.node,THUMBNAIL_NAMESPACE,thumbnail_name(want.key,&t.name)) + break + } + } +} + +// Stops what reads and takes from the peers; the node was closed, so they end at once. +@(private) +collab_library_destroy :: proc(st:^Collab_State) { + for &t in st.thumbs { + if t.used {share.fetch_end(st.node,&t.fetch)} + t.used=false + } + delete(st.thumbs_absent) + st.thumbs_absent=nil + if st.libraries!=nil {collab_libraries_free(st.libraries)} + if st.reading!=nil {collab_libraries_free(st.reading)} + if st.take!=nil {collab_take_free(st.take)} + st.libraries,st.reading,st.take=nil,nil,nil +} + +// ---------------------------------------------------------------- packages.peers, packages.peer_add + +Api_Packages_Peers_Params :: struct { + refresh:bool `mirai:"optional,label=Read the libraries again now"`, + seconds:int `mirai:"optional,min=0,max=600,unit=s,label=How long it waits for the reading to end; 0: it answers at once"`, +} + +Api_Peer_Package :: struct { + id:base.Asset_Id, + name:string, + kinds:[]string, + assets:int, + bytes:i64 `mirai:"label=Of the build the peer has; -1: unknown"`, + built:bool `mirai:"label=The peer has a build of it this engine reads"`, + here:bool `mirai:"label=This machine has it with that build: adding it fetches nothing"`, + pinned:bool `mirai:"label=The project pins it"`, + thumbnail:string `mirai:"label=The key of the small thumbnail of its root, which the peer is asked for"`, + thumbnail_here:bool `mirai:"label=The store of this machine keeps that thumbnail"`, +} + +Api_Peer_Library :: struct { + peer:u8 `mirai:"label=The id of the editor in the session, as collab.status gives it"`, + name:string, + error:string `mirai:"label=Why its library could not be read"`, + packages:[]Api_Peer_Package, +} + +Api_Peer_Adding :: struct { + name:string, + adding:bool `mirai:"label=Its fetch runs"`, + received:u64 `mirai:"label=Chunk bytes received since it began"`, + error:string `mirai:"label=Why it was not added; the project did not change"`, +} + +Api_Packages_Peers_Result :: struct { + listing:bool `mirai:"label=The libraries are being read"`, + libraries:[]Api_Peer_Library `mirai:"label=Of the editors of the session that are connected; one that left is not listed"`, + adding:Api_Peer_Adding `mirai:"label=The last package taken from a peer"`, +} + +Api_Packages_Peer_Add_Params :: struct { + peer:int `mirai:"min=0,max=7,label=The editor whose library has it, as packages.peers lists it"`, + name:string `mirai:"label=The package: its id or its name, as packages.peers lists them"`, + folder:string `mirai:"optional,label=The folder under the asset folder it is listed in; none: the root of the assets"`, + seconds:int `mirai:"optional,min=1,max=7200,unit=s,label=How long it waits for the end of the add; past that it answers adding and the add goes on"`, +} + +@(private) +collab_library_api_register :: proc(s:^Session,st:^Api_State) { + api_register(s,st,{name="packages.peers",summary="The packages the machines of the other editors of the session expose in their libraries, by editor, each with its size and whether this machine has it. Read on a thread: with seconds it waits for the reading.",params=type_info_of(Api_Packages_Peers_Params),result=type_info_of(Api_Packages_Peers_Result),run=api_packages_peers}) + api_register(s,st,{name="packages.peer_add",summary="Takes a package of a peer's library into this machine's, its build included, and adds it to the project as packages.add does; if the peer leaves before it is here, the project does not change and adding says why.",params=type_info_of(Api_Packages_Peer_Add_Params),result=type_info_of(Api_Packages_Peers_Result),run=api_packages_peer_add}) +} + +@(private) +api_packages_peers_answer :: proc(s:^Session,call:^api.Call) { + r:=api.call_result(call,Api_Packages_Peers_Result) + st:=collab_state(s) + if st==nil {return} + a:=call.allocator + pinned:=make(map[base.Asset_Id]bool,context.temp_allocator) + if list,_,error:=asset.packages_load(s.project_root,context.temp_allocator);error==.None { + for entry in list.packages {pinned[entry.id]=true} + } + collab_libraries_promote(st) + r.listing=st.reading!=nil + if l:=st.libraries;l!=nil { + { + out:=make([dynamic]Api_Peer_Library,a) + for &library in l.libraries[:l.count] { + if !st.peers[library.peer].used {continue} + entry:=Api_Peer_Library{peer=u8(library.peer),name=strings.clone(collab_peer_name(s,library.peer),a),error=strings.clone(library.error,a)} + packages:=make([]Api_Peer_Package,len(library.list),a) + for &p,k in library.list { + kinds:=make([]string,len(p.kinds),a) + for kind,i in p.kinds { + kinds[i]="other" + if info,known:=asset.kind_info(kind);known {kinds[i]=strings.clone(info.name,a)} + } + packages[k]={id=p.id,name=strings.clone(p.name,a),kinds=kinds,assets=p.assets,bytes=p.bytes,built=p.built,here=p.here,pinned=pinned[p.id]} + packages[k].thumbnail,packages[k].thumbnail_here=api_thumbnail_of(s,p.root,p.objects,a) + } + entry.packages=packages + append(&out,entry) + } + r.libraries=out[:] + } + } + if t:=st.take;t!=nil { + done:=sync.atomic_load(&t.done) + r.adding={name=strings.clone(base.text_get(t.name[:]),a),adding=!done,received=share.node_totals(st.node).received-t.received} + if done {r.adding.error=strings.clone(t.error,a)} + } +} + +@(private) +api_packages_peers :: proc(s:^Session,call:^api.Call) { + params:=api.call_params(call,Api_Packages_Peers_Params) + st:=collab_state(s) + if st==nil || st.node==nil || st.phase!=.Joined { + api.call_fail(call,api.INVALID_PARAMS,"this editor is in no session: the libraries of others are those of the editors of a session (collab.host or collab.join)") + return + } + collab_libraries_read(st,params.refresh) + listing :: proc(s:^Session,user:rawptr) -> bool { + st:=collab_state(s) + if st==nil {return false} + collab_libraries_promote(st) + return st.reading!=nil + } + if params.seconds>0 {api_wait(s,f64(params.seconds),nil,listing,"listing")} + api_packages_peers_answer(s,call) +} + +@(private) +api_packages_peer_add :: proc(s:^Session,call:^api.Call) { + params:=api.call_params(call,Api_Packages_Peer_Add_Params) + if problem:=collab_library_add(s,params.peer,params.name,params.folder);problem!="" { + api.call_fail(call,api.INVALID_PARAMS,problem) + return + } + adding :: proc(s:^Session,user:rawptr) -> bool { + st:=collab_state(s) + return st!=nil && st.take!=nil && (st.take.thread!=nil || s.importer.running!=nil) + } + api_wait(s,f64(params.seconds>0?params.seconds:600),nil,adding,"adding") + api_packages_peers_answer(s,call) +} diff --git a/editor/collab_play.odin b/editor/collab_play.odin index 9f3c498ed..5b041b700 100644 --- a/editor/collab_play.odin +++ b/editor/collab_play.odin @@ -17,7 +17,7 @@ import "mirai:std/serial" collab_host_seen :: proc(st:^Collab_State) -> bool { if st==nil || st.role!=.Guest || st.phase!=.Joined {return false} host:=&st.peers[0] - return host.used && host.has_pose && wire.host_time(&st.host)-host.seen<=COLLAB_PRESENCE_SEEN + return host.used && host.has_pose && collab_now(st)-host.seen<=COLLAB_PRESENCE_SEEN } // A guest: whether it is in a session, and whether it hears the host now. diff --git a/editor/collab_project.odin b/editor/collab_project.odin new file mode 100644 index 000000000..7f28da9dd --- /dev/null +++ b/editor/collab_project.odin @@ -0,0 +1,312 @@ +package editor + +import "base:runtime" +import "core:fmt" +import "core:log" +import "core:mem" +import "core:os" +import "core:sync" +import "core:thread" +import "mirai:cook/share" +import "mirai:platform" +import wire "mirai:net" +import "mirai:project" +import "mirai:std/base" +import "mirai:std/jobs" +import "mirai:store" + +// The project travels with the session, as mirai share and mirai join carry +// it (cook/share): the host's net host also carries the channels of the +// exchange, and a node of cook/share over it serves the snapshot of the +// project and the library of the store the editor started with. A guest, +// before it says Hello, brings a folder to that snapshot with the join of +// cook/share: its open project, by three ways, or a folder it names, which it +// opens once the join is done and joins the session again from there. + +COLLAB_EXCHANGE_CHANNEL :: 2 // where the channels of the exchange start +COLLAB_LINES :: 8 // kept, failed and missing lines collab.status gives +COLLAB_LINE_BYTES :: 200 +// What the host changes during the session reaches the guests: once its files +// are quiet for COLLAB_OFFER_QUIET it takes the snapshot again, at most once a +// COLLAB_OFFER_EVERY, and tells the guests in the session when it differs; +// a guest joins it again as soon as the update it runs ends. +COLLAB_OFFER_QUIET :: 0.5 // seconds +COLLAB_OFFER_EVERY :: 2.0 // seconds +@(private) COLLAB_WATCH_BUDGET :: 512 + +@(private) +collab_channels :: proc() -> (channels:[COLLAB_EXCHANGE_CHANNEL+2]wire.Channel_Desc) { + channels[0]={kind=.Reliable,queue_bytes=256<<10} + channels[1]={kind=.Unreliable} + exchange_channels:=share.channels() + channels[COLLAB_EXCHANGE_CHANNEL]=exchange_channels[0] + channels[COLLAB_EXCHANGE_CHANNEL+1]=exchange_channels[1] + return +} + +// A guest's join: on a thread of its own, which waits for the steps of the node. +@(private) +Collab_Join :: struct { + desc:share.Join_Desc, + progress:share.Join_Progress, + report:share.Join_Report, + error:string, + done:bool, + thread:^thread.Thread, + arena:mem.Dynamic_Arena, + allocator:runtime.Allocator, + digits:[64]u8, + code:[64]u8, // of the invitation, which the editor opened on folder joins with + folder:string, // where the join writes + reopen:bool, // folder is not the open project: it is opened once the join is done + update:bool, // of a guest in the session, after the host offered a new snapshot + opening:bool, // and its editor library is being built to open it + finished:bool, // the end of the join was taken on the editor's thread +} + +// The node of a session over its host: what this editor serves and fetches. +@(private) +collab_node_start :: proc(s:^Session,st:^Collab_State) -> bool { + // The exchange reads, hashes and stores on jobs of its own, which this thread owns as it + // pumps; on the network of a test they run on the pump, as the tests of the exchange do. + worker:=jobs.Worker_Ref{} + if !st.virtual {st.node_jobs=new(jobs.Job_System)} + if st.node_jobs!=nil && jobs.system_init(st.node_jobs,{threads=jobs.THREADS_AUTO,allocator=runtime.heap_allocator()}) { + worker={system=st.node_jobs,worker=jobs.OWNER} + } else if st.node_jobs!=nil { + free(st.node_jobs) + st.node_jobs=nil + } + st.node=new(share.Node) + if share.node_init(st.node,&s.store,runtime.heap_allocator(),worker,COLLAB_EXCHANGE_CHANNEL)!=.None { + free(st.node) + st.node=nil + return false + } + return true +} + +// The host offers its project under the token of the session: its snapshot, +// with what its packages need, held by the lease of the node. +@(private) +collab_project_offer :: proc(s:^Session,st:^Collab_State) -> (problem:string) { + js:jobs.Job_System + threaded:=jobs.system_init(&js,{threads=jobs.THREADS_AUTO,allocator=runtime.heap_allocator()}) + defer if threaded {jobs.system_destroy(&js)} + worker:=jobs.Worker_Ref{} + if threaded {worker={system=&js,worker=jobs.OWNER}} + o,error:=share.offer(st.node,s.project_root,st.invite.secret,worker,context.temp_allocator) + for line in o.taken.skipped {log.warnf("the session does not carry it: %s",line)} + for line in o.problems {log.warnf("a guest will not get it: %s",line)} + if error=="" {st.offered=o.address} + return error +} + +// The host watches the files of its project; without a platform (a test) it is told with collab_project_touched. +@(private) +collab_watch_start :: proc(s:^Session,st:^Collab_State) { + if s.app.platform==nil {return} + watch,error:=platform.watch_create(s.app.platform,{path=s.project_root,recursive=true,file_capacity=1<<16}) + if error!=.None { + log.warnf("the files of %s are not watched (%v): what changes on this side reaches the guests when they join again",s.project_root,error) + return + } + st.watch,st.watching=watch,true +} + +// A file of the host's project changed: the next quiet moment offers the snapshot again. +collab_project_touched :: proc(s:^Session) { + st:=collab_state(s) + if st!=nil && st.role==.Host && st.phase==.Joined {st.changed_at=max(collab_now(st),1e-9)} +} + +// The host: what changed in its folder, and a new snapshot offered to the guests when it differs. +@(private) +collab_host_changes :: proc(s:^Session,st:^Collab_State,now:f64) { + if st.watching { + events:[64]platform.Watch_Event + count,_:=platform.watch_poll(s.app.platform,st.watch,COLLAB_WATCH_BUDGET,events[:]) + for e in events[:count] { + if project.own_path(e.path) {st.changed_at=now;break} + } + } + if st.changed_at==0 || now-st.changed_at bool { + j:=new(Collab_Join) + j.update=update + base.text_set(j.code[:],code) + mem.dynamic_arena_init(&j.arena,runtime.heap_allocator(),runtime.heap_allocator()) + j.allocator=base.dynamic_arena_allocator(&j.arena) + j.folder=fmt.aprint(folder,allocator=j.allocator) + j.reopen=reopen + j.desc={node=st.node,offer=share.offer_name(st.invite.secret,&j.digits),folder=j.folder,progress=&j.progress,connect_seconds=COLLAB_HELLO_TIMEOUT} + // The importers of the project's plugins, for a package no peer has a build of. + if s.has_project {j.desc.library=import_library} + st.join=j + // With a context of its own: the temp allocator of the editor's is freed every frame. + j.thread=thread.create_and_start_with_poly_data(j,proc(j:^Collab_Join) { + defer runtime.default_temp_allocator_destroy(&runtime.global_default_temp_allocator_data) + js:jobs.Job_System + // A core stays with the editor, which keeps drawing. + threaded:=jobs.system_init(&js,{threads=max(os.get_processor_core_count()-2,0),allocator=runtime.heap_allocator()}) + if threaded {j.desc.worker={system=&js,worker=jobs.OWNER}} + j.report,j.error=share.join(j.desc,j.allocator) + j.desc.worker={} + if threaded {jobs.system_destroy(&js)} + sync.atomic_store(&j.done,true) + }) + return j.thread!=nil +} + +// The node stepped after the host was polled, and the end of a guest's join +// taken: the open project relisted and the Hello sent, or the folder opened. +@(private) +collab_project_update :: proc(s:^Session,st:^Collab_State,now:f64) { + if st.node==nil {return} + share.node_step(st.node,now) + collab_take_update(s,st) + collab_thumbs_update(s,st) + if st.role==.Host { + collab_host_changes(s,st,now) + return + } + j:=st.join + if st.update_due && (j==nil || j.finished) { + st.update_due=false + code:=j!=nil?j.code:{} + if j!=nil {collab_join_free(j)} + st.join=nil + if !collab_join_start(s,st,base.text_get(code[:]),s.project_root,false,update=true) {log.warn("the project of the host could not be updated: no thread to do it on")} + return + } + if j==nil {return} + if j.opening { + p:=s.projects + if !projects_opening(p) && !p.next {collab_end(st,fmt.tprintf("%s could not be opened: %s",j.folder,string(p.error[:p.error_length])))} + return + } + if j.finished || !sync.atomic_load(&j.done) {return} + j.finished=true + thread.join(j.thread) + thread.destroy(j.thread) + j.thread=nil + for line in j.report.kept {log.infof("kept: %s",line)} + for line in j.report.failed {log.warnf("not written: %s",line)} + for line in j.report.packages.missing {log.warnf("%s",line)} + if j.update { + // What changed on disk reloads as any file changed on disk does; the packages are listed again. + if j.error!="" {log.warnf("the project of the host could not be updated: %s",j.error)} + library_relist(s,{root=s.project_root}) + st.updates+=1 + return + } + if j.error!="" { + collab_end(st,fmt.tprintf("the project of the host could not be brought here: %s",j.error)) + return + } + if j.reopen { + if problem:=projects_open(s,j.folder);problem!="" { + collab_end(st,fmt.tprintf("the project of the host is in %s, which could not be opened: %s",j.folder,problem)) + return + } + s.projects.join=j.code + c:=collab_of(s) + base.text_set(s.projects.join_name[:],string(c.name[:c.name_length])) + j.opening=true + return + } + library_relist(s,{root=s.project_root}) + collab_hello(s,st) +} + +// Stops the join, which ends at its next wait, and frees the node. +@(private) +collab_project_destroy :: proc(s:^Session,st:^Collab_State) { + if st.node!=nil {share.node_close(st.node)} + collab_library_destroy(st) + if st.watching {platform.watch_destroy(s.app.platform,st.watch)} + st.watching=false + if j:=st.join;j!=nil { + collab_join_free(j) + st.join=nil + } + if st.node!=nil { + if st.role==.Host && s.store.root!="" { + digits:[64]u8 + store.ref_remove(&s.store,share.NAMESPACE,share.offer_name(st.invite.secret,&digits)) + } + share.node_destroy(st.node) + free(st.node) + st.node=nil + } + if st.node_jobs!=nil { + jobs.system_destroy(st.node_jobs) + free(st.node_jobs) + st.node_jobs=nil + } +} + +@(private) +collab_join_free :: proc(j:^Collab_Join) { + if j.thread!=nil { + thread.join(j.thread) + thread.destroy(j.thread) + } + mem.dynamic_arena_destroy(&j.arena) + free(j) +} + +// What a phase of a join is doing, as Session says it. +@(private) +collab_join_text :: proc(j:^Collab_Join) -> string { + state:=share.join_progress_read(&j.progress) + if j.opening {return fmt.tprintf("Opening %s...",project_name_of(j.folder))} + switch state.phase { + case .Connecting: return "Connecting..." + case .Snapshot: return "Asking for the project..." + case .Files: + left:=state.left>=0?fmt.tprintf(", %s left",share.seconds_text(state.left)):"" + return fmt.tprintf("Files: %d of %d, %s of %s, %s/s%s",state.files_done,state.files,share.bytes_text(state.bytes_done),share.bytes_text(state.bytes),share.bytes_text(i64(state.rate)),left) + case .Packages: return fmt.tprintf("Builds: %s, %s/s",share.bytes_text(i64(state.received)),share.bytes_text(i64(state.rate))) + case .Done: + } + return "" +} + +@(private) +project_name_of :: proc(folder:string) -> string { + name:=os.base(folder) + return name!=""?name:folder +} + +// The folder a guest joins into: its open project when folder is empty or +// names it; another folder is opened once the join is done. +@(private) +collab_join_folder :: proc(s:^Session,folder:string) -> (target:string,reopen:bool) { + if folder=="" {return s.project_root,false} + absolute:=project.path_absolute(folder,context.temp_allocator) + if s.has_project && project.path_same(absolute,s.project_root) {return s.project_root,false} + return absolute,true +} diff --git a/editor/collab_share_test.odin b/editor/collab_share_test.odin new file mode 100644 index 000000000..e347473cd --- /dev/null +++ b/editor/collab_share_test.odin @@ -0,0 +1,289 @@ +package editor + +import "core:fmt" +import "core:mem" +import "core:os" +import "core:strings" +import "core:sync" +import "core:testing" +import "core:thread" +import "mirai:asset" +import "mirai:cook/import/sources" +import "mirai:cook/share" +import "mirai:core" +import "mirai:editor/history" +import wire "mirai:net" +import "mirai:std/base" +import "mirai:store" + +// An editor of a session over a simulated network: its project at its root, +// its assets under assets/ and a store of its own. +@(private="file") +Share_End :: struct { + using f:Prefab_Fixture, + store_root:string, +} + +@(private="file") +share_end_init :: proc(t:^testing.T,e:^Share_End,name:string) -> bool { + if !prefab_fixture_init(t,&e.f,name) {return false} + assets:=fmt.aprintf("%s/assets",e.root) + os.make_directory_all(assets) + asset.registry_destroy(&e.assets.registry) + asset.registry_init(&e.assets.registry,assets) + e.s.assets_root=assets + e.s.project_root=e.root + e.s.has_project=true + e.store_root=base.test_folder(t,fmt.tprintf("%s-store",name)) + opened,error:=store.open(e.store_root) + if !testing.expect_value(t,error,store.Error.None) {return false} + e.s.store=opened + w:=edited_world(&e.s) + core.world_settle(w) + history.edits_reset(&e.edits,w) + return true +} + +@(private="file") +share_end_destroy :: proc(e:^Share_End) { + collab_close(&e.s) + store.close(&e.s.store) + delete(e.s.assets_root) + prefab_fixture_destroy(&e.f) +} + +// frame: the temporary memory of the editors' frames, freed and written over at the end of each. +@(private="file") +Share_Net :: struct { + sim:wire.Sim_Network, + now:f64, + host,guest:^Share_End, + frame:mem.Arena, + frame_bytes:[]u8, +} + +// One millisecond of the network: each editor updates its session as a frame +// does between frames, then its host sends, and the frame's temporary memory goes. +@(private="file") +share_step :: proc(n:^Share_Net) { + context.temp_allocator=mem.arena_allocator(&n.frame) + n.now+=0.001 + wire.sim_network_advance(&n.sim,n.now) + for e in ([2]^Share_End{n.host,n.guest}) { + if st:=collab_state(&e.s);st!=nil { + st.now=n.now + collab_update(&e.s) + } + if st:=collab_state(&e.s);st!=nil {wire.host_update(&st.host,n.now)} + prefab_fixture_step(&e.f) + } + // As the end of a frame, and the next one writing over that memory: what the thread of a + // join holds is its own. + free_all(context.temp_allocator) + mem.set(raw_data(n.frame_bytes),0xaa,64<<10) +} + +// Steps until the guest is in the scene or its session ended, at most 600 +// virtual seconds. The clock moves only while its join waits for it, as a +// join of cook/share runs in its tests. +@(private="file") +share_until_joined :: proc(t:^testing.T,n:^Share_Net,updates:=0) -> bool { + for steps:=0;steps<600_000; { + st:=collab_state(&n.guest.s) + if !testing.expect(t,st!=nil) {return false} + if st.phase==.Joined && st.updates>=updates && (updates==0 || st.join.finished) {return true} + if st.phase==.Ended {return testing.expectf(t,false,"the session ended: %s",st.reason)} + if j:=st.join;j!=nil && !j.finished && !sync.atomic_load(&j.done) && !share.node_waiting(st.node) { + thread.yield() + continue + } + share_step(n) + steps+=1 + } + return testing.expect(t,false,"the guest did not join in 600 virtual seconds") +} + +// The host shares, the guest joins with the code of its invitation at the host's place on the +// network; each session on endpoints of its own, which nothing of an earlier one reaches. +@(private="file") +share_join :: proc(t:^testing.T,n:^Share_Net,first:=0) -> bool { + // Called between frames, as the API is. + context.temp_allocator=mem.arena_allocator(&n.frame) + if error:=collab_host(&n.host.s,0,"Ana",wire.sim_endpoint_transport(&n.sim,first));!testing.expectf(t,error==.None,"%s",collab_error_say(&n.host.s,error)) {return false} + code:=wire.invite_encode(wire.invite_make(collab_state(&n.host.s).invite,COLLAB_PORT),context.temp_allocator) + if !testing.expect_value(t,collab_join(&n.guest.s,code,"Ben",at=wire.address_memory(first),transport=wire.sim_endpoint_transport(&n.sim,first+1)),Collab_Error.None) {return false} + return share_until_joined(t,n) +} + +@(private="file") +share_file :: proc(root,path:string,text:string) { + full:=fmt.tprintf("%s/%s",root,path) + os.make_directory_all(os.dir(full)) + _=os.write_entire_file(full,transmute([]u8)text) +} + +@(private="file") +share_text :: proc(root,path:string) -> string { + data,error:=os.read_entire_file(fmt.tprintf("%s/%s",root,path),context.temp_allocator) + return error==nil?string(data):"(none)" +} + +// A short sound in a WAV file. +@(private="file") +share_wav :: proc(samples:int) -> []u8 { + file:=make([]u8,44+samples*2,context.temp_allocator) + put :: proc(file:[]u8,at:int,value:u32,size:int) {for i in 0..>uint(8*i))}} + copy(file,"RIFF");put(file,4,u32(len(file)-8),4);copy(file[8:],"WAVEfmt ") + put(file,16,16,4);put(file,20,1,2);put(file,22,1,2);put(file,24,48000,4);put(file,28,96000,4) + put(file,32,2,2);put(file,34,16,2) + copy(file[36:],"data");put(file,40,u32(samples*2),4) + for i in 0..>7)} + return file +} + +// The host's project: files of its own and a package whose source then leaves +// the machine, so a guest gets its build from the host alone; and a crate in its scene. +@(private="file") +share_host_project :: proc(t:^testing.T,n:^Share_Net) -> (crate:base.Entity_Id,ok:bool) { + root:=n.host.root + share_file(root,"project.mirai","{\n \"name\": \"Game\"\n}\n") + share_file(root,"game/game.odin","package game\n") + share_file(root,"assets/notes.txt","host 1\n") + source_folder:=base.test_folder(t,"collab-share-sources") + _=os.write_entire_file(fmt.tprintf("%s/beep.wav",source_folder),share_wav(4800)) + options:=sources.Library_Options{store=&n.host.s.store,root=root} + report,error:=sources.ingest(fmt.tprintf("%s/beep.wav",source_folder),options,allocator=context.temp_allocator) + if !testing.expectf(t,error.code==.None,"%s",error.message) {return} + added:=sources.project_add(root,report.manifest,"assets/sounds",nil,options,context.temp_allocator) + if !testing.expectf(t,len(added.failures)==0 && len(added.missing)==0,"%v",added) {return} + testing.expect(t,os.remove(fmt.tprintf("%s/beep.wav",source_folder))==nil) + s:=&n.host.s + w:=edited_world(s) + made:=entity_create(s,{name="Crate",named=true}) + core.world_settle(w) + history.edits_reset(&n.host.edits,w) + return api_entity_id(w,made),true +} + +@(private="file") +share_net_init :: proc(t:^testing.T,n:^Share_Net,host,guest:^Share_End,name:string) -> bool { + if !testing.expect(t,wire.sim_network_init(&n.sim,{seed=5,endpoints=4,packets=1<<16})) {return false} + n.frame_bytes=make([]u8,16<<20) + mem.arena_init(&n.frame,n.frame_bytes) + n.host,n.guest=host,guest + return share_end_init(t,host,fmt.tprintf("%s-host",name)) && share_end_init(t,guest,fmt.tprintf("%s-guest",name)) +} + +// A guest with an empty folder enters the session with the host's files, its +// packages listed and built from the host's store, and then its scene. +@(test) +a_guest_with_an_empty_folder_joins_with_the_hosts_files_and_packages_then_its_scene :: proc(t:^testing.T) { + n:=new(Share_Net);defer free(n) + host,guest:=new(Share_End),new(Share_End) + defer {free(host);free(guest)} + ok:=share_net_init(t,n,host,guest,"collab-share-empty") + defer {share_end_destroy(guest);share_end_destroy(host);wire.sim_network_destroy(&n.sim);delete(n.frame_bytes)} + if !ok {return} + crate,made:=share_host_project(t,n) + if !made || !share_join(t,n) {return} + + testing.expect_value(t,share_text(guest.root,"assets/notes.txt"),"host 1\n") + testing.expect_value(t,share_text(guest.root,"game/game.odin"),"package game\n") + testing.expect_value(t,share_text(guest.root,asset.PACKAGES_FILE),share_text(host.root,asset.PACKAGES_FILE)) + st:=collab_state(&guest.s) + testing.expectf(t,len(st.join.report.kept)==0 && len(st.join.report.failed)==0 && len(st.join.report.packages.missing)==0,"%v",st.join.report) + testing.expect_value(t,st.join.report.packages.fetched,1) + listed,listed_error:=asset.root_packages(guest.root,guest.store_root,context.temp_allocator) + testing.expect_value(t,listed_error,asset.Error.None) + testing.expectf(t,len(listed.problems)==0 && len(listed.listed)>0,"%v",listed) + for l in listed.listed { + a:asset.Asset + if testing.expectf(t,asset.asset_open_object(&a,&guest.s.store,l.object,context.temp_allocator)==.None,"%s is built on the guest",l.path) {asset.asset_close(&a)} + } + testing.expect(t,api_entity(edited_world(&guest.s),crate)!=core.NONE,"the guest has the host's scene") + state:=share.join_progress_read(&st.join.progress) + testing.expect_value(t,state.phase,share.Join_Phase.Done) +} + +// A file the guest changed and the host changed too stays the guest's, and +// the guest is told; the rest comes from the host. +@(test) +a_guest_that_joins_again_keeps_its_own_edit_and_is_told :: proc(t:^testing.T) { + n:=new(Share_Net);defer free(n) + host,guest:=new(Share_End),new(Share_End) + defer {free(host);free(guest)} + ok:=share_net_init(t,n,host,guest,"collab-share-kept") + defer {share_end_destroy(guest);share_end_destroy(host);wire.sim_network_destroy(&n.sim);delete(n.frame_bytes)} + if !ok {return} + if _,made:=share_host_project(t,n);!made || !share_join(t,n) {return} + collab_leave(&guest.s) + collab_leave(&host.s) + + share_file(host.root,"assets/notes.txt","host 2, longer\n") + share_file(host.root,"game/game.odin","package game // 2\n") + share_file(guest.root,"assets/notes.txt","mine\n") + if !share_join(t,n,2) {return} + testing.expect_value(t,share_text(guest.root,"assets/notes.txt"),"mine\n") + testing.expect_value(t,share_text(guest.root,"game/game.odin"),"package game // 2\n") + st:=collab_state(&guest.s) + status:Api_Collab_Join + api_collab_join_result(st.join,&status) + testing.expect_value(t,status.kept_count,1) + testing.expect(t,strings.contains(base.text_get(status.kept[0].text[:]),"assets/notes.txt"),base.text_get(status.kept[0].text[:])) +} + +// What the host changes in its folder during the session reaches the guest +// without leaving it: a change, a new file, a removal and a move are the +// host's; a file both changed and one only the guest changed stay the +// guest's, the first listed. Only what differs travels, and a scene edit the +// session carries live arrives once. +@(test) +what_the_host_changes_in_the_session_reaches_the_guest_by_three_ways :: proc(t:^testing.T) { + n:=new(Share_Net);defer free(n) + host,guest:=new(Share_End),new(Share_End) + defer {free(host);free(guest)} + ok:=share_net_init(t,n,host,guest,"collab-share-update") + defer {share_end_destroy(guest);share_end_destroy(host);wire.sim_network_destroy(&n.sim);delete(n.frame_bytes)} + if !ok {return} + if _,made:=share_host_project(t,n);!made {return} + Case :: struct {path,host,guest,then:string,kept:bool} + cases:=[?]Case{ + {path="assets/change.txt",host="host 2, changed\n",then="host 2, changed\n"}, + {path="assets/added/new.txt",host="new\n",then="new\n"}, + {path="assets/delete.txt",host="(none)",then="(none)"}, + {path="assets/move.txt",host="(none)",then="(none)"}, + {path="assets/moved/move.txt",host="moved\n",then="moved\n"}, + {path="assets/both.txt",host="host 2, both\n",guest="guest 2\n",then="guest 2\n",kept=true}, + {path="assets/mine.txt",guest="guest only\n",then="guest only\n"}, + } + for c in cases { + if c.path!="assets/added/new.txt" && c.path!="assets/moved/move.txt" {share_file(host.root,c.path,c.path=="assets/move.txt"?"moved\n":"1\n")} + } + if !share_join(t,n) {return} + gst:=collab_state(&guest.s) + first:=share.node_totals(gst.node).received + for c in cases { + switch c.host { + case "": + case "(none)": os.remove(fmt.tprintf("%s/%s",host.root,c.path)) + case: share_file(host.root,c.path,c.host) + } + if c.guest!="" {share_file(guest.root,c.path,c.guest)} + } + // A scene edit of the host travels live as ever, while the update runs. + hw:=edited_world(&host.s) + made:=entity_create(&host.s,{name="Lamp",named=true}) + core.world_settle(hw) + lamp:=api_entity_id(hw,made) + collab_project_touched(&host.s) + if !share_until_joined(t,n,updates=1) {return} + + for c in cases {testing.expectf(t,share_text(guest.root,c.path)==c.then,"%s: %q",c.path,share_text(guest.root,c.path))} + kept:=strings.join(gst.join.report.kept[:],"\n",context.temp_allocator) + testing.expectf(t,len(gst.join.report.kept)==1 && strings.contains(kept,"assets/both.txt"),"what both changed is listed: %s",kept) + testing.expect_value(t,collab_state(&host.s).offers,1) + update:=share.node_totals(gst.node).received-first + testing.expectf(t,update>0 && update u64 { case .Recent: return CONTENT_RECENT_GROUP case .Scene: return CONTENT_SCENE_GROUP case .Favorites: return CONTENT_FAVORITES_GROUP + case .Library: return 0 } return 0 } @@ -965,6 +980,15 @@ content_tree_add :: proc(c:^Content,b:^Content_Build,entries:[]Content_Entry) { content_cards_add(c,b,files) content_tree_rows_add(c,first,1,group,header) } + for root,at in c.libraries.roots { + group:=content_library_group(at) + header:=i32(len(c.lines)) + append(&c.lines,Content_Line{kind=.Header,group=group,name=root.name,virtual=.Library,library=i32(root.peer+1),parent=-1}) + if content_folded(c,group) {continue} + first:=len(c.cards) + content_package_cards_add(c,at,"") + content_tree_rows_add(c,first,1,group,header) + } content_branch_add(c,b,entries,"",0,-1) } @@ -1016,6 +1040,7 @@ content_folder_cells_add :: proc(c:^Content,b:^Content_Build,entries:[]Content_E for virtual in ([2]Content_Virtual{.Scene,.Favorites}) { if len(content_virtual_files(c,b,virtual))>0 {append(&c.cards,Content_Card{file=-1,name=labels[virtual],virtual=virtual})} } + for root in c.libraries.roots {append(&c.cards,Content_Card{file=-1,name=root.name,virtual=.Library,library=i32(root.peer+1)})} } for child in content_children(under,prefix) {append(&c.cards,Content_Card{file=-1,name=child.name,folder=child.path,cover=content_cover_of(registry,fmt.tprintf("%s/",child.path))})} content_cards_add(c,b,content_direct_files(under,prefix)) @@ -1032,6 +1057,13 @@ content_search_add :: proc(c:^Content,b:^Content_Build,entries:[]Content_Entry,p file:=&b.registry.files[index] append(hits,Hit{index,content_score(c,file,b.filter),asset_name(file.path)}) } + if c.at.virtual==.Library && !b.rows { + // Inside a root of libraries, its packages. + first:=len(c.cards) + content_package_cards_add(c,content_library_root_of(c,c.at.library),b.filter) + content_rows_add(c,first,b.columns) + return + } if c.at.virtual!=.None && !b.rows { for index in content_virtual_files(c,b,c.at.virtual) {add(c,b,&hits,index)} } else { @@ -1045,6 +1077,19 @@ content_search_add :: proc(c:^Content,b:^Content_Build,entries:[]Content_Entry,p first:=len(c.cards) content_cards_add(c,b,files) if b.rows {content_tree_rows_add(c,first,0)} else {content_rows_add(c,first,b.columns)} + // Folders searches the libraries too, each under the name of its root. + if !b.rows {return} + for root,at in c.libraries.roots { + first:=len(c.cards) + if content_package_cards_add(c,at,b.filter)==0 {continue} + cards:=c.cards[first:] + moved:=make([]Content_Card,len(cards),context.temp_allocator) + copy(moved,cards) + resize(&c.cards,first) + append(&c.lines,Content_Line{kind=.Label,name=root.name,parent=-1}) + append(&c.cards,..moved) + content_tree_rows_add(c,first,0,content_library_group(at)) + } } // What the last asset of the trail uses and what uses it; files that left the @@ -1112,6 +1157,10 @@ content_lines :: proc(c:^Content,registry:^asset.Registry,width:f32,opened:[]ass case len(c.trail)>0: content_relations_add(c,b) case filter!="": content_search_add(c,b,entries[:],prefix) case b.rows: content_tree_add(c,b,entries[:]) + case c.at.virtual==.Library: + first:=len(c.cards) + content_package_cards_add(c,content_library_root_of(c,c.at.library),"") + content_rows_add(c,first,b.columns,content_library_group(content_library_root_of(c,c.at.library))) case c.at.virtual!=.None: first:=len(c.cards) content_cards_add(c,b,content_virtual_files(c,b,c.at.virtual)) @@ -1198,14 +1247,14 @@ content_place_set :: proc(c:^Content,folder:string) {c.place_length=copy(c.place @(private) content_at_is :: proc(a,b:^Content_At) -> bool { - return a.virtual==b.virtual && string(a.folder[:a.length])==string(b.folder[:b.length]) + return a.virtual==b.virtual && a.library==b.library && string(a.folder[:a.length])==string(b.folder[:b.length]) } // Enters a folder or a virtual one. reset: it forgets where the browser was, // as the root of the breadcrumb and a chip do. -content_enter :: proc(c:^Content,virtual:Content_Virtual,folder:="",reset:=false) { +content_enter :: proc(c:^Content,virtual:Content_Virtual,folder:="",reset:=false,library:i32=0) { next:Content_At - next.virtual=virtual + next.virtual,next.library=virtual,library next.length=copy(next.folder[:],folder) if reset { clear(&c.history) @@ -1290,6 +1339,8 @@ content_folder_exists :: proc(registry:^asset.Registry,folder:string) -> bool { @(private) content_navigation_restore :: proc(c:^Content,registry:^asset.Registry) { c.restore=false + // A root of libraries is there while its peer is. + if c.at.virtual==.Library && content_library_root_of(c,c.at.library)<0 {c.at={}} place:=string(c.place[:c.place_length]) for place!="" && !content_folder_exists(registry,place) {place=content_folder_of(place)} c.place_length=len(place) @@ -1412,6 +1463,11 @@ content_move_items :: proc(s:^Session,id:base.Asset_Id,from:string) { @(private) content_drop :: proc(s:^Session,folder:string) { g:=s.gui + // A package of a library carried onto a folder of the project is added there. + if g.drag.kind==.Package { + if !content_package_busy(s) {content_package_add(s,g.drag.library,g.drag.asset_id,folder)} + return + } if g.drag.kind!=.Asset || base.text_get(g.drag.part[:])!="" {return} r:=files_registry(s) if r==nil {return} @@ -1698,9 +1754,9 @@ content_preview :: proc(s:^Session,file:^asset.Registry_File,points:f32,clip:u32 a:=s.app.assets if a.renderer==nil {return} switch file.kind { - case asset.KIND_MATERIAL: if preview_load_turn(s,file.id) {quad=ui.quad_whole(host.UI_QUAD_MATERIAL_PREVIEW,host.assets_material_row(a,file.id))} + case asset.KIND_MATERIAL: quad=preview_material(s,file.id,points) case asset.KIND_MESH: quad=preview_geometry(s,.None,file.id,points) - case asset.KIND_TEXTURE: if preview_load_turn(s,file.id) {quad=texture_quad(s,file.id)} + case asset.KIND_TEXTURE: quad=preview_texture(s,file.id,points) case asset.KIND_SCENE: quad=preview_document(s,file.id,points) if asset.registry_prefab(file) else preview_scene(s,file.id,points) case: quad=preview_provided(s,file,clip,points) } @@ -2027,6 +2083,7 @@ content_folder_icon :: proc(virtual:Content_Virtual) -> ui.Icon { case .Recent: return .History case .Scene: return .Scene case .Favorites: return ui.icon_user(u8(User_Icon.Star)) + case .Library: return .Import } return .None } @@ -2078,6 +2135,7 @@ content_header :: proc(s:^Session,line:Content_Line,id:ui.Id,width:f32,marked:bo case .Recent: icon,color=.History,ui.tone(ctx,"text_muted") case .Scene: icon,color=ui.icon_user(u8(User_Icon.Scene)),ui.tone(ctx,"text_muted") case .Favorites: icon,color=ui.icon_user(u8(User_Icon.Star_Filled)),STAR_COLOR + case .Library: icon,color=.Import,ui.tone(ctx,"text_muted") } box:=asset_row_box(ctx,id,CONTENT_INSET+f32(line.depth)*CONTENT_TREE_INDENT,marked,it.hot,content_keyboard(ctx,it)) box.width,box.gap=ui.fixed(width),CONTENT_TREE_GAP @@ -2106,7 +2164,7 @@ content_header_take :: proc(s:^Session,act:^Content_Actions,id:ui.Id,it:ui.Inter if line.virtual==.None {content_place_set(s.content,line.path)} it.clicked=false } - content_folder_take(s,act,id,it,line.virtual,line.path) + content_folder_take(s,act,id,it,line.virtual,line.path,library=line.library) } // What the name that stays on top covers past the edge of the list: at a @@ -2129,6 +2187,7 @@ content_fold_flip :: proc(c:^Content,group:u64) { Content_Actions :: struct { enter,reset:bool, // go to folder or virtual; reset: forgetting where it was virtual:Content_Virtual, + library:i32, // of a virtual Library folder:string, back:bool, cut:int, // the trail keeps this many assets; 0: as it is @@ -2147,19 +2206,19 @@ content_folder_marked :: proc(s:^Session,id:ui.Id,virtual:Content_Virtual,folder if virtual!=.None {return false} if ui.popup_is_open(ctx,content_create_id(g)) && string(s.content.create_into[:s.content.create_into_length])==folder {return true} rect,built:=ui.widget_rect(ctx,id) - return built && g.drag.kind==.Asset && drag_over(g,rect) + return built && (g.drag.kind==.Asset || g.drag.kind==.Package) && drag_over(g,rect) } // What a name, a chip or a cell of a folder does: a click enters it, the right // button opens Create in it and a card let go on it moves there. A virtual // folder is only entered. @(private) -content_folder_take :: proc(s:^Session,act:^Content_Actions,id:ui.Id,it:ui.Interaction,virtual:Content_Virtual,folder:string,reset:=false) { +content_folder_take :: proc(s:^Session,act:^Content_Actions,id:ui.Id,it:ui.Interaction,virtual:Content_Virtual,folder:string,reset:=false,library:i32=0) { g:=s.gui ctx:=g.ctx - if it.clicked {act.enter,act.reset,act.virtual,act.folder=true,reset,virtual,folder} + if it.clicked {act.enter,act.reset,act.virtual,act.folder,act.library=true,reset,virtual,folder,library} if virtual!=.None {return} - if g.drag.kind==.Asset { + if g.drag.kind==.Asset || g.drag.kind==.Package { if rect,built:=ui.widget_rect(ctx,id);built && drag_dropped(g,rect) {act.dropped,act.drop=true,strings.clone(folder,context.temp_allocator)} } if it.right { @@ -2249,15 +2308,16 @@ content_strip_row :: proc(s:^Session) { content_tabs_build :: proc(s:^Session,act:^Content_Actions,width:f32) -> (height:f32) { c:=s.content ctx:=s.gui.ctx - Chip :: struct {virtual:Content_Virtual,label,path:string,row:int} + Chip :: struct {virtual:Content_Virtual,label,path:string,row:int,library:i32} chips:=make([dynamic]Chip,context.temp_allocator) labels:=CONTENT_VIRTUAL_LABELS append(&chips,Chip{label="All"}) for virtual in ([3]Content_Virtual{.Recent,.Favorites,.Scene}) {append(&chips,Chip{virtual=virtual,label=labels[virtual]})} + for root,at in c.libraries.roots {append(&chips,Chip{virtual=.Library,label=root.name,library=i32(root.peer+1),path="",row=0})} for tab in c.tabs {append(&chips,Chip{label=tab.name,path=tab.path,row=tab.row})} at:=string(c.at.folder[:c.at.length]) chip_width :: proc(ctx:^ui.Context,chip:Chip) -> f32 { - if chip.virtual!=.None {return CONTENT_TAB_ICON+2*CONTENT_TAB_PADDING} + if chip.virtual!=.None && chip.virtual!=.Library {return CONTENT_TAB_ICON+2*CONTENT_TAB_PADDING} return ui.text_width(&ctx.font,chip.label,ui.metric(ctx,"font"))+2*CONTENT_TAB_PADDING+1 } rows:=0 @@ -2288,18 +2348,20 @@ content_tabs_build :: proc(s:^Session,act:^Content_Actions,width:f32) -> (height if ui.box(ctx,{width=ui.GROW,height=ui.fixed(CONTENT_TAB),gap=CONTENT_TAB_GAP}) { for chip in chips[first:end] { group:=content_virtual_group(chip.virtual) + if chip.virtual==.Library {group=content_library_group(content_library_root_of(c,chip.library))} if chip.virtual==.None && chip.path=="" {group=CONTENT_ALL_GROUP} if chip.path!="" {group=base.hash64(transmute([]u8)chip.path)} id:=ui.id(ctx,"content_folder",int(group)) it:=ui.interact(ctx,id,{.Focusable}) switch { + case chip.virtual==.Library: content_tab_draw(ctx,id,chip.label,c.at.virtual==.Library && c.at.library==chip.library,it,false) case chip.virtual!=.None: on:=c.at.virtual==chip.virtual;content_tab_draw(ctx,id,"",on,it,false,content_folder_icon(chip.virtual)) case chip.path=="": content_tab_draw(ctx,id,chip.label,c.at.virtual==.None && at=="",it,content_folder_marked(s,id,.None,"")) case: content_tab_draw(ctx,id,chip.label,c.at.virtual==.None && (at==chip.path || asset_path_inside(at,chip.path)),it,content_folder_marked(s,id,.None,chip.path)) } if it.hot && chip.path!="" {ui.tooltip(ctx,chip.path)} - if it.hot && chip.virtual!=.None {ui.tooltip(ctx,chip.label)} - content_folder_take(s,act,id,it,chip.virtual,chip.path,reset=true) + if it.hot && chip.virtual!=.None && chip.virtual!=.Library {ui.tooltip(ctx,chip.label)} + content_folder_take(s,act,id,it,chip.virtual,chip.path,reset=true,library=chip.library) } } first=end @@ -2362,8 +2424,10 @@ content_crumbs_build :: proc(s:^Session,act:^Content_Actions) { content_folder_take(s,act,root,content_crumb(ctx,root,"Project",c.at.length==0 && c.at.virtual==.None,false,content_folder_marked(s,root,.None,"")),.None,"",reset=true) if c.at.virtual!=.None { labels:=CONTENT_VIRTUAL_LABELS + label:=labels[c.at.virtual] + if root:=content_library_root_of(c,c.at.library);c.at.virtual==.Library && root>=0 {label=c.libraries.roots[root].name} ui.label(ctx,"/",ui.metric(ctx,"font"),ui.tone(ctx,"text_faint")) - ui.label(ctx,labels[c.at.virtual],ui.metric(ctx,"font"),ui.tone(ctx,"text")) + ui.label(ctx,label,ui.metric(ctx,"font"),ui.tone(ctx,"text")) } else if folder!="" { start,index:=0,1 for i in 0..=len(folder) { @@ -2666,6 +2730,10 @@ content_build :: proc(s:^Session,w:core.World_Ref,height:f32,width:f32=CONTENT_W if c.creating && asset_create_refusal(s)!="" {content_create_end(c)} if c.created!={} {content_created(s,&a.registry)} if c.reveal!={} {content_reveal_open(c,&a.registry)} + content_libraries_update(s,c) + clear(&c.thumb_wants) + // A root whose peer left goes, and the browser with it. + if c.at.virtual==.Library && content_library_root_of(c,c.at.library)<0 {content_enter(c,.None,reset=true)} lines:=content_lines(c,&a.registry,width,opened,scene_kinds[:],&ctx.font,ui.metric(ctx,"font_small")) searching:=strings.trim_space(string(c.filter[:c.filter_length]))!="" && len(c.trail)==0 if take_first && searching && len(c.cards)>0 && c.cards[0].file>=0 {content_arm(s,&a.registry.files[c.cards[0].file],c.cards[0].name)} @@ -2758,6 +2826,11 @@ content_build :: proc(s:^Session,w:core.World_Ref,height:f32,width:f32=CONTENT_W content_header_take(s,&act,id,content_header(s,line,id,width,marked),line) case .Row: card:=c.cards[line.first] + if card.pack>0 { + it,_:=content_package_item(s,card,line.group,width,int(line.depth)) + content_package_action(s,card,it) + continue + } it,id:=content_item(s,card,line.group,width,whole_line,first_card && line.first==0,depth=int(line.depth)) if it.focused {c.card_focus,c.focus_asset=id,card.clip>0?base.Asset_Id{}:a.registry.files[card.file].id} if path:=content_asset_action(s,w,card,it);path!="" {act.open=path} @@ -2772,13 +2845,19 @@ content_build :: proc(s:^Session,w:core.World_Ref,height:f32,width:f32=CONTENT_W _,cell:=content_cells(c.view,width) if ui.box(ctx,{width=ui.fixed(width),height=ui.fixed(c.heights[index]),gap=CONTENT_GAP if c.view.mode==.Grid else 0}) { for card,at in c.cards[line.first:][:line.count] { + if card.pack>0 { + it,_:=content_package_item(s,card,line.group,cell) + content_package_action(s,card,it) + continue + } if card.file<0 { group:=content_virtual_group(card.virtual) if card.virtual==.None {group=base.hash64(transmute([]u8)card.folder)} + if card.virtual==.Library {group=content_library_group(content_library_root_of(c,card.library))} id:=ui.id(ctx,"content_folder",int(group)) marked:=content_folder_marked(s,id,card.virtual,card.folder) it:=content_folder_cell(s,card,id,cell,line.names,marked) - content_folder_take(s,&act,id,it,card.virtual,card.folder) + content_folder_take(s,&act,id,it,card.virtual,card.folder,library=card.library) continue } it,id:=content_item(s,card,line.group,cell,whole_line,first_card && line.first==0 && at==0,line.names) @@ -2832,10 +2911,11 @@ content_build :: proc(s:^Session,w:core.World_Ref,height:f32,width:f32=CONTENT_W c.revision+=1 } if act.back {content_back(c)} - if act.enter {content_enter(c,act.virtual,act.folder,act.reset)} + if act.enter {content_enter(c,act.virtual,act.folder,act.reset,act.library)} if act.finish {content_create_finish(s)} if act.dropped {content_drop(s,act.drop)} if path:=content_menu(s,w);path!="" {act.open=path} + content_package_menu(s) content_create_menu(s) if act.open!="" {file_open_path(s,act.open)} host.assets_commit(a) diff --git a/editor/content_library.odin b/editor/content_library.odin new file mode 100644 index 000000000..6ef3289ae --- /dev/null +++ b/editor/content_library.odin @@ -0,0 +1,452 @@ +package editor + +import "core:fmt" +import "core:mem" +import "base:runtime" +import "core:slice" +import "core:strings" +import "core:sync" +import "core:os" +import "mirai:asset" +import "mirai:cook/share" +import "mirai:host" +import "mirai:project" +import "mirai:std/base" +import "mirai:store" +import "mirai:ui" + +// The libraries in the content browser: beside the project's own content, a +// root for this machine's library and, in a session, one for the library of +// each other editor, named as the session names it. A root holds the packages +// the project does not pin; a peer's package this machine has is listed once, +// under this machine. Two packages of one name in a root are told apart by +// where their source was. Adding one is the browser's gesture for bringing +// content in: carried onto a folder of the project, or Add to project of its +// menu (library.odin and collab_library.odin do the adding). + +// A root of libraries: this machine's (peer -1) or that of the editor at peer in the session. +Content_Library_Root :: struct { + peer:int, + name:string, +} + +// A package as a root of the browser shows it. +Content_Package :: struct { + root:int, // of Content_Libraries.roots + id:base.Asset_Id, + name:string, + label:string, // its name, with where its source was when another of the root has that name + kind:asset.Fourcc, // of its first asset + assets:int, + bytes:i64, // of its build; -1: unknown + here:bool, // this machine has its build: nothing has to be fetched + built:bool, // a build of it this engine reads is here or at its peer + shown:store.Hash, // the object of its root in that build, which its card shows; zero: none + shown_id:base.Asset_Id, // this machine's: the asset of that object, which the library mount lists + objects:[]store.Hash, // of every asset of that build, sorted: with shown, the key of its thumbnail +} + +// A thumbnail of a card in the atlas (slot), or one the store did not have when it was looked for (slot -1). +@(private) +Content_Thumb :: struct {key:u64,slot:i32,frame:u64} +CONTENT_THUMBS :: 64 +CONTENT_THUMB_RECHECK :: 30 // frames before a store that lacked a thumbnail is looked at again + +// A thumbnail a visible card of a peer's root lacks, which the session asks that peer for. +Content_Thumb_Want :: struct {key:u64,peer:int} + +// What a peer's library gave, as the listing takes it. +Content_Peer_Listing :: struct { + peer:int, + name:string, + list:[]share.Peer_Package, +} + +@(private) +Content_Libraries :: struct { + arena:^mem.Dynamic_Arena, // what roots and packages hold, apart from the storage of the browser + ready:bool, + roots:[dynamic]Content_Library_Root, + packages:[dynamic]Content_Package, + key:[4]u64, + mounted:bool, // LIBRARY_MOUNT lists the packages of mounted_key + mounted_key:u64, +} + +// The roots and their packages from this machine's library, the packages the +// project pins and the libraries of the peers, in allocator. machine: this +// machine has a library (a store); without one it has no root. +content_library_listing :: proc(local:[]Library_Package,peers:[]Content_Peer_Listing,machine:=true,allocator:=context.allocator) -> (roots:[dynamic]Content_Library_Root,packages:[dynamic]Content_Package) { + roots=make([dynamic]Content_Library_Root,allocator) + packages=make([dynamic]Content_Package,allocator) + pinned:=make(map[base.Asset_Id]bool,context.temp_allocator) + here:=make(map[base.Asset_Id]bool,context.temp_allocator) + for &p in local { + here[p.id]=true + if p.pinned {pinned[p.id]=true} + } + Origin :: struct {at:int,origin:string} + origins:=make([dynamic]Origin,context.temp_allocator) + if machine {append(&roots,Content_Library_Root{peer=-1,name="This machine"})} + for &p in local { + if p.pinned || !machine {continue} + entry:=Content_Package{root=0,id=p.id,name=strings.clone(p.name,allocator),assets=len(p.assets),here=len(p.assets)>0,built=true} + for a in p.assets { + if entry.kind==0 {entry.kind=a.kind} + entry.bytes+=a.bytes + entry.here&&=a.built + } + entry.built=entry.here + entry.shown,entry.shown_id,entry.objects=p.root,p.shown,slice.clone(p.objects,allocator) + append(&origins,Origin{len(packages),p.origin}) + append(&packages,entry) + } + for &listing in peers { + root:=len(roots) + append(&roots,Content_Library_Root{peer=listing.peer,name=strings.clone(listing.name,allocator)}) + for &p in listing.list { + if pinned[p.id] || here[p.id] {continue} + entry:=Content_Package{root=root,id=p.id,name=strings.clone(p.name,allocator),assets=p.assets,bytes=p.bytes,here=p.here,built=p.built} + if len(p.kinds)>0 {entry.kind=p.kinds[0]} + entry.shown,entry.objects=p.root,slice.clone(p.objects,allocator) + append(&origins,Origin{len(packages),p.origin}) + append(&packages,entry) + } + } + // A name two packages of a root share takes where each one's source was: its folder, or all of it when that is shared too. + for &o in origins {packages[o.at].label=packages[o.at].name} + for i in 0.. bool {return a.root string { + path,_:=strings.replace_all(origin,"\\","/",context.temp_allocator) + path=strings.trim_right(path,"/") + slash:=strings.last_index_byte(path,'/') + if slash<=0 {return ""} + folder:=path[:slash] + if at:=strings.last_index_byte(folder,'/');at>=0 {folder=folder[at+1:]} + return folder +} + +// Reads the libraries again when what they come from changed: the registry +// (what the project pins), the libraries of the peers or a package taken. +@(private) +content_libraries_update :: proc(s:^Session,c:^Content) { + l:=&c.libraries + st:=collab_state(s) + peers_key:u64 + if st!=nil && st.node!=nil && st.phase==.Joined { + collab_libraries_read(st) + collab_libraries_promote(st) + peers_key=u64(uintptr(st.libraries)) + for peer in st.peers {peers_key=base.hash_combine(peers_key,u64(peer.used))} + } + registry:u64 + if a:=s.app.assets;a!=nil {registry=a.registry.revision} + key:=[4]u64{registry,peers_key,c.library_revision,u64(len(s.store.root))} + if l.ready && key==l.key {return} + l.key,l.ready=key,true + if l.arena==nil { + l.arena=new(mem.Dynamic_Arena,runtime.heap_allocator()) + mem.dynamic_arena_init(l.arena,runtime.heap_allocator(),runtime.heap_allocator()) + } + l.roots,l.packages=nil,nil + mem.dynamic_arena_free_all(l.arena) + allocator:=base.dynamic_arena_allocator(l.arena) + local:[]Library_Package + if s.store.root!="" {local,_=library_list(s,s.project_root,context.temp_allocator)} + listings:=make([dynamic]Content_Peer_Listing,context.temp_allocator) + if st!=nil && st.libraries!=nil && st.phase==.Joined { + for &library in st.libraries.libraries[:st.libraries.count] { + if !st.peers[library.peer].used || library.error!="" {continue} + name:=collab_peer_name(s,library.peer) + append(&listings,Content_Peer_Listing{peer=library.peer,name=name!="" ? name : "Editor",list=library.list}) + } + } + roots,packages:=content_library_listing(local,listings[:],s.store.root!="",allocator) + l.roots,l.packages=roots,packages + library_mount_update(s,local) + c.revision+=1 +} + +// The group of the row of a root, apart from those of folders and virtual ones. +@(private) +content_library_group :: proc "contextless" (root:int) -> u64 {return base.hash_combine(0x6c6962,u64(root+2))} + +// Whether a package shows with the filters of the browser: a search of its +// label and the kind chosen; the filters of the project's use show none. +@(private) +content_package_matches :: proc(c:^Content,p:^Content_Package,filter:string) -> bool { + if c.unused || c.selection_only {return false} + if c.category!=.None { + category,_:=content_category(p.kind) + if category!=c.category {return false} + } + if filter=="" {return true} + for token in strings.fields(filter,context.temp_allocator) { + if !strings.contains(strings.to_lower(p.label,context.temp_allocator),strings.to_lower(token,context.temp_allocator)) {return false} + } + return true +} + +// The cards of the packages of root the filters take. +@(private) +content_package_cards_add :: proc(c:^Content,root:int,filter:string) -> int { + count:=0 + for &p,k in c.libraries.packages { + if p.root!=root || !content_package_matches(c,&p,filter) {continue} + append(&c.cards,Content_Card{file=-1,pack=i32(k+1),name=p.label}) + count+=1 + } + return count +} + +// The root a virtual Library folder shows: Content_At.library is its peer, +1. +@(private) +content_library_root_of :: proc(c:^Content,library:i32) -> int { + for root,at in c.libraries.roots { + if i32(root.peer+1)==library {return at} + } + return -1 +} + +// The state of fetching a package of a peer, as its card says it. +@(private) +content_package_state :: proc(s:^Session,p:^Content_Package) -> (text:string,failed:bool,why:string) { + st:=collab_state(s) + if st!=nil && st.take!=nil && st.take.id==p.id { + t:=st.take + if !sync.atomic_load(&t.done) || t.thread!=nil { + received:=share.node_totals(st.node).received-t.received + if p.bytes>0 {return fmt.tprintf("Fetching %d%%",min(int(received*100/u64(p.bytes)),99)),false,""} + return "Fetching",false,"" + } + if t.error!="" {return "Not added",true,t.error} + } + // A package of a peer's root is not in this machine's library yet: adding takes it, even when its build is here. + peer:=content_package_peer(s,p) + switch { + case peer<0: return p.bytes>0 ? share.bytes_text(p.bytes) : "",false,"" + case p.bytes>=0: return fmt.tprintf("%s to fetch",share.bytes_text(p.bytes)),false,"" + } + return "To fetch",false,"" +} + +@(private) +content_package_peer :: proc(s:^Session,p:^Content_Package) -> int { + roots:=s.content.libraries.roots + return p.root (it:ui.Interaction,id:ui.Id) { + c:=s.content + ctx:=s.gui.ctx + p:=&c.libraries.packages[card.pack-1] + id=ui.id(ctx,"content_package",int(base.hash_combine(group,p.id[0]~p.id[1]))) + it=ui.interact(ctx,id,{.Focusable}) + selected:=c.package_selected==p.id && c.package_root==p.root + icon,color:=asset_icon(s,p.kind) + state,failed,why:=content_package_state(s,p) + quad:=content_package_thumbnail(s,p,20 if depth>=0 || c.view.mode==.List else cell) + ink:=ui.tone(ctx,"danger_soft") if failed else ui.tone(ctx,"text_muted") + if depth>=0 || c.view.mode==.List { + box:=asset_row_box(ctx,id,CONTENT_INSET+f32(max(depth,0))*CONTENT_TREE_INDENT,selected,it.hot,content_keyboard(ctx,it)) + if depth>=0 {box.width,box.gap=ui.fixed(cell),CONTENT_TREE_GAP} + if ui.box(ctx,box) { + if depth>=0 {ui.spacer(ctx,CONTENT_TREE_ARROW,1)} + if ui.box(ctx,{width=ui.fixed(20),height=ui.fixed(20),align_x=.Center,align_y=.Center,radius=2,clip_x=true,clip_y=true}) { + if quad.kind!=.Solid {ui.image(ctx,{quad=quad,width=ui.GROW,height=ui.GROW})} else {ui.icon(ctx,icon,color,14)} + } + ui.label(ctx,card.name,ui.metric(ctx,"font"),ui.tone(ctx,"text"),0,font=ui.FONT_MEDIUM if selected else 0) + ui.spacer(ctx,-1,1) + if state!="" {ui.label(ctx,state,ui.metric(ctx,"font_small"),ink)} + ui.spacer(ctx,4,1) + } + } else { + names:=u8(2) + if ui.box(ctx,asset_card_box(ctx,id,c.view,cell,selected,content_keyboard(ctx,it),names)) { + stroke,stroke_width:=asset_frame_edge(ctx,it.hot,selected) + asset_frame(s,cell,CONTENT_FRAME,stroke,stroke_width,quad,icon,color,p.kind) + if content_shows_names(c.view) {content_name(ctx,card.name,selected,cell,names)} + if content_package_peer(s,p)>=0 || failed { + if ui.box(ctx,{width=ui.FIT,height=ui.FIT,padding={4,1,4,1},radius=2,fill=ui.tone(ctx,"panel"),float={to=.Parent,anchor=.Top_Right,pivot=.Top_Right,offset={-3,3}}}) { + ui.icon(ctx,.Import if !failed else .Cross,ink,10) + } + } + } + } + if it.hot { + tip:=fmt.tprintf("%s, %s",card.name,state) if state!="" else card.name + if why!="" {tip=fmt.tprintf("%s: %s",tip,why)} + ui.tooltip(ctx,tip) + } + return +} + +// What a click, a press and the right button do on a package: select it, +// carry it to a folder of the project, open its menu. +@(private) +content_package_action :: proc(s:^Session,card:Content_Card,it:ui.Interaction) { + c:=s.content + g:=s.gui + p:=&c.libraries.packages[card.pack-1] + if it.pressed { + d:=Drag{kind=.Package,asset_id=p.id,asset_kind=p.kind,library=i32(c.libraries.roots[p.root].peer+1)} + d.icon,d.color=asset_icon(s,p.kind) + drag_begin(g,d,card.name) + } + if it.clicked || it.right {c.package_selected,c.package_root=p.id,p.root} + if it.right { + c.package_menu,c.package_menu_library=p.id,i32(c.libraries.roots[p.root].peer+1) + ui.popup_open(g.ctx,content_package_menu_id(g),{g.ctx.pointer.x,g.ctx.pointer.y,0,0}) + } +} + +@(private) +content_package_menu_id :: proc(g:^Editor_Ui) -> ui.Id {return shell_id(g,"content_package_menu")} + +// The menu of a package: Add to project, into the folder the browser makes in. +@(private) +content_package_menu :: proc(s:^Session) { + c:=s.content + g:=s.gui + ctx:=g.ctx + popup:=content_package_menu_id(g) + if !ui.popup_is_open(ctx,popup) {return} + if ui.popup(ctx,popup,{min_width=180}) { + ui.id_push(ctx,popup) + defer ui.id_pop(ctx) + if ui.menu_item(ctx,ui.id(ctx,"add"),"Add to project",{icon=.Import,disabled=s.mode!=.Edit || content_package_busy(s)}) { + content_package_add(s,c.package_menu_library,c.package_menu,content_create_place(c)) + } + } +} + +// A package being taken from a peer or an import running: nothing else is added meanwhile. +@(private) +content_package_busy :: proc(s:^Session) -> bool { + if s.importer.running!=nil {return true} + st:=collab_state(s) + return st!=nil && st.take!=nil && (st.take.thread!=nil || !sync.atomic_load(&st.take.done)) +} + +// Adds the package id of the library (its peer +1, 0: this machine) to folder of the project. +@(private) +content_package_add :: proc(s:^Session,library:i32,id:base.Asset_Id,folder:string) { + buffer:[base.ASSET_ID_TEXT]u8 + text:=base.asset_id_format(id,buffer[:]) + problem:=library==0 ? library_add(s,text,nil,folder) : collab_library_add(s,int(library-1),text,folder) + if problem!="" {notify(s,problem)} +} + +// The thumbnail of the card of a package at points: in the atlas, else from +// the store of this machine, read a few a frame; a card of a peer's root that +// the store lacks asks the session for it (Content.thumb_wants), which asks +// that peer. Nothing is drawn for a card, and nothing of the package fetched. +@(private) +content_package_thumbnail :: proc(s:^Session,p:^Content_Package,points:f32) -> ui.Quad { + c:=s.content + if p.shown=={} || s.store.root=="" {return {}} + // One of this machine: the preview its asset has, which the browser draws + // and keeps as it does for the project's (the library mount lists it). + if content_package_peer(s,p)<0 { + if a:=s.app.assets;a!=nil { + if file,found:=asset.registry_find(&a.registry,p.shown_id);found { + _,_,quad:=content_preview(s,file,points) + return quad + } + } + } + level:=Thumbnail_Level.Large if points*max(s.app.density,1)>THUMBNAIL_SIZE else .Small + key:=thumbnail_package_key(level,thumbnail_programs(s),p.shown,p.objects) + held:^Content_Thumb + oldest:=0 + for &t,at in c.thumbs { + if t.key==key {held=&t;break} + if t.frame=0 { + held.frame=s.frame + if quad:=thumbnail_image_quad(s,int(held.slot),key);quad.kind!=.Solid {return quad} + held.slot=-1 + } + if held!=nil && held.slot<0 && s.frame=THUMBNAIL_LOADS { + held.frame=0 // its turn comes in another frame + return {} + } + th.loads+=1 + sides:=THUMBNAIL_SIDES + if thumbnail_store_load(&s.store,sides[level],key,&th.file,&th.image) { + held.slot=i32(thumbnail_image_put(s,key,level,th.image[:])) + return thumbnail_image_quad(s,int(held.slot),key) + } + } + if peer:=content_package_peer(s,p);peer>=0 && len(c.thumb_wants)0 && wanted<30,"the visible cards ask: %d",wanted) + first:=thumbnail_package_key(.Small,thumbnail_programs(&session),peers[0].root,peers[0].objects) + last:=thumbnail_package_key(.Small,thumbnail_programs(&session),peers[199].root,peers[199].objects) + asked_first,asked_last:=false,false + for want in c.thumb_wants { + testing.expect_value(t,want.peer,1) + asked_first||=want.key==first + asked_last||=want.key==last + } + testing.expect(t,asked_first && !asked_last,"the first card is in sight, the last is not") +} diff --git a/editor/editor.odin b/editor/editor.odin index c348c9319..a3406360d 100644 --- a/editor/editor.odin +++ b/editor/editor.odin @@ -21,6 +21,7 @@ import "mirai:project" import "mirai:std/base" import "mirai:std/jobs" import "mirai:std/subprocess" +import "mirai:store" import "mirai:ui" // The editor of a project: the application with its window, the World that @@ -96,11 +97,11 @@ Run :: struct { @(private) USAGE_TEXT :: `Usage: [--project ] [--scene ] [--headless] [--hidden] [--no-gpu] [--frames ] [--capture ] [--validation] [--width ] [--height ] [--density ] [--engine ] [--api-port ] [--no-api] - [--share [port]] [--join ] [--name ] [--preferences] + [--share [port]] [--join ] [--name ] [--preferences] [--store ] --api-port port of the agent API (JSON-RPC and MCP); 7940 if missing, or a free one when it is taken; 0: a free one, never 7940 --no-api without the agent API --share shares the scene with other editors; port 7950 if missing. Prints the invitation code - --join joins the session of another editor with the code of its invitation (R2M0-253B-DK5G-0W87-M0MA-3CP3-TKJZ-C1RR-EJT) + --join joins the session of another editor with the code of its invitation (1R2M-0253-BDK5-G0W8-7DDP-ERX1-4HQA-72WY-3G7Z-9TCN-6G4Z-H69R-53KS-D1YS-JQ3K-6666-YHZK-J1MS-WDVS-Q5ZM-MZ0N-BE4) --name name in the shared session --project project folder; the current one if missing --scene scene to open: a file or a path relative to the assets folder @@ -111,11 +112,13 @@ USAGE_TEXT :: `Usage: [--project ] [--scene ] [--headless] [--hidd --frames ends after n frames --capture writes the last frame as a PNG at the end --validation turns on the validation of the graphics API - --engine engine root for Build; if missing, MIRAI_ROOT or the engine above the executable or the working folder` + --engine engine root for Build; if missing, MIRAI_ROOT or the engine above the executable or the working folder + --store the store of the packages it imports and lists; the machine's if missing` // The arguments the editor was started with. Options :: struct { project,scene,engine,capture:string, + store:string, // the store of the library; "": the machine's documents:string, // the tabs a restart left (documents_restore) gpu_lost:string, // what stopped the GPU of the editor whose restart this is built:bool, // a Build of the editor whose restart this is succeeded: its library is the one loaded @@ -180,6 +183,9 @@ Session :: struct { engine_stamp:string, // of the binaries of the engine it runs (build.engine_stamp_of): a build of the engine afterwards changes the one beside them library_stamp:string, // of the library of the project it loaded last (build.target_stamp_of) assets_root:string, + // The machine store the library keeps the packages of imports in (--store, + // else the machine's); its root is "" when the machine has none. + store:store.Store, has_project:bool, // the folder has a project.mirai settings_unsaved:bool, // a setting changed that the file of its layer does not have yet settings_layer:project.Settings_Layer, // the one the settings dialog edits @@ -414,14 +420,13 @@ open :: proc(info:^core.World_Info,args:[]string) -> (s:^Session,code:int) { maximized=!options.headless && options.width==0 && options.height==0, } if options.launcher && options.width==0 && options.height==0 {window.size,window.fixed_size,window.maximized=PROJECTS_WINDOW,true,false} - mounts:=make([dynamic]asset.Registry_Mount,context.temp_allocator) has_project:=false if p,error:=project.open(root,engine,context.temp_allocator);error.code==.None { has_project=true - append(&mounts,..host.project_asset_mounts(&p,context.temp_allocator)) if title:=project.window_title(&p.manifest);title!="" {window.title=strings.clone(title,context.temp_allocator)} } - append(&mounts,asset.Registry_Mount{folder=project.path(engine,EDITOR_ASSETS,allocator=context.temp_allocator),prefix=EDITOR_ASSETS_PREFIX}) + mounts:=[1]asset.Registry_Mount{{folder=project.path(engine,EDITOR_ASSETS,allocator=context.temp_allocator),prefix=EDITOR_ASSETS_PREFIX}} + store_root:=options.store!=""?project.path_absolute(options.store,context.temp_allocator):store.default_root(context.temp_allocator) windowed:=!options.headless app,app_error:=host.app_create({ info=info,root=root,engine_root=engine,threads=jobs.THREADS_AUTO, @@ -429,10 +434,10 @@ open :: proc(info:^core.World_Info,args:[]string) -> (s:^Session,code:int) { window_desc=window,vsync=windowed,fixed_dt=windowed?0:HEADLESS_DT, // Without a window the mixer plays to nowhere, in real time: what play.record mixes is there. audio=windowed || !options.no_gpu?.Device:.None,audio_driver=windowed?"":"dummy", - assets_root=os.is_dir(assets)?assets:"",assets_imported=project.imported_path(root,context.temp_allocator),assets_cache=project.cache_path(root,project.REGISTRY_CACHE,allocator=context.temp_allocator), + assets_root=os.is_dir(assets)?assets:"",assets_cache=project.cache_path(root,project.REGISTRY_CACHE,allocator=context.temp_allocator), animation_cache=project.cache_path(root,project.ANIMATION_CACHE,allocator=context.temp_allocator), collision_cache=project.content_cache_path(root,project.COLLISION_CACHE,allocator=context.temp_allocator), - asset_mounts=mounts[:],shaders_root=root, + asset_mounts=mounts[:],shaders_root=root,store_root=store_root, gpu_late_record=project.path(root,".mirai",GPU_LATE_RECORD,allocator=context.temp_allocator) if has_project else "", }) if app==nil { @@ -452,10 +457,15 @@ open :: proc(info:^core.World_Info,args:[]string) -> (s:^Session,code:int) { s.project_root=strings.clone(root) s.engine_root=strings.clone(engine) program,_:=os.get_executable_path(context.temp_allocator) - s.program,_=strings.replace_all(program,"\\","/") + // replace_all gives back its input when nothing changes: the clone owns it either way. + forward,_:=strings.replace_all(program,"\\","/",context.temp_allocator) + s.program=strings.clone(forward) s.engine_stamp=engine_stamp_here(s) s.library_stamp=library_stamp_here(s) s.assets_root=strings.clone(assets) + if store_root!="" { + if opened,error:=store.open(store_root);error==.None {s.store=opened} else {log.warnf("the store at %s does not open (%v): nothing imports until the editor starts with --store ",store_root,error)} + } s.has_project=has_project if has_project {s.build.sources=code_fingerprint(root)} s.headless=options.headless @@ -1236,6 +1246,8 @@ session_destroy :: proc(s:^Session) { s.recipes=nil console_destroy(s.console) settings_engine_free(s) + if s.store.root!="" {store.close(&s.store)} + s.store={} delete(s.project_root);delete(s.engine_root);delete(s.program);delete(s.engine_stamp);delete(s.library_stamp);delete(s.assets_root);delete(s.capture);delete(s.gpu_lost_before);delete(s.start_documents);delete(s.start_select) for arg in s.args {delete(arg)} delete(s.args) @@ -1260,6 +1272,7 @@ options_read :: proc(args:[]string) -> (o:Options,ok:bool) { case "--project": o.project=value(args,&i) or_return case "--scene": o.scene=value(args,&i) or_return case "--engine": o.engine=value(args,&i) or_return + case "--store": o.store=value(args,&i) or_return case "--capture": o.capture=value(args,&i) or_return case "--documents": o.documents=value(args,&i) or_return case "--gpu-lost": o.gpu_lost=value(args,&i) or_return @@ -1319,7 +1332,7 @@ options_passed :: proc(args:[]string) -> []string { for i:=0;i.import) keeps its settings and the id of each -// asset, and importing it again gives the same assets; what an import builds -// that nobody edits is not a file of the asset folder but of the imported -// folder of the project, which mirai build brings up to date with the -// sources (sources.sync). The Import button of the Assets tab and File > -// Import ask for the file; a file dropped on the window is imported the same -// way. +// Import from the content browser: a source file or folder anywhere on disk +// is ingested into the library of the machine where it is, nothing copied, +// and its package is pinned in the project (sources.project_ingest), as mirai +// import does, with the formats and whole sources of plugins too +// (Import_Format_Provider, Import_Source_Provider). Its package is listed in +// the folder of the browser, or for a source in the assets, in its own +// folder: its meshes, textures, sounds and fonts are blobs of the store the +// registry lists there, and its documents (materials, skeletons, clips, the +// prefab of a model) are files of that folder. Importing it again gives the +// same assets. The Import button of the Assets tab and File > Import ask for +// the file; a file dropped on the window is imported the same way, and Add +// from library pins a package the library has (library.odin). // // The import runs in a thread of its own, with its own job system, while the // editor goes on; the content browser shows the dot of what works. When it -// ends, between frames, the registry looks again at what it wrote, what was -// loaded from the files it replaced is read again and, when it wrote Surface -// programs, the programs of the project are published, so its materials -// show. One import runs at a time, and while it does nothing else writes -// assets (import_busy). +// ends, between frames, the registry lists the packages again and looks at +// what it wrote, what was loaded of it is read again and, when it wrote +// Surface programs, the programs of the project are published, so its +// materials show. One import runs at a time, and while it does nothing else +// writes assets (import_busy). It is no edit: undo does not take it back. DIALOG_IMPORT :: u32(0x4d494d50) DIALOG_AI_REFERENCE :: u32(0x4d414952) @@ -66,24 +62,30 @@ Importer :: struct { profiled:bool, // the last one was profiled: the publication it asked for is a zone too } -// An import in a thread of its own. Its memory is that of allocator, which +// An import in a thread of its own: of a source (source), or the add of a +// package of the library (manifest). Its memory is that of allocator, which // any thread can call. Import_Job :: struct { thread:^thread.Thread, - source:string, - kind:sources.Source_Kind, - formats:[]importer.Format, // of a model: the engine's, then plugin_formats' + source:string, // the source, or the name of the package added + root,folder:string, // the project or plugin it is pinned in, and the folder of its assets it is listed in + prefix:string, // of that root in the registry: "" for the project + manifest:store.Hash, // of the package added; zero for a source + assets:[]base.Asset_Id, // of the package added, those it uses; none: all + settings:importer.Import_Settings, + settled:bool, // settings replace those its package keeps + claimed:bool, // a plugin imports the source whole + imports:sources.Library_Imports, // what the plugins give, prepared on the thread of the editor plugin_formats:[]Import_Format_Provider, // with the format.data their prepare gave - whole:Source_Importer, // of a source a plugin imports whole; run is nil otherwise - state:rawptr, // what whole.prepare gave - options:importer.Import_Options, + wholes:[]Source_Importer, // with the state their prepare gave, in imports.wholes + store:store.Store, // the session's + options:sources.Library_Options, allocator:mem.Allocator, started:time.Tick, profiled:bool, profile:base.Profile, // its zones, when profiled // What the thread leaves once done is set. report:importer.Import_Report, - scene:string, error:importer.Error, seconds:f64, done:bool, @@ -128,19 +130,30 @@ import_place :: proc(s:^Session) -> string { return s.content!=nil?string(s.content.place[:s.content.place_length]):"" } -// Starts importing a source file; false, with the reason shown, when it -// cannot. A source in the assets of the project or of a plugin imports where -// it is; one from outside the project is brought first into folder, a folder -// of the assets (asset_folder_clean; none is the root), and imports there: -// it is versioned with its sidecar beside what it makes, and its meshes, -// textures, sounds and fonts go to the imported folder. A source a plugin -// imports whole is brought by its importer with the files it reads -// (Source_Importer.bring), any other is copied alone. A source imported -// before writes the assets its sidecar lists, wherever they are. profiled -// records the zones of the import (importer.profile_totals) for -// asset.import_status. +// The root of the registry folder is of, the project or a plugin, and +// where the folder is under its assets; false for a folder of the editor or +// of the engine, which pins nothing. +@(private) +import_root_of :: proc(s:^Session,r:^asset.Registry,folder:string) -> (root:Library_Root,rest:string,ok:bool) { + for mount in r.mounts { + if folder!=mount.prefix && !asset_path_inside(folder,mount.prefix) {continue} + if strings.has_prefix(mount.prefix,".") || filepath.base(mount.folder)!=project.FOLDER_NAMES[.Assets] {return} + rest=folder[min(len(mount.prefix)+1,len(folder)):] + return {filepath.dir(mount.folder),mount.prefix},rest,true + } + if s.project_root=="" {return} + return {s.project_root,""},folder,true +} + +// Starts importing a source file or folder from anywhere on disk; false, +// with the reason shown, when it cannot. It is ingested into the library +// where it is and its package is pinned in the project, or the plugin whose +// folder folder is, listed in folder, a folder of the assets +// (asset_folder_clean; none is the root, or for a source in the assets, its +// own folder). profiled records the zones of the import +// (importer.profile_totals) for asset.import_status. // rig, when given, has the rig, rig_crotch and rig_neck that replace the -// source's settings, which a model inside the project keeps for its next import. +// settings its package keeps for its next import. // image, when given, is the use of an image that replaces its settings the same // way; without it, its settings say, else its name (sources.image_use). // skeleton, when not zero, is the skeleton of the project the clips of a model @@ -160,45 +173,49 @@ import_file :: proc(s:^Session,source:string,folder:="",profiled:=false,rig:Mayb import_say(s,"%s",busy) return false } - name:=filepath.base(source) - source:=source + if s.store.root=="" { + import_say(s,"The editor has no store to keep imports in: start it with --store ") + return false + } + if importer.unsafe_path(source) || !os.exists(source) { + import_say(s,"%s does not exist: give the path of a file or a folder on disk",source) + return false + } + origin:=importer.path_key(source) + name:=filepath.base(origin) + place:=folder + if place=="" { + if relative,inside:=asset.registry_relative(&a.registry,origin);inside {place=content_folder_of(relative)} + } + root,rest,rooted:=import_root_of(s,&a.registry,place) + if !rooted { + import_say(s,"%s is a folder of the editor or the engine: import into a folder of the project or of a plugin",place) + return false + } // A source a plugin imports whole goes before the formats, which may read its bytes too. - whole,code,claimed:=import_whole(s,source) - if _,in_assets:=sources.source_root(source,context.temp_allocator);!in_assets { - for at:=filepath.dir(project.path_absolute(source,context.temp_allocator));filepath.dir(at)!=at;at=filepath.dir(at) { - if !project.path_same(at,s.project_root) {continue} - import_say(s,"%s is in the project but in no assets folder: move it under assets/, beside what it makes, and import it there",name) - return false - } - into:=asset.registry_file_path(&a.registry,folder,context.temp_allocator) - if claimed && whole.bring!=nil { - brought,bring_error:=whole.bring(source,into,context.temp_allocator) - if bring_error.code!=.None { - import_say(s,"%s: %s",name,bring_error.message) - return false - } - if _,inside:=sources.source_root(brought,context.temp_allocator);!inside { - import_say(s,"%s: %s brought it to %s, which is in no assets folder",name,whole.name,brought) - return false - } - source=brought - } else { - copied:=fmt.tprintf("%s/%s",into,name) - if os.exists(copied) { - import_say(s,"%s is already in %s: import that one, or rename the file to import",name,folder==""?"the assets":folder) - return false - } - if os.copy_file(copied,source)!=nil { - import_say(s,"%s could not be copied into %s",name,folder==""?"the assets":folder) - return false - } - source=copied - } + whole,code,claimed:=import_whole(s,origin) + kind:=sources.Source_Kind.None if claimed else sources.source_kind(origin,import_formats(s)) + switch { + case kind==.None && !claimed: + import_say(s,"%s is not a file the editor imports",name) + return false + case rig!=nil && kind!=.Model: + import_say(s,"%s is not a model: only a model takes a rig",name) + return false + case skeleton!={} && kind!=.Model: + import_say(s,"%s is not a model: only the clips of a model go on a skeleton",name) + return false + case front!=nil && kind!=.Model: + import_say(s,"%s is not a model: only a model has a front",name) + return false + case image!=nil && kind!=.Image: + import_say(s,"%s is not an image: only an image imported on its own takes a use; the textures of a model take theirs from its materials",name) + return false } - if changed:=import_changed(s,source);len(changed)>0 && !discard { + if changed:=import_changed(s,root.root,origin);len(changed)>0 && !discard { if ask && s.gui!=nil { modal_close(s.gui) - base.text_set(s.gui.import_source[:],source) + base.text_set(s.gui.import_source[:],origin) append(&s.gui.import_changed,..changed) s.gui.modal=.Import_Changes return false @@ -213,92 +230,79 @@ import_file :: proc(s:^Session,source:string,folder:="",profiled:=false,rig:Mayb name,strings.join(paths,", ",context.temp_allocator),one?"it":"them",one?"that change":"those changes",one?"it":"them") return false } - root,_:=sources.source_root(source,context.temp_allocator) allocator:=runtime.heap_allocator() - options:=importer.Import_Options{ - root=strings.clone(root,allocator), - world=s.info, - texture_cache=project.content_cache_path(s.project_root,project.TEXTURE_CACHE,allocator=allocator), - mesh_cache=project.content_cache_path(s.project_root,project.MESH_CACHE,allocator=allocator), - fast_textures=true, - changed=.Discard if discard else .Refuse, - } - formats:=import_formats(s) - kind:=sources.Source_Kind.None if claimed else sources.source_kind(source,formats) - switch kind { - case .None: - case .Model: - _,rigged:=rig.? - _,faced:=front.? - if rigged || faced || skeleton!={} { - stored,_,settings_error:=importer.settings_load(source) - if settings_error.code!=.None { - import_say(s,"%s: %s",name,settings_error.message) - return false - } - options.settings,options.ignore_stored=stored,true - if wanted,rigged:=rig.?;rigged {options.rig,options.rig_crotch,options.rig_neck=wanted.rig,wanted.rig_crotch,wanted.rig_neck} - if skeleton!={} {options.skeleton=skeleton} - if stated,given:=front.?;given {options.front=stated} + job:=new(Import_Job,allocator) + job^={allocator=allocator,started=time.tick_now(),profiled=profiled,claimed=claimed} + job.store=s.store + job.source=strings.clone(origin,allocator) + job.root,job.prefix,job.folder=strings.clone(root.root,allocator),strings.clone(root.prefix,allocator),strings.clone(rest,allocator) + // The settings its package keeps, changed by what was asked: given only when they change it. + if kind==.Model || kind==.Image { + stored,found,settings_error:=importer.settings_load({store=&job.store},origin) + if settings_error.code!=.None { + import_job_free(job) + import_say(s,"%s: %s",name,settings_error.message) + return false } - case .Image: - options.image=sources.image_use(source) - if use,given:=image.?;given { - stored,_,settings_error:=importer.settings_load(source) - if settings_error.code!=.None { - import_say(s,"%s: %s",name,settings_error.message) - return false - } - options.settings,options.ignore_stored=stored,true - options.image=use + chosen:=stored + // An image with no settings yet holds what its name says. + if kind==.Image && !found {chosen.image=sources.image_use(origin)} + if use,given:=image.?;given {chosen.image=use} + if wanted,given:=rig.?;given {chosen.rig,chosen.rig_crotch,chosen.rig_neck=wanted.rig,wanted.rig_crotch,wanted.rig_neck} + if skeleton!={} {chosen.skeleton=skeleton} + if stated,given:=front.?;given {chosen.front=stated} + job.settings=chosen + job.settled=chosen!=stored || !found + } + import_plugins_prepare(s,job,kind==.Model,whole if claimed else {},code) + job.options.changed=.Discard if discard else .Refuse + import_job_start(s,job,"Importing %s",name) + return job.thread!=nil +} + +// The formats and the whole importers of the plugins job imports with, with +// what their prepare gives from the library: models takes the formats, +// whole the one importer that claims its source, with the identity of the +// code of its plugin. +@(private) +import_plugins_prepare :: proc(s:^Session,job:^Import_Job,models:bool,whole:Source_Importer,code:u64) { + job.imports.world=s.info + if models { + found:[PROVIDER_CAPACITY]Import_Format_Provider + plugins:=found[:provided_all(s,found[:])] + job.plugin_formats=slice.clone(plugins,job.allocator) + formats:=make([]importer.Format,len(plugins),job.allocator) + for &provider,i in job.plugin_formats { + if provider.prepare!=nil {provider.format.data=provider.prepare(&job.store,job.root,s.info,job.allocator)} + formats[i]=provider.format } - case .Sound,.Font,.Prompts: - } - job:=new(Import_Job,allocator) - job^={kind=kind,options=options,allocator=allocator,started=time.tick_now(),profiled=profiled} - job.source=strings.clone(source,allocator) - if kind==.Model {import_formats_prepare(s,job)} - if kind==.None && !claimed { - import_job_free(job) - import_say(s,"%s is not a file the editor imports",name) - return false - } - if _,given:=rig.?;given && kind!=.Model { - import_job_free(job) - import_say(s,"%s is not a model: only a model takes a rig",name) - return false - } - if skeleton!={} && kind!=.Model { - import_job_free(job) - import_say(s,"%s is not a model: only the clips of a model go on a skeleton",name) - return false - } - if _,given:=front.?;given && kind!=.Model { - import_job_free(job) - import_say(s,"%s is not a model: only a model has a front",name) - return false + job.imports.formats=formats } - if _,given:=image.?;given && kind!=.Image { - import_job_free(job) - import_say(s,"%s is not an image: only an image imported on its own takes a use; the textures of a model take theirs from its materials",name) - return false - } - if claimed { - job.whole=whole - job.options.importer_code=code - if whole.prepare!=nil {job.state=whole.prepare(&a.registry,s.info,allocator)} + if whole.run!=nil { + job.wholes=make([]Source_Importer,1,job.allocator) + job.wholes[0]=whole + wholes:=make([]sources.Whole_Source,1,job.allocator) + wholes[0]={name=whole.name,version=whole.version,claims=whole.claims,run=whole.run,code=code} + if whole.prepare!=nil {wholes[0].state=whole.prepare(&job.store,job.root,s.info,job.allocator)} + job.imports.wholes=wholes } + job.options=sources.Library_Options{store=&job.store,fast_textures=true,root=job.root,imports=&job.imports} +} + +// Starts the thread of job, which is the import that runs once started; it +// is freed when it cannot start, with the reason shown. +@(private) +import_job_start :: proc(s:^Session,job:^Import_Job,format:string,args:..any) { job.thread=thread.create_and_start_with_poly_data(job,import_run) if job.thread==nil { + import_say(s,"%s: no thread could be started to import it",filepath.base(job.source)) import_job_free(job) - import_say(s,"%s: no thread could be started to import it",name) - return false + return } s.importer.running=job s.importer.status={importing=true} - base.text_set(s.importer.status.path[:],source) - import_say(s,"Importing %s",name) - return true + base.text_set(s.importer.status.path[:],job.source) + import_say(s,format,..args) } // The import of job, in its thread: a job system of its own cooks on the @@ -313,36 +317,64 @@ import_run :: proc(job:^Import_Job) { defer if threaded {jobs.system_destroy(&js)} if threaded {job.options.worker={system=&js,worker=jobs.OWNER}} defer job.options.worker={} - if job.profiled && !importer.profile_begin(&job.profile,&job.options,job.allocator) {job.profiled=false} + if job.profiled { + timed:=importer.Import_Options{worker=job.options.worker} + if importer.profile_begin(&job.profile,&timed,job.allocator) {job.options.worker=timed.worker} else {job.profiled=false} + } + added:sources.Library_Report error:importer.Error - if job.whole.run!=nil { - // One store per value until odin-lang/Odin#7937 (see ODIN_WORKAROUNDS). - report,scene,run_error:=job.whole.run(job.state,job.source,job.options,job.allocator) - job.report=report - job.scene,error=scene,run_error + if job.manifest!={} { + added=sources.project_add(job.root,job.manifest,job.folder,job.assets,job.options,job.allocator) + import_added_report(job) } else { - report,import_error:=sources.import_source(job.source,job.options,job.formats,allocator=job.allocator) + settings:=&job.settings if job.settled else nil + // One store per value until odin-lang/Odin#7937 (see ODIN_WORKAROUNDS). + report,ingested,ingest_error:=sources.project_ingest(job.root,job.source,job.folder,job.options,settings,job.allocator) job.report=report - error=import_error + added=ingested + error=ingest_error + } + defer sources.library_report_destroy(&added) + if error.code==.None && len(added.failures)+len(added.missing)>0 { + error={.Write,strings.join(len(added.failures)>0?added.failures[:]:added.missing[:],"; ",context.temp_allocator)} } + for warning in added.warnings {append(&job.report.warnings,strings.clone(warning,job.allocator))} if error.code!=.None {job.error={error.code,strings.clone(error.message,job.allocator)}} job.seconds=time.duration_seconds(time.tick_since(job.started)) intrinsics.atomic_store(&job.done,true) } +// What the add of a package gives: its assets the project uses. +@(private) +import_added_report :: proc(job:^Import_Job) { + importer.report_init(&job.report,job.allocator) + p,error:=asset.package_get(&job.store,job.manifest,context.temp_allocator) + if error!=.None {return} + job.report.package_id,job.report.manifest=p.id,job.manifest + for &a in p.assets { + if len(job.assets)>0 && !slice.contains(job.assets,a.id) {continue} + append(&job.report.assets,importer.Written{id=a.id,kind=a.kind}) + } +} + @(private) import_job_free :: proc(job:^Import_Job) { allocator:=job.allocator if job.thread!=nil {thread.destroy(job.thread)} importer.report_destroy(&job.report) - delete(job.source,allocator);delete(job.scene,allocator) - delete(job.options.root,allocator);delete(job.options.texture_cache,allocator);delete(job.options.mesh_cache,allocator) + delete(job.source,allocator) + delete(job.root,allocator);delete(job.prefix,allocator);delete(job.folder,allocator) + delete(job.assets,allocator) for provider in job.plugin_formats { if provider.prepare!=nil && provider.release!=nil {provider.release(provider.format.data,allocator)} } delete(job.plugin_formats,allocator) - delete(job.formats,allocator) - if job.whole.release!=nil {job.whole.release(job.state,allocator)} + delete(job.imports.formats,allocator) + for source,i in job.wholes { + if source.prepare!=nil && source.release!=nil {source.release(job.imports.wholes[i].state,allocator)} + } + delete(job.wholes,allocator) + delete(job.imports.wholes,allocator) delete(job.error.message,allocator) if job.profiled {base.profile_destroy(&job.profile)} free(job,allocator) @@ -367,26 +399,20 @@ import_poll :: proc(s:^Session) { status.seconds=time.duration_seconds(time.tick_since(job.started)) if job.profiled {import_zones_take(s,profile)} } + a:=s.app.assets + // What its package lists is listed again, also after a failure that pinned it. + if a!=nil { + registry_zone:=base.profile_begin(profile,jobs.OWNER,"editor.registry") + library_relist(s,{job.root,job.prefix}) + base.profile_end(registry_zone) + } if job.error.code!=.None { import_say(s,"%s: %s",name,job.error.message) base.text_set(status.message[:],string(s.importer.message[:s.importer.message_length])) return } - a:=s.app.assets report:=&job.report if a!=nil { - registry_zone:=base.profile_begin(profile,jobs.OWNER,"editor.registry") - // The registry looks again at the files it wrote, and what was loaded from the ones it replaced is read again. - paths:=make([dynamic]string,0,len(report.assets)+1,context.temp_allocator) - written:=true - for entry in report.assets { - path,inside:=asset.registry_relative(&a.registry,entry.path) - if !inside {written=false;break} - append(&paths,path) - } - if path,inside:=asset.registry_relative(&a.registry,report.saved);inside {append(&paths,path)} - if written {host.assets_refresh(a,paths[:])} else {host.assets_scan(a)} - base.profile_end(registry_zone) materials_zone:=base.profile_begin(profile,jobs.OWNER,"editor.materials") for entry in report.assets { if entry.kind==asset.KIND_MATERIAL {material_rebase(s,entry.id)} @@ -402,23 +428,25 @@ import_poll :: proc(s:^Session) { base.profile_end(programs_zone) s.importer.publishing=time.tick_now() } - s.content.revision+=1 + if s.content!=nil {s.content.revision+=1} s.importer.imports+=1 status.ok=true for warning in report.warnings {append(&s.importer.warnings,strings.clone(warning))} + scene:="" for entry in report.assets { path:=entry.path if a!=nil { - if relative,inside:=asset.registry_relative(&a.registry,entry.path);inside {path=relative} + if file,found:=asset.registry_find(&a.registry,entry.id);found {path=file.path} } + if job.claimed && entry.kind==asset.KIND_SCENE && scene=="" {scene=path} append(&s.importer.written,Import_Written{id=entry.id,kind=entry.kind,kept=entry.kept,path=strings.clone(path)}) } switch { - case job.scene!="": - scene:=job.scene - if a!=nil {scene,_=asset.registry_relative(&a.registry,job.scene)} + case scene!="": base.text_set(status.scene[:],scene) import_say(s,"Imported %s: the scene %s and %d assets, %d warnings, in %.1f s",name,scene,len(report.assets)-1,len(report.warnings),job.seconds) + case job.manifest!={}: + import_say(s,"Added %s: %d assets",name,len(report.assets)) case: import_say(s,"Imported %s: %d assets",name,len(report.assets)) } @@ -426,14 +454,19 @@ import_poll :: proc(s:^Session) { for warning in report.warnings {log.warn(warning)} } -// The documents the import of a source wrote that changed since it wrote -// them (importer.outputs_changed), in the temp allocator; none for a source -// never imported, or whose sidecar the import itself refuses. +// The documents of the package the source at origin was ingested as that +// the project at root pins and that changed since an import or a refresh +// last wrote them (importer.documents_changed), in the temp allocator; none +// for a source never ingested or a package root does not pin. @(private) -import_changed :: proc(s:^Session,source:string) -> []base.Asset_Id { - sidecar,found,error:=importer.sidecar_load(source,context.temp_allocator) +import_changed :: proc(s:^Session,root,origin:string) -> []base.Asset_Id { + p,_,found,error:=importer.package_previous(&s.store,{},origin,context.temp_allocator) if !found || error.code!=.None {return nil} - return importer.outputs_changed(sidecar.outputs[:],&s.app.assets.registry,context.temp_allocator) + list,_,_:=asset.packages_load(root,context.temp_allocator) + for &entry in list.packages { + if entry.id==p.id {return importer.documents_changed(&p,entry.written[:],&s.app.assets.registry,context.temp_allocator)} + } + return nil } // Asks before importing a source again over documents of it that changed @@ -466,38 +499,6 @@ import_changes_modal :: proc(s:^Session) { } } -// The sidecar of the import an asset is the root of (the prefab of a model, -// the texture of an image, the sound of a sound file), and the registry files -// of what that import keeps in the imported folder; "" for any other asset. -// Deleting the root gives the import up: its sidecar goes with it, and with -// the sidecar what only the import can build. -@(private) -import_root :: proc(s:^Session,r:^asset.Registry,id:base.Asset_Id) -> (sidecar:string,cached:[]u32) { - file,found:=asset.registry_find(r,id) - if !found || s.project_root=="" {return} - types:asset.Types - asset.types_init(&types,context.temp_allocator) - defer asset.types_destroy(&types) - // Every imported asset names the source it was imported from, relative to the root it is of. - full:=asset.registry_file_path(r,file.path,context.temp_allocator) - _,info,imported,error:=asset.import_peek(full,&types) - if error!=.None || !imported {return} - source:=base.text_get(info.source[:]) - root,in_assets:=sources.source_root(full,context.temp_allocator) - if file.imported!=0 || !in_assets {root=s.project_root} - for mount in r.mounts { - if file.imported!=0 && strings.has_prefix(file.path,mount.prefix) {root=filepath.dir(mount.folder)} - } - source=project.path(root,source,allocator=context.temp_allocator) - listed,has,load_error:=importer.sidecar_load(source,context.temp_allocator) - if load_error.code!=.None || !has || len(listed.outputs)==0 || listed.outputs[0].id!=id {return} - files:=make([dynamic]u32,context.temp_allocator) - for output in listed.outputs { - if at,known:=r.by_id[output.id];known && r.files[at].imported!=0 {append(&files,at)} - } - return strings.concatenate({source,importer.SIDECAR_EXTENSION},context.temp_allocator),files[:] -} - // Waits for an import that runs and lets go of what the last one wrote: // when the editor ends. import_end :: proc(s:^Session) { @@ -560,35 +561,33 @@ import_working :: proc(s:^Session) -> bool { } // A format of models a plugin adds: its files import as those of the engine, -// in the editor and in the sync of every build (import_library). +// in the editor and in the refresh of every build (import_library). Import_Format_Provider :: struct { format:importer.Format, // Before a model is imported: the options of the format for it - // (Format.data) from the assets of the project and the library of its - // plugins, in allocator; nil keeps format.data. - prepare:proc(assets:^asset.Registry,info:^core.World_Info,allocator:mem.Allocator) -> rawptr, + // (Format.data), in allocator; nil keeps format.data. library: the store + // the packages of the root the import is for are in (asset.root_packages + // of root lists them), info: the types of its game. + prepare:proc(library:^store.Store,root:string,info:^core.World_Info,allocator:mem.Allocator) -> rawptr, release:proc(data:rawptr,allocator:mem.Allocator), // of what prepare gave, once the import ended } // A source a plugin imports whole with importer.Outputs, the one source of // every asset it makes (an Unreal map: a scene and the meshes, materials and // textures it names, the packages it reads being its inputs), in the editor -// and in the sync of every build (import_library). +// and in the refresh of every build (import_library). It is read where it is. Source_Importer :: struct { - name:string, // "Unreal maps": its filter in the file dialog and in messages + name:string, // "Unreal maps": its filter in the file dialog and in messages, and the importer its packages record + version:u16, // the format version its run gives importer.import_version: a build is found by it without the source extensions:[importer.FORMAT_EXTENSIONS]string, // lower case, with the dot, "" after the last claims:proc(path:string,head:[]u8) -> bool, // head: up to importer.SNIFF_BYTES from the start of the file - // Before the import starts: what run needs, from the assets of the - // project and the library of its plugins, in allocator; nil for nothing. - prepare:proc(assets:^asset.Registry,info:^core.World_Info,allocator:mem.Allocator) -> rawptr, + // Before the import starts: what run needs, in allocator; nil for + // nothing. library, root and info as Import_Format_Provider.prepare has them. + prepare:proc(library:^store.Store,root:string,info:^core.World_Info,allocator:mem.Allocator) -> rawptr, // In the thread of the import, with a job system of its own in // options.worker. scene: the scene it wrote, in allocator, "" for none. run:proc(state:rawptr,source:string,options:importer.Import_Options,allocator:mem.Allocator) -> (report:importer.Import_Report,scene:string,error:importer.Error), release:proc(state:rawptr,allocator:mem.Allocator), // of what prepare gave, once the import ended - // A source outside the assets: copies it with the files it reads into - // folder (absolute, in the assets) and gives where the source is now, in - // allocator. nil: the source is copied alone. - bring:proc(source,folder:string,allocator:mem.Allocator) -> (brought:string,error:importer.Error), } Import_Source_Provider :: struct {source:Source_Importer} @@ -603,27 +602,10 @@ import_formats :: proc(s:^Session) -> []importer.Format { return out[:] } -// The formats of the import of a model by job, with the options the plugins -// prepare for it. -@(private) -import_formats_prepare :: proc(s:^Session,job:^Import_Job) { - found:[PROVIDER_CAPACITY]Import_Format_Provider - plugins:=found[:provided_all(s,found[:])] - for &provider in plugins { - if provider.prepare!=nil {provider.format.data=provider.prepare(&s.app.assets.registry,s.info,job.allocator)} - } - formats:=make([dynamic]importer.Format,0,len(sources.FORMATS)+len(plugins),job.allocator) - append(&formats,..sources.FORMATS[:]) - for provider in plugins {append(&formats,provider.format)} - job.formats=formats[:] - job.plugin_formats=slice.clone(plugins,job.allocator) -} - // What import_library holds until its release. @(private="file") Import_Library_Held :: struct { library:host.Project_Library, - registry:asset.Registry, providers:[]Import_Format_Provider, formats:[]importer.Format, importers:[]Source_Importer, @@ -632,26 +614,30 @@ Import_Library_Held :: struct { } // The formats of models and the importers of whole sources the plugins of -// the project at root add, prepared for its assets, and the types of its +// the project at root add, prepared with the store s, and the types of its // game, from the library of its editor (sources.Library): what the import -// program of the engine syncs and imports its sources with. -import_library :: proc(root:string,allocator:mem.Allocator) -> sources.Library_Imports { +// program of the engine refreshes and imports its sources with. +import_library :: proc(root:string,s:^store.Store,allocator:mem.Allocator) -> (imports:sources.Library_Imports) { path:=project.project_library(root,"editor",context.temp_allocator) - if !os.is_file(path) {return {problem="the project has no built editor library, where the importers of its plugins and the types of its game are: mirai build editor builds it"}} + if !os.is_file(path) { + imports.problem="the project has no built editor library, where the importers of its plugins and the types of its game are: mirai build editor builds it" + return + } held:=new(Import_Library_Held,allocator) held.allocator=allocator // One store per value until odin-lang/Odin#7937 (see ODIN_WORKAROUNDS). library,error:=host.project_library_load(path,0) - if error!=.None {return {problem=host.library_error_text(error,path)}} + if error!=.None { + imports.problem=host.library_error_text(error,path) + return + } held.library=library - asset.registry_init(&held.registry,project.folder_path(root,.Assets,context.temp_allocator),"",allocator,project.imported_path(root,context.temp_allocator)) - asset.registry_scan(&held.registry) found:[PROVIDER_CAPACITY]Import_Format_Provider held.providers=slice.clone(found[:provided_in(&held.library.info,found[:])],allocator) held.formats=make([]importer.Format,len(held.providers),allocator) for provider,i in held.providers { held.formats[i]=provider.format - if provider.prepare!=nil {held.formats[i].data=provider.prepare(&held.registry,&held.library.info,allocator)} + if provider.prepare!=nil {held.formats[i].data=provider.prepare(s,root,&held.library.info,allocator)} } claimed:[PROVIDER_CAPACITY]Import_Source_Provider plugins:[PROVIDER_CAPACITY]string @@ -661,10 +647,14 @@ import_library :: proc(root:string,allocator:mem.Allocator) -> sources.Library_I engine:=project.engine_root_find("",context.temp_allocator) for provider,i in claimed[:count] { held.importers[i]=provider.source - held.wholes[i]={claims=provider.source.claims,run=provider.source.run,code=plugin_code(root,engine,plugins[i])} - if provider.source.prepare!=nil {held.wholes[i].state=provider.source.prepare(&held.registry,&held.library.info,allocator)} - } - return {formats=held.formats,wholes=held.wholes,world=&held.library.info,state=held,release=proc(state:rawptr) { + held.wholes[i]={name=provider.source.name,version=provider.source.version,claims=provider.source.claims,run=provider.source.run,code=plugin_code(root,engine,plugins[i])} + if provider.source.prepare!=nil {held.wholes[i].state=provider.source.prepare(s,root,&held.library.info,allocator)} + } + imports.formats=held.formats + imports.wholes=held.wholes + imports.world=&held.library.info + imports.state=held + imports.release=proc(state:rawptr) { held:=(^Import_Library_Held)(state) for provider,i in held.providers { if provider.prepare!=nil && provider.release!=nil {provider.release(held.formats[i].data,held.allocator)} @@ -672,9 +662,9 @@ import_library :: proc(root:string,allocator:mem.Allocator) -> sources.Library_I for source,i in held.importers { if source.prepare!=nil && source.release!=nil {source.release(held.wholes[i].state,held.allocator)} } - asset.registry_destroy(&held.registry) host.project_library_unload(&held.library) - }} + } + return } // The importer of a plugin that claims source whole, with the identity of diff --git a/editor/import_api_test.odin b/editor/import_api_test.odin index 2a56e15e7..dcc5fdeeb 100644 --- a/editor/import_api_test.odin +++ b/editor/import_api_test.odin @@ -1,14 +1,21 @@ package editor +import "base:intrinsics" import "core:fmt" import "core:os" +import "core:path/filepath" import "core:strings" import "core:testing" +import "core:thread" import "mirai:asset" +import img "mirai:cook/image" +import importer "mirai:cook/import" import "mirai:cook/import/sources" import "mirai:editor/api" import "mirai:host" +import "mirai:project" import "mirai:std/base" +import "mirai:store" // asset.import takes by name what an image holds and where a model stands // facing, and by its id the skeleton the clips of a model go on, and says @@ -54,9 +61,207 @@ asset_import_reads_a_relative_source_from_the_project :: proc(t:^testing.T) { testing.expect_value(t,asset_import_source(&s,"C:/elsewhere/walk.glb"),"C:/elsewhere/walk.glb") } -// A source whose import wrote a document that changed since is not imported -// again without a word: asset.import fails naming it and how to go on, and the -// import of the content browser asks, with the documents that changed. +// A project of the editor over a store of its own, as a session has them. +@(private="file") +Library_Test :: struct { + using files:Files_Test, + assets:host.Assets, + app:host.App, + importer:Importer, +} + +@(private="file") +library_test_open :: proc(t:^testing.T,l:^Library_Test,name,store_folder:string) -> bool { + files_test_open(t,&l.files,name) or_return + if !testing.expect(t,host.assets_init(&l.assets,{root=project.path(l.root,"assets",allocator=context.temp_allocator),capacity=16})) {return false} + l.app=host.App{assets=&l.assets} + l.s.app,l.s.importer=&l.app,&l.importer + opened,error:=store.open(store_folder) + l.s.store=opened + return testing.expect_value(t,error,store.Error.None) +} + +@(private="file") +library_test_close :: proc(l:^Library_Test) { + import_end(&l.s) + store.close(&l.s.store) + host.assets_destroy(&l.assets) + files_test_close(&l.files) +} + +// Waits for the import that runs and takes what it wrote, as the frames do. +@(private="file") +library_test_end :: proc(t:^testing.T,l:^Library_Test,loc:=#caller_location) -> bool { + job:=l.s.importer.running + if !testing.expect(t,job!=nil,"an import runs",loc=loc) {return false} + for !intrinsics.atomic_load(&job.done) {thread.yield()} + import_poll(&l.s) + status:=&l.s.importer.status + return testing.expectf(t,status.ok,"%s",base.text_get(status.message[:]),loc=loc) +} + +// The files under folder, by path. +@(private="file") +library_test_files :: proc(folder:string) -> []string { + out:=make([dynamic]string,context.temp_allocator) + todo:=make([dynamic]string,context.temp_allocator) + append(&todo,folder) + for next:=0;next []asset.Project_Package { + list,_,_:=asset.packages_load(root,context.temp_allocator) + return list.packages[:] +} + +// A PNG outside the project, as a download is. +@(private="file") +library_test_source :: proc(t:^testing.T,folder:string) -> string { + pixels:=make([]u8,8*8*4,context.temp_allocator) + for &p,at in pixels {p=u8(at*7)} + path:=fmt.tprintf("%s/downloads/sign.png",folder) + os.make_directory_all(filepath.dir(path)) + testing.expect_value(t,img.image_write(path,{pixels=pixels,width=8,height=8}),img.Error.None) + return path +} + +// Importing a source from outside the project ingests it where it is and +// pins its package: the project lists its texture from the store and copies +// nothing. Moving its root changes where the package is listed, and deleting +// it unpins the package and leaves the store and the source as they were. +@(test) +a_source_from_outside_is_pinned_listed_moved_and_unpinned_without_copies :: proc(t:^testing.T) { + l:=new(Library_Test,context.temp_allocator) + top:=base.test_folder(t,"editor-library") + if top=="" || !library_test_open(t,l,"editor-library-game",fmt.tprintf("%s/store",top)) {return} + defer library_test_close(l) + s:=&l.s + r:=&l.assets.registry + source:=library_test_source(t,top) + before,_:=os.read_entire_file(source,context.temp_allocator) + + if !testing.expect(t,import_file(s,source,"props"),string(s.importer.message[:s.importer.message_length])) || !library_test_end(t,l) {return} + pins:=library_test_pins(l.root) + if !testing.expect_value(t,len(pins),1) {return} + testing.expect_value(t,pins[0].folder,"props") + file,found:=asset.registry_find_path(r,"props/sign.png.tex") + if !found {file,found=asset.registry_find_path(r,"props/sign.tex")} + if !testing.expectf(t,found && file.listed,"the texture is listed from the package: %v",r.listed) {return} + id:=file.id + object,built:=asset.registry_listed_object(r,file.path) + testing.expect(t,built && store.chunk_has(&s.store,object),"the texture is an object of the store") + testing.expectf(t,len(library_test_files(project.path(l.root,"assets",allocator=context.temp_allocator)))==0,"nothing is copied into the assets") + after,_:=os.read_entire_file(source,context.temp_allocator) + testing.expect(t,string(before)==string(after),"the source is as it was") + + // Its root moves where the package is listed, keeping its name. + _,message:=asset_move(s,id,"props","other") + testing.expect(t,strings.has_prefix(message,"name:"),message) + path:string + path,message=asset_move(s,id,"textures","") + testing.expectf(t,message=="" && strings.has_prefix(path,"textures/"),"%s %s",path,message) + testing.expect_value(t,library_test_pins(l.root)[0].folder,"textures") + moved,_:=asset.registry_find(r,id) + testing.expect(t,moved!=nil && moved.path==path) + + // Deleting it unpins the package: the store and the source stay. + gone,refused:=asset_delete(s,id) + testing.expectf(t,refused=="" && len(gone)==1 && gone[0]==path,"%v %s",gone,refused) + testing.expect_value(t,len(library_test_pins(l.root)),0) + _,listed:=asset.registry_find(r,id) + testing.expect(t,!listed,"what it listed left the project") + testing.expect(t,store.chunk_has(&s.store,object),"the object stays in the store") + testing.expect(t,os.is_file(source),"the source outside the project stays") +} + +// A package of the library added to a second project lists the blobs the +// first one does; packages.list and packages.add say for each hostile +// argument what failed and how to fix it. +@(test) +a_package_of_the_library_goes_to_a_second_project_with_the_same_blobs :: proc(t:^testing.T) { + top:=base.test_folder(t,"editor-library-add") + if top=="" {return} + kept:=fmt.tprintf("%s/store",top) + first:=new(Library_Test,context.temp_allocator) + if !library_test_open(t,first,"editor-library-first",kept) {return} + defer library_test_close(first) + source:=library_test_source(t,top) + if !testing.expect(t,import_file(&first.s,source,"")) || !library_test_end(t,first) {return} + pin:=library_test_pins(first.root)[0] + file:^asset.Registry_File + for &f in first.assets.registry.files { + if f.listed {file=&f} + } + if !testing.expect(t,file!=nil) {return} + blob:=strings.clone(asset.registry_file_path(&first.assets.registry,file.path,context.temp_allocator),context.temp_allocator) + + second:=new(Library_Test,context.temp_allocator) + if !library_test_open(t,second,"editor-library-second",kept) {return} + defer library_test_close(second) + s:=&second.s + call :: proc(s:^Session,run:proc(s:^Session,call:^api.Call),params:^$P,$R:typeid) -> api.Call { + method:=api.Method{params_id=P,result_id=R} + call:=api.Call{method=&method,params=params,result=new(R,context.temp_allocator),allocator=context.temp_allocator} + run(s,&call) + call.method,call.params=nil,nil + return call + } + listing:Api_Packages_List_Params + listed:=call(s,api_packages_list,&listing,Api_Packages_List_Result) + result:=(^Api_Packages_List_Result)(listed.result) + if !testing.expectf(t,!listed.failed && result!=nil && len(result.packages)==1,"%s",listed.error.message) {return} + p:=result.packages[0] + testing.expect(t,p.id==pin.id && !p.pinned && len(p.assets)==1 && p.assets[0].built && p.assets[0].bytes>0 && p.assets[0].kind=="texture") + + Hostile :: struct {name:string,assets:[]string,folder,says:string} + buffer:[base.ASSET_ID_TEXT]u8 + named:=strings.clone(base.asset_id_format(pin.id,buffer[:]),context.temp_allocator) + hostile:=[?]Hostile{ + {"",nil,"","packages.list"}, + {"nothing such",nil,"","packages.list"}, + {named,{"zz"},"","packages.list"}, + {named,{"0123456789abcdef0123456789abcdef"},"","packages.list"}, + {named,nil,"../out","folder:"}, + {named,nil,".editor/x","folder:"}, + } + for h in hostile { + params:=Api_Packages_Add_Params{name=h.name,assets=h.assets,folder=h.folder} + answer:=call(s,api_packages_add,¶ms,Import_Status_Result) + testing.expectf(t,answer.failed && answer.error.code==api.INVALID_PARAMS && strings.contains(answer.error.message,h.says),"%v: %s",h,answer.error.message) + testing.expect(t,s.importer.running==nil,"nothing starts") + } + params:=Api_Packages_Add_Params{name=named,folder="shared"} + added:=call(s,api_packages_add,¶ms,Import_Status_Result) + if !testing.expectf(t,!added.failed,"%s",added.error.message) || !library_test_end(t,second) {return} + pins:=library_test_pins(second.root) + testing.expect(t,len(pins)==1 && pins[0].id==pin.id && pins[0].folder=="shared") + there,found:=asset.registry_find(&second.assets.registry,file.id) + testing.expect(t,found && strings.has_prefix(there.path,"shared/") && asset.registry_file_path(&second.assets.registry,there.path,context.temp_allocator)==blob,"the same blob") + + // Without a store nothing lists or imports, and each says how to give one. + store.close(&s.store) + without:=call(s,api_packages_list,&listing,Api_Packages_List_Result) + testing.expect(t,without.failed && strings.contains(without.error.message,"--store"),without.error.message) + testing.expect(t,!import_file(s,source,"")) + testing.expect(t,strings.contains(string(s.importer.message[:s.importer.message_length]),"--store")) + opened,_:=store.open(kept) + s.store=opened + testing.expect(t,!import_file(s,fmt.tprintf("%s/none.png",top),"")) + testing.expect(t,strings.contains(string(s.importer.message[:s.importer.message_length]),"does not exist")) +} + +// A source whose import wrote a document of the project that changed since is +// not imported again without a word: asset.import fails naming it and how to +// go on, and the import of the content browser asks, with the documents that +// changed. @(test) importing_over_a_changed_document_asks_first :: proc(t:^testing.T) { root:=base.test_folder(t,"import-changed") @@ -72,17 +277,32 @@ importing_over_a_changed_document_asks_first :: proc(t:^testing.T) { _,_=asset.scene_entity_add(&scene,asset.scene_new_id(&scene),{},"Added") testing.expect_value(t,asset.scene_save(&scene,fmt.tprintf("%s/hall.scene",folder)),asset.Error.None) asset.scene_destroy(&scene) - source:=fmt.tprintf("%s/hall.level",folder) - _=os.write_entire_file(source,"LEVEL") - outputs:=make([dynamic]asset.Import_Output,context.temp_allocator) - append(&outputs,asset.Import_Output{kind=asset.KIND_SCENE,name="hall",id=hall,written=1}) - _=os.write_entire_file(fmt.tprintf("%s.import",source),asset.sidecar_text("{}",{outputs=outputs},context.temp_allocator)) + source:=fmt.tprintf("%s/hall.gltf",folder) + _=os.write_entire_file(source,"{}") testing.expect_value(t,asset.registry_scan(&assets.registry),asset.Error.None) + // The package the source was ingested as, whose scene the pin says was written as something else. + kept,opened:=store.open(fmt.tprintf("%s/store",root)) + if !testing.expect_value(t,opened,store.Error.None) {return} + defer store.close(&kept) + p:=asset.Package{id={5,5},name="hall",origin={path="assets/hall.gltf",importer="gltf"}} + p.assets=make([dynamic]asset.Package_Asset,context.temp_allocator) + append(&p.assets,asset.Package_Asset{kind=asset.KIND_SCENE,name="hall",id=hall}) + manifest,put_error:=asset.package_put(&kept,&p) + if !testing.expect_value(t,put_error,asset.Error.None) {return} + name:[64]u8 + testing.expect_value(t,store.ref_set(&kept,asset.REF_ORIGINS,asset.origin_name(importer.path_key(source),&name),manifest),store.Error.None) + list:asset.Packages + list.packages=make([dynamic]asset.Project_Package,context.temp_allocator) + entry:=asset.Project_Package{id=p.id,manifest=manifest} + entry.written=make([dynamic]asset.Package_Written,context.temp_allocator) + append(&entry.written,asset.Package_Written{hall,1}) + append(&list.packages,entry) + testing.expect_value(t,asset.packages_save(root,&list),asset.Error.None) imports:Importer gui:Editor_Ui defer delete(gui.import_changed) - s:=Session{app=&host.App{assets=&assets},importer=&imports,gui=&gui} + s:=Session{app=&host.App{assets=&assets},importer=&imports,gui=&gui,project_root=root,store=kept} params:Import_Params base.text_set(params.path[:],source) method:=api.Method{params_id=Import_Params} @@ -91,5 +311,5 @@ importing_over_a_changed_document_asks_first :: proc(t:^testing.T) { testing.expectf(t,call.failed && strings.contains(call.error.message,"hall.scene") && strings.contains(call.error.message,"discard_changes"),"%s",call.error.message) testing.expect(t,gui.modal==.None,"a call asks nobody") testing.expect(t,!import_file(&s,source,ask=true) && imports.running==nil,"nothing is imported") - testing.expect(t,gui.modal==.Import_Changes && len(gui.import_changed)==1 && gui.import_changed[0]==hall && base.text_get(gui.import_source[:])==source,"the user is asked, with what changed") + testing.expect(t,gui.modal==.Import_Changes && len(gui.import_changed)==1 && gui.import_changed[0]==hall && base.text_get(gui.import_source[:])==importer.path_key(source),"the user is asked, with what changed") } diff --git a/editor/library.odin b/editor/library.odin new file mode 100644 index 000000000..ce815b3ca --- /dev/null +++ b/editor/library.odin @@ -0,0 +1,350 @@ +package editor + +import "base:runtime" +import "core:fmt" +import "core:log" +import "core:slice" +import "core:strings" +import "core:time" +import "mirai:asset" +import "mirai:cook/import/sources" +import "mirai:editor/api" +import "mirai:host" +import "mirai:std/base" +import "mirai:std/serial" +import "mirai:store" +import "mirai:ui" + +// The library of the machine in the editor: the packages the store of the +// session keeps (cook/import/sources/library.odin) and what a root of the +// editor, the project or a plugin, pins of them in its packages.mirai. +// Importing ingests a source where it is and pins its package (import.odin); +// packages.list lists the packages of the library with their assets, and +// packages.add, or a package of a library root of the content browser carried +// onto a folder (content_library.odin), pins one or some +// of its assets in a folder of the project, as an import that runs in its +// thread. Deleting the root of a package unpins it and moving it moves where +// the package is listed (asset_delete.odin, asset_move.odin); none of them +// removes a blob of the store or touches a source outside the project. + +// A root of the registry: the project (prefix "") or a plugin mounted under prefix. +Library_Root :: struct {root,prefix:string} + +// The package a root pins that gives an asset: its entry in the list of the +// root and its manifest, in the temp allocator. +@(private) +Library_Pin :: struct { + root:Library_Root, + entry:asset.Project_Package, + p:asset.Package, +} + +// A package of the library, as packages.list tells it. +@(private) +Library_Package :: struct { + id:base.Asset_Id, + manifest:store.Hash, + name,origin:string, + pinned:bool, // the project pins it + folder:string, // where the project lists it, when pinned + assets:[]Library_Asset, + root:store.Hash, // the object its card shows (asset.build_shown) in the build of this machine; zero: none + shown:base.Asset_Id, // the asset of that object + objects:[]store.Hash, // of every asset of that build, sorted +} + +@(private) +Library_Asset :: struct { + id:base.Asset_Id, + name:string, + kind:asset.Fourcc, + bytes:i64, // of its file in the build of this machine; 0 when not built + built:bool, +} + +// Lists again what the packages of root list, from the store of the session, +// and scans the registry. What they cannot list is told in the log. +@(private) +library_relist :: proc(s:^Session,root:Library_Root) { + r:=files_registry(s) + if r==nil {return} + packages,error:=asset.root_packages(root.root,s.store.root,context.temp_allocator) + if error!=.None { + log.warnf("the packages of %s could not be listed: %v",root.root,error) + return + } + for problem in packages.problems {log.warnf("assets: %s",problem)} + if !asset.registry_listed_set(r,root.prefix,packages.listed) {log.warnf("the packages of %s could not be listed: no memory",root.root)} + host.assets_scan(s.app.assets) +} + +// The package a root pins that gives the asset of file, with its manifest, +// for what a package lists (by the package the registry lists it from) and +// for a scene, which may be the root of one; false for any other asset, an +// asset no package gives, or without the store. +@(private) +library_pin_of :: proc(s:^Session,r:^asset.Registry,file:^asset.Registry_File) -> (pin:Library_Pin,found:bool) { + if s.store.root=="" || !file.listed && file.kind!=asset.KIND_SCENE {return} + root,_,rooted:=import_root_of(s,r,content_folder_of(file.path)) + if !rooted {return} + // What a package lists names it: only its manifest is read. + listed:base.Asset_Id + if file.listed { + entries:=r.listed + for mount in r.mounts { + if mount.prefix==root.prefix && root.prefix!="" {entries=mount.listed} + } + for &l in entries { + if l.id==file.id {listed=l.package_id} + } + if listed=={} {return} + } + list,_,error:=asset.packages_load(root.root,context.temp_allocator) + if error!=.None {return} + for &entry in list.packages { + if listed!={} && entry.id!=listed {continue} + p,read_error:=asset.package_get(&s.store,entry.manifest,context.temp_allocator) + if read_error!=.None || p.id!=entry.id {continue} + for &a in p.assets { + if a.id==file.id {return {root,entry,p},true} + } + } + return +} + + +// Where the root of a package is in the registry, or its id when it is not there. +@(private) +library_root_named :: proc(r:^asset.Registry,p:^asset.Package) -> string { + root:=asset.package_root(p) + if file,found:=asset.registry_find(r,root);found {return file.path} + buffer:[base.ASSET_ID_TEXT]u8 + return fmt.tprintf("%s (%s)",p.name,base.asset_id_format(root,buffer[:])) +} + +// Pins entry again in its root, with what changed of it, and lists again. +@(private) +library_repin :: proc(s:^Session,root:Library_Root,entry:asset.Project_Package) -> string { + if error:=sources.project_pin(root.root,&s.store,entry);error!=.None { + return fmt.tprintf("%s/%s could not be written (%v): check that it is not read-only, then try again",root.root,asset.PACKAGES_FILE,error) + } + library_relist(s,root) + return "" +} + +// The packages of the library of the session, by name, with whether the +// root at root pins them. In allocator. +@(private) +library_list :: proc(s:^Session,root:string,allocator:=context.allocator) -> (list:[]Library_Package,problem:string) { + if s.store.root=="" {return nil,"The editor has no store, where the library keeps its packages: start it with --store "} + refs,error:=store.ref_list(&s.store,asset.REF_PACKAGES,context.temp_allocator) + if error!=.None && error!=.Missing {return nil,fmt.tprintf("The packages of the store at %s could not be listed (%v): check that the folder reads",s.store.root,error)} + pins:asset.Packages + if root!="" {pins,_,_=asset.packages_load(root,context.temp_allocator)} + out:=make([dynamic]Library_Package,0,len(refs),allocator) + for ref in refs { + p,read_error:=asset.package_get(&s.store,ref.hash,allocator) + if read_error!=.None {continue} + entry:=Library_Package{id=p.id,manifest=ref.hash,name=p.name,origin=p.origin.path} + for &pinned in pins.packages { + if pinned.id==p.id {entry.pinned,entry.folder=true,strings.clone(pinned.folder,allocator)} + } + outputs:=make(map[base.Asset_Id]asset.Build_Output,context.temp_allocator) + digits:[64]u8 + if built,ref_error:=store.ref_get(&s.store,asset.REF_BUILDS,store.hex(asset.build_select(ref.hash,&p),&digits));ref_error==.None { + if b,build_error:=asset.build_get(&s.store,built,context.temp_allocator);build_error==.None && b.package_id==p.id { + for o in b.outputs {outputs[o.id]=o} + } + } + assets:=make([]Library_Asset,len(p.assets),allocator) + for &a,at in p.assets { + assets[at]={id=a.id,name=a.name,kind=a.kind} + if o,found:=outputs[a.id];found && o.kind==a.kind && asset.object_whole(&s.store,o.object) { + assets[at].built=true + dependencies:=make([dynamic]base.Asset_Id,context.temp_allocator) + if header,header_error:=asset.object_header(&s.store,o.object,&dependencies);header_error==.None {assets[at].bytes=i64(header.total_size)} + } + } + entry.assets=assets + entry.root,entry.objects,entry.shown,_=library_build_objects(s,ref.hash,&p,{},allocator) + append(&out,entry) + } + slice.sort_by(out[:],proc(a,b:Library_Package) -> bool {return a.name string { + r:=files_registry(s) + if r==nil {return "The project has no assets folder to add into"} + if busy:=import_busy(s);busy!="" {return busy} + clean,refused:=asset_folder_clean(r,folder) + if refused!="" {return fmt.tprintf("folder: %s",refused)} + root,rest,rooted:=import_root_of(s,r,clean) + if !rooted {return fmt.tprintf("folder: %s is a folder of the editor or the engine: give a folder of the project or of a plugin",clean)} + packages,problem:=library_list(s,root.root,context.temp_allocator) + if problem!="" {return problem} + wanted:=strings.trim_space(named) + if wanted=="" {return "name: give the id or the name of a package, as packages.list lists them"} + chosen:=-1 + id,is_id:=serial.text_parse_asset_id(wanted) + for &p,at in packages { + if !(is_id && p.id==id || !is_id && p.name==wanted) {continue} + if chosen>=0 {return fmt.tprintf("name: more than one package is named %q: give its id, as packages.list lists them",wanted)} + chosen=at + } + if chosen<0 {return fmt.tprintf("name: the library at %s has no package %q: packages.list lists them, and asset.import ingests a source",s.store.root,wanted)} + p:=&packages[chosen] + used:=make([dynamic]base.Asset_Id,0,len(ids),context.temp_allocator) + for text in ids { + asset_id,parsed:=serial.text_parse_asset_id(strings.trim_space(text)) + known:=false + for &a in p.assets {known||=parsed && a.id==asset_id} + if !known {return fmt.tprintf("assets: %q is no asset of the package %s: give the ids packages.list lists for it, or none for all",text,p.name)} + if !slice.contains(used[:],asset_id) {append(&used,asset_id)} + } + allocator:=runtime.heap_allocator() + job:=new(Import_Job,allocator) + job^={allocator=allocator,manifest=p.manifest,started=time.tick_now()} + job.store=s.store + job.source=strings.clone(p.name,allocator) + job.root,job.prefix,job.folder=strings.clone(root.root,allocator),strings.clone(root.prefix,allocator),strings.clone(rest,allocator) + job.assets=slice.clone(used[:],allocator) + import_plugins_prepare(s,job,true,{},0) + import_job_start(s,job,"Adding %s",p.name) + if job.thread==nil {return string(s.importer.message[:s.importer.message_length])} + return "" +} + +// ---------------------------------------------------------------- packages.list, packages.add + +Api_Packages_List_Params :: struct { + name:string `mirai:"optional,label=Only the packages whose name holds it, in any case; empty: all"`, +} + +Api_Packages_List_Result :: struct { + store:string `mirai:"label=The folder of the store the library keeps its packages in"`, + packages:[]Api_Package, +} +Api_Package :: struct { + id:base.Asset_Id, + name:string, + source:string `mirai:"label=Where its source was when it was ingested on this machine"`, + pinned:bool `mirai:"label=The project pins it"`, + folder:string `mirai:"optional,label=Of the assets, where the project lists it, when pinned"`, + thumbnail:string `mirai:"label=The key of the small thumbnail of its root; empty: it has no build here"`, + thumbnail_here:bool `mirai:"label=The store of this machine keeps that thumbnail"`, + assets:[]Api_Package_Asset, +} +Api_Package_Asset :: struct { + id:base.Asset_Id, + name:string `mirai:"label=In the source"`, + kind:string, + bytes:i64 `mirai:"label=Of its file in the build of this machine; 0 when not built"`, + built:bool `mirai:"label=This machine has a build of it this engine reads"`, +} + +Api_Packages_Add_Params :: struct { + name:string `mirai:"label=The package: its id or its name, as packages.list lists them"`, + assets:[]string `mirai:"optional,label=Of the package, the ids of the assets the project uses, with what they depend on; none: all"`, + folder:string `mirai:"optional,label=The folder under the asset folder it is listed in; none: the root of the assets"`, + seconds:int `mirai:"optional,min=1,max=7200,unit=s,label=How long it waits for the end of the add; past that it answers importing and the add goes on"`, +} + +@(private) +api_packages_list :: proc(s:^Session,call:^api.Call) { + params:=api.call_params(call,Api_Packages_List_Params) + packages,problem:=library_list(s,s.project_root,call.allocator) + if problem!="" { + api.call_fail(call,api.SERVER_ERROR,problem) + return + } + wanted:=strings.to_lower(strings.trim_space(params.name),context.temp_allocator) + out:=make([dynamic]Api_Package,0,len(packages),call.allocator) + for &p in packages { + if wanted!="" && !strings.contains(strings.to_lower(p.name,context.temp_allocator),wanted) {continue} + assets:=make([]Api_Package_Asset,len(p.assets),call.allocator) + for a,at in p.assets { + kind:="other" + if info,known:=asset.kind_info(a.kind);known {kind=info.name} + assets[at]={a.id,a.name,kind,a.bytes,a.built} + } + entry:=Api_Package{id=p.id,name=p.name,source=p.origin,pinned=p.pinned,folder=p.folder,assets=assets} + entry.thumbnail,entry.thumbnail_here=api_thumbnail_of(s,p.root,p.objects,call.allocator) + append(&out,entry) + } + result:=api.call_result(call,Api_Packages_List_Result) + result.store=strings.clone(s.store.root,call.allocator) + result.packages=out[:] +} + +@(private) +api_packages_add :: proc(s:^Session,call:^api.Call) { + params:=api.call_params(call,Api_Packages_Add_Params) + if problem:=library_add(s,params.name,params.assets,params.folder);problem!="" { + api.call_fail(call,api.INVALID_PARAMS,problem) + return + } + importing :: proc(s:^Session,user:rawptr) -> bool {return s.importer.running!=nil} + api_wait(s,f64(params.seconds),nil,importing,"adding") + api_import_status_answer(s,call) +} + +// Of the build this machine chose for a package: the object of its asset id +// (none: the one it shows, asset.build_shown) and those of all its assets, +// sorted, as thumbnail_package_key takes them. +@(private) +library_build_objects :: proc(s:^Session,manifest:store.Hash,p:^asset.Package,id:base.Asset_Id,allocator:=context.temp_allocator) -> (object:store.Hash,objects:[]store.Hash,shown:base.Asset_Id,built:bool) { + digits:[64]u8 + hash,ref_error:=store.ref_get(&s.store,asset.REF_BUILDS,store.hex(asset.build_select(manifest,p),&digits)) + if ref_error!=.None {return} + b,error:=asset.build_get(&s.store,hash,context.temp_allocator) + if error!=.None || b.package_id!=p.id {return} + objects=make([]store.Hash,len(b.outputs),allocator) + shown=id + if id=={} {object,shown=asset.build_shown(&b,p)} + for o,k in b.outputs { + objects[k]=o.object + if id!={} && o.id==id {object=o.object} + } + slice.sort_by(objects,proc(a,b:store.Hash) -> bool { + a,b:=a,b + return string(a[:]) (name:string,here:bool) { + if object=={} {return} + key:=thumbnail_package_key(.Small,thumbnail_programs(s),object,objects) + buffer:[16]u8 + name=strings.clone(thumbnail_name(key,&buffer),allocator) + _,error:=store.ref_get(&s.store,THUMBNAIL_NAMESPACE,name) + return name,error==.None +} + +// The manifest and package of this machine's library a listed output of the registry comes from. +@(private) +library_package_of_output :: proc(s:^Session,r:^asset.Registry,id:base.Asset_Id) -> (manifest:store.Hash,p:asset.Package,found:bool) { + package_id:base.Asset_Id + lists:=make([dynamic][]asset.Package_Listed,context.temp_allocator) + append(&lists,r.listed) + for mount in r.mounts {append(&lists,mount.listed)} + search: for list in lists { + for &l in list { + if l.id==id {package_id=l.package_id;break search} + } + } + if package_id=={} || s.store.root=="" {return} + buffer:[base.ASSET_ID_TEXT]u8 + hash,error:=store.ref_get(&s.store,asset.REF_PACKAGES,base.asset_id_format(package_id,buffer[:])) + if error!=.None {return} + read,read_error:=asset.package_get(&s.store,hash,context.temp_allocator) + if read_error!=.None || read.id!=package_id {return} + return hash,read,true +} diff --git a/editor/player_processes.odin b/editor/player_processes.odin index eee44b7c7..82c908a53 100644 --- a/editor/player_processes.odin +++ b/editor/player_processes.odin @@ -109,6 +109,7 @@ player_process_command :: proc(s:^Session,player:string,args:[]string,k:int,disp tiles:=p.first+p.count command:=make([dynamic]string,context.temp_allocator) append(&command,player,"--attached") + if s.store.root!="" {append(&command,"--store",s.store.root)} append(&command,..args) if p.embedded { pixels:=player_pixels(s,player_tile(player_region(s),tiles,p.first+k)) diff --git a/editor/presence_menu.odin b/editor/presence_menu.odin index aea61b8e1..a7416366e 100644 --- a/editor/presence_menu.odin +++ b/editor/presence_menu.odin @@ -123,7 +123,7 @@ presence_menu_items :: proc(s:^Session,menu:Presence_Menu) { case peer.playing: state=" plays alone" } header(ctx,fmt.tprintf("%s%s",name,state)) - posed:=peer.has_pose && wire.host_time(&st.host)-peer.seen<=COLLAB_PRESENCE_SEEN + posed:=peer.has_pose && collab_now(st)-peer.seen<=COLLAB_PRESENCE_SEEN if ui.menu_item(ctx,ui.id(ctx,"go"),"Go to",{icon=.Target,disabled=!posed || !editing}) { forward,_,_:=orbit_basis(peer.orbit) presence_go_to(s,orbit_position(peer.orbit),presence_yaw(forward)) @@ -139,7 +139,7 @@ presence_menu_items :: proc(s:^Session,menu:Presence_Menu) { case .Remote_Agent: st:=collab_state(s) if st==nil || st.phase!=.Joined {ui.popup_close(ctx);return} - agents:=collab_remote_agents(s.ai,st,menu.peer,wire.host_time(&st.host)) + agents:=collab_remote_agents(s.ai,st,menu.peer,collab_now(st)) if menu.agent>=len(agents) {ui.popup_close(ctx);return} agent:=agents[menu.agent] header(ctx,fmt.tprintf("%s of %s",string(agent.name[:agent.name_length]),collab_peer_name(s,menu.peer))) @@ -186,7 +186,7 @@ collab_peer_at :: proc(s:^Session,point:[2]f32) -> (peer:int,rect:ui.Rect,found: if st==nil || st.phase!=.Joined || s.mode!=.Edit {return} _,_,high,loaded:=presence_mesh(s.app.assets) if !loaded {return} - now:=wire.host_time(&st.host) + now:=collab_now(st) for &other,i in st.peers { if i==int(st.self) || !other.used || !other.has_pose || now-other.seen>COLLAB_PRESENCE_SEEN {continue} r,shown:=presence_glyph_rect(s,presence_shown(&st.figures[i],orbit_position(other.orbit))+{0,high.y,0},s.viewport.view_projection) diff --git a/editor/projects.odin b/editor/projects.odin index 8c4ea1bee..80adb1324 100644 --- a/editor/projects.odin +++ b/editor/projects.odin @@ -121,6 +121,8 @@ Projects :: struct { starts:bool, starting:bool, // the build starts once the dialog of the opening was drawn start_frame:u64, + join:[64]u8, // the invitation of a session the editor started on it joins (collab_project.odin) + join_name:[COLLAB_NAME_BYTES]u8, // and the name it joins with } // Shows the screen, with the list as the file of the engine has it now. @@ -350,7 +352,7 @@ projects_build_start :: proc(s:^Session) { built^=opened p.built=built p.shaders=build.shaders_start(built) - p.job=build.job_start({project=built,target=.Editor,dev=build.ENGINE_MODE_HERE==.Dev,engine=build.ENGINE_MODE_HERE,runs=engine_folder_here(s)}) + p.job=build.job_start({project=built,target=.Editor,dev=build.ENGINE_MODE_HERE==.Dev,engine=build.ENGINE_MODE_HERE,runs=engine_folder_here(s),store_root=s.store.root}) log.infof("building %s to open it",absolute) } @@ -378,7 +380,8 @@ projects_poll :: proc(s:^Session) { p.install=nil p.starting,p.start_frame=true,s.frame } - if p!=nil && p.starting && s.frame>p.start_frame+1 {projects_build_start(s)} + // Without a window no dialog is waited for: a manual editor draws no frame by itself. + if p!=nil && p.starting && (s.frame>p.start_frame+1 || s.headless) {projects_build_start(s)} if p==nil || p.job==nil {return} if p.shaders!=nil && !build.shaders_poll(p.shaders) {return} if !build.job_poll(p.job) {return} @@ -424,6 +427,10 @@ projects_start :: proc(s:^Session) -> bool { if remote:=strings.trim_space(string(p.remote[:p.remote_length]));remote!="" {append(&command,"--git-remote",remote)} } for arg in s.args {append(&command,arg)} + if code:=base.text_get(p.join[:]);code!="" { + append(&command,"--join",code) + if name:=base.text_get(p.join_name[:]);name!="" {append(&command,"--name",name)} + } turn:=gpu_turn_pass(s,&command) preferences_save(s) started,error:=subprocess.start({command=command[:],stdin=os.stdin,stdout=os.stdout,stderr=os.stderr},inherit=turn) diff --git a/editor/shell.odin b/editor/shell.odin index 7fd21cabb..7321d14f7 100644 --- a/editor/shell.odin +++ b/editor/shell.odin @@ -95,7 +95,8 @@ RUN_COMMANDS := [Run_Kind]string{.Play="play",.Simulate="simulate",.Play_From_Ca // Rectangles of the last frame, in window points; no size: not shown. Shell_Rects :: struct {bar,left,right,bottom:ui.Rect} -Drag_Kind :: enum u8 {None,Preset,Asset,Entity} +// Package: a package of a library of the content browser (content_library.odin). +Drag_Kind :: enum u8 {None,Preset,Asset,Entity,Package} // Something carried from a panel: to the scene, to a row of the outliner, to a field. Drag :: struct { @@ -106,6 +107,7 @@ Drag :: struct { asset_id:base.Asset_Id, asset_kind:asset.Fourcc, part:[64]u8, // Asset: the clip of a library it carries; empty: the whole asset + library:i32, // Package: the peer of its library +1; 0: this machine's entity:core.Entity, press:[2]f32, moved:bool, // it travelled: a drag, not a click @@ -626,7 +628,7 @@ drag_icon :: proc(s:^Session,d:^Drag) -> (ui.Icon,ui.Color) { case .None: return .None,{} case .Preset: return d.icon,d.color case .Entity: return .Group,GROUP_COLOR - case .Asset: return asset_icon(s,d.asset_kind) + case .Asset,.Package: return asset_icon(s,d.asset_kind) } return .None,{} } diff --git a/editor/thumbnails.odin b/editor/thumbnails.odin index 9b2e700f6..fb7fb25cb 100644 --- a/editor/thumbnails.odin +++ b/editor/thumbnails.odin @@ -2,6 +2,7 @@ package editor import "core:fmt" import "core:hash" +import "core:math" import "core:mem" import "core:os" import "core:slice" @@ -15,6 +16,7 @@ import "mirai:std/base" import "mirai:std/geom" import mm "mirai:std/math" import "mirai:std/serial" +import "mirai:store" import "mirai:ui" // Thumbnails of geometry: images with premultiplied alpha that the renderer @@ -26,13 +28,19 @@ import "mirai:ui" // content browser at its largest at a density of 2. One is drawn per frame, // and the icon stands in until it is there. // -// The preview of an asset of the project is kept in the thumbs folder of its -// caches, a file per asset and size, under a key of what it is drawn from: -// the content of the asset and of everything it needs, as the registry has -// it, the shaders the renderer runs and THUMBNAIL_RENDERER. A file with -// another key is a preview to draw again, as when the asset or one of its -// dependencies changes, or a program does. Up to -// THUMBNAIL_LOADS are read per frame, besides the one drawn. +// The preview of an asset is derived data of the machine's store: an object +// under THUMBNAIL_NAMESPACE named by a key of what it is drawn from, so it is +// drawn once per machine and a peer of a session gives it to another +// (content_library.odin). The key of an asset of a package is that of its +// object and of every object of the build it is in (thumbnail_package_key), +// which any machine that has the manifest and the build computes without the +// asset's bytes; that of an asset of the project's own files is the content of +// the asset and of everything it needs, as the registry has it +// (thumbnail_store_key). Both take the shaders the renderer runs and +// THUMBNAIL_RENDERER: a change of either is a preview to draw again. Up to +// THUMBNAIL_LOADS are read per frame, besides the one drawn. The view of a +// scene the editor took is no derived data: it stays in the thumbs folder of +// the project (thumbnail_subjects.odin). Thumbnail_Level :: enum u8 {Small,Large} @@ -56,6 +64,7 @@ THUMBNAIL_LOADS :: 8 THUMBNAIL_FILE_MAGIC :: u32('M')|u32('T')<<8|u32('H')<<16|u32('B')<<24 THUMBNAIL_FILE_VERSION :: 3 THUMBNAIL_FILE_HEADER :: 24 // magic, version, key, side and the bytes of the stream of the image after it +THUMBNAIL_NAMESPACE :: asset.REF_CACHE_PREFIX+"thumbnails" // : the preview, as its file holds it // A stored preview as its file holds it, and the coders of its image. Thumbnail_File :: struct { @@ -115,10 +124,10 @@ Thumbnail_Cell :: struct { signature:u64, // of what its image was drawn from seen:u64, // the last frame it was asked for ready:bool, // false: its subject had nothing to draw - stored:u64, // the key in the thumbs folder of the image it holds; 0: none + stored:u64, // the key in the store of the image it holds; 0: none wanted:u64, // the key there of the subject whose signature is checked checked:u64, - absent:u64, // a key the thumbs folder did not have + absent:u64, // a key the store did not have } @(storage) @@ -126,7 +135,7 @@ Thumbnails :: struct { pixels:[dynamic]u8, // the atlas; allocated with the first preview readback:[dynamic]u8, // what the renderer drew, before it is reduced image:[dynamic]u8, // the preview being drawn - file:Thumbnail_File, // a preview of the thumbs folder, as its file holds it + file:Thumbnail_File, // a preview as the store or the thumbs folder holds it loads:int, // previews looked for there in the frame loaded loaded:u64, cells:[THUMBNAIL_CELLS]Thumbnail_Cell, @@ -150,7 +159,7 @@ Thumbnails :: struct { Thumbnail_Picture :: struct { slot:int, key,signature:u64, // of the cell when it asked; key 0: none asks - wanted:u64, // its key in the thumbs folder; 0: not kept there + wanted:u64, // its key in the store; 0: not kept there stored:base.Asset_Id, subject:Thumbnail_Subject, } @@ -216,7 +225,8 @@ thumbnail_key :: proc(named:[]byte) -> u64 { preview_request :: proc(s:^Session,key:u64,subject:^Thumbnail_Subject,points:f32,stored:base.Asset_Id={}) -> ui.Quad { p:=s.thumbnails a:=s.app.assets - if key==0 || a==nil || a.renderer==nil {return {}} + // A texture is read on the processor; the others are drawn. + if key==0 || a==nil || a.renderer==nil && subject.provider!=asset.KIND_TEXTURE {return {}} level:=Thumbnail_Level.Large if points*max(s.app.density,1)>THUMBNAIL_SIZE else .Small key:=key if level==.Large {key=thumbnail_key(mem.ptr_to_bytes(&key))} @@ -248,12 +258,11 @@ preview_request :: proc(s:^Session,key:u64,subject:^Thumbnail_Subject,points:f32 cell:=&p.cells[slot] cell.seen=frame if cell.signature==signature || s.edits.open {return thumbnail_quad(cell,slot)} - folder:=thumbnail_folder(s) if stored!={} else "" - if folder!="" && cell.checked!=signature { - programs:=base.hash_combine(base.hash64(transmute([]u8)string(gpu.SHADERS_INTERFACE)),s.app.shaders.identity) - cell.wanted,cell.checked=thumbnail_store_key(&a.registry,stored,subject^,level,programs),signature + kept:=stored!={} && s.store.root!="" + if kept && cell.checked!=signature { + cell.wanted,cell.checked=thumbnail_key_of(s,stored,subject^,level),signature } - wanted:=cell.wanted if folder!="" else 0 + wanted:=cell.wanted if kept else 0 sides:=THUMBNAIL_SIDES side:=sides[level] if wanted!=0 && cell.stored==wanted && cell.ready { @@ -264,7 +273,7 @@ preview_request :: proc(s:^Session,key:u64,subject:^Thumbnail_Subject,points:f32 if wanted!=0 && cell.absent!=wanted { if p.loads>=THUMBNAIL_LOADS {return thumbnail_quad(cell,slot)} p.loads+=1 - if thumbnail_load(folder,stored,side,wanted,&p.file,&p.image) { + if thumbnail_store_load(&s.store,side,wanted,&p.file,&p.image) { thumbnail_put(p,slot,p.image[:]) cell.signature,cell.stored,cell.ready=signature,wanted,true return thumbnail_quad(cell,slot) @@ -276,6 +285,17 @@ preview_request :: proc(s:^Session,key:u64,subject:^Thumbnail_Subject,points:f32 return thumbnail_quad(cell,slot) } if p.drawn==frame+1 {return thumbnail_quad(cell,slot)} + if subject.provider==asset.KIND_TEXTURE { + // Read on the processor from its smallest levels, not drawn. + p.drawn=frame+1 + cell.signature,cell.stored=signature,0 + resize(&p.image,side*side*4) + texture:base.Asset_Id + mem.copy(&texture,&subject.data[0],size_of(texture)) + cell.ready=thumbnail_texture_image(s,texture,side,p.image[:]) + thumbnail_drawn(s,slot,side,wanted) + return thumbnail_quad(cell,slot) + } if provider,found:=thumbnail_provider(s,subject.provider);subject.provider!=0 && found && provider.image!=nil { // Drawn between frames: the cell shows what it had until then. if p.picture.key!=0 {return thumbnail_quad(cell,slot)} @@ -288,7 +308,7 @@ preview_request :: proc(s:^Session,key:u64,subject:^Thumbnail_Subject,points:f32 cell.signature,cell.stored=signature,0 resize(&p.image,side*side*4) cell.ready=preview_render(s,subject,level,p.image[:]) - thumbnail_drawn(p,slot,side,folder,stored,wanted) + thumbnail_drawn(s,slot,side,wanted) return thumbnail_quad(cell,slot) } @@ -307,7 +327,7 @@ preview_load_turn :: proc(s:^Session,id:base.Asset_Id) -> bool { } // The image of the cell its provider asked to draw between frames -// (Thumbnail_Picture), kept in the thumbs folder. Between frames. +// (Thumbnail_Picture), kept in the store. Between frames. @(private) thumbnail_picture_draw :: proc(s:^Session) { p:=s.thumbnails @@ -323,17 +343,18 @@ thumbnail_picture_draw :: proc(s:^Session) { resize(&p.image,side*side*4) cell.stored=0 cell.ready=provider.image(s,&picture.subject,side,p.image[:]) - thumbnail_drawn(p,picture.slot,side,thumbnail_folder(s) if picture.wanted!=0 else "",picture.stored,picture.wanted) + thumbnail_drawn(s,picture.slot,side,picture.wanted) } // Puts the image of side pixels just drawn into its cell, a blank one when -// nothing covered it, and keeps it in the thumbs folder under wanted. +// nothing covered it, and keeps it in the store under wanted. @(private="file") -thumbnail_drawn :: proc(p:^Thumbnails,slot,side:int,folder:string,stored:base.Asset_Id,wanted:u64) { +thumbnail_drawn :: proc(s:^Session,slot,side:int,wanted:u64) { + p:=s.thumbnails cell:=&p.cells[slot] if !cell.ready {mem.zero_slice(p.image[:])} thumbnail_put(p,slot,p.image[:]) - if cell.ready && wanted!=0 && folder!="" && thumbnail_save(folder,stored,side,wanted,p.image[:],&p.file) {cell.stored=wanted} + if cell.ready && wanted!=0 && s.store.root!="" && thumbnail_store_save(&s.store,side,wanted,p.image[:],&p.file) {cell.stored=wanted} } @(private="file") @@ -428,7 +449,7 @@ thumbnail_file :: proc(folder:string,id:base.Asset_Id,side:int) -> string { return fmt.tprintf("%s/%s-%d.thumb",folder,base.asset_id_format(id,buffer[:]),side) } -// The previews of assets that no file is any more leave the thumbs folder, +// The views of scenes that no file is any more leave the thumbs folder, // as the registry notes them (Registry.removed): deleted, also while the editor // was closed, or another asset now. One that changed keeps its file, which its // next drawing overwrites. @@ -436,12 +457,12 @@ thumbnail_file :: proc(folder:string,id:base.Asset_Id,side:int) -> string { thumbnails_forget :: proc(s:^Session) { a:=s.app.assets if a==nil || len(a.registry.removed)==0 {return} - if folder:=thumbnail_folder(s);folder!="" {thumbnail_store_forget(folder,a.registry.removed[:])} + if folder:=thumbnail_folder(s);folder!="" {thumbnail_views_forget(folder,a.registry.removed[:])} clear(&a.registry.removed) } @(private) -thumbnail_store_forget :: proc(folder:string,ids:[]base.Asset_Id) { +thumbnail_views_forget :: proc(folder:string,ids:[]base.Asset_Id) { sides:=THUMBNAIL_SIDES for id in ids { for side in sides {os.remove(thumbnail_file(folder,id,side))} @@ -455,7 +476,15 @@ thumbnail_load :: proc(folder:string,id:base.Asset_Id,side:int,key:u64,file:^Thu pixels:=side*side*4 resize(&file.bytes,THUMBNAIL_FILE_HEADER+asset.pack_bound(pixels)) head,size,error:=serial.file_read_head(thumbnail_file(folder,id,side),file.bytes[:]) - if error!=.None || size!=i64(len(head)) || len(head) bool { + pixels:=side*side*4 + if len(head) bool { - if len(image)!=side*side*4 {return false} - if file.encoder.data==nil && asset.stream_encoder_init(&file.encoder)!=.None {return false} + bytes:=thumbnail_encode(side,key,image,file) + if bytes==nil {return false} + if !os.is_dir(folder) && os.make_directory_all(folder)!=nil {return false} + return serial.file_write_atomic(thumbnail_file(folder,id,side),bytes)==.None +} + +// The bytes of a preview of side under key, in file.bytes; nil when it cannot be packed. +@(private) +thumbnail_encode :: proc(side:int,key:u64,image:[]u8,file:^Thumbnail_File) -> []u8 { + if len(image)!=side*side*4 {return nil} + if file.encoder.data==nil && asset.stream_encoder_init(&file.encoder)!=.None {return nil} resize(&file.bytes,THUMBNAIL_FILE_HEADER+asset.pack_bound(len(image))) packed,error:=asset.pack(&file.encoder,.Fast,.Open,image,file.bytes[THUMBNAIL_FILE_HEADER:]) - if error!=.None {return false} + if error!=.None {return nil} w:serial.Writer serial.writer_init(&w,allocator=context.temp_allocator) serial.write_u32(&w,THUMBNAIL_FILE_MAGIC) @@ -481,8 +519,80 @@ thumbnail_save :: proc(folder:string,id:base.Asset_Id,side:int,key:u64,image:[]u serial.write_u32(&w,u32(side)) serial.write_u32(&w,u32(packed)) copy(file.bytes[:THUMBNAIL_FILE_HEADER],serial.writer_bytes(&w)) - if !os.is_dir(folder) && os.make_directory_all(folder)!=nil {return false} - return serial.file_write_atomic(thumbnail_file(folder,id,side),file.bytes[:THUMBNAIL_FILE_HEADER+packed])==.None + return file.bytes[:THUMBNAIL_FILE_HEADER+packed] +} + +// The name of the preview of key in THUMBNAIL_NAMESPACE. +thumbnail_name :: proc(key:u64,buffer:^[16]u8) -> string { + digits:=transmute([]u8)string("0123456789abcdef") + DIGITS:=digits + for k in 0..<16 {buffer[k]=DIGITS[(key>>(4*uint(15-k)))&15]} + return string(buffer[:]) +} + +// The preview of side the store keeps under key, into image: false when it has none. +@(private) +thumbnail_store_load :: proc(s:^store.Store,side:int,key:u64,file:^Thumbnail_File,image:^[dynamic]u8) -> bool { + if s==nil || s.root=="" || key==0 {return false} + buffer:[16]u8 + address,ref_error:=store.ref_get(s,THUMBNAIL_NAMESPACE,thumbnail_name(key,&buffer)) + if ref_error!=.None {return false} + o,open_error:=store.object_open(s,address) + if open_error!=.None {return false} + defer store.object_close(&o) + size:=int(store.object_size(&o)) + if size>THUMBNAIL_FILE_HEADER+asset.pack_bound(side*side*4) {return false} + resize(&file.bytes,size) + if store.object_read(&o,0,file.bytes[:])!=.None {return false} + return thumbnail_decode(file.bytes[:],side,key,file,image) +} + +// Keeps the preview of side in the store under key; false when it cannot. +@(private) +thumbnail_store_save :: proc(s:^store.Store,side:int,key:u64,image:[]u8,file:^Thumbnail_File) -> bool { + bytes:=thumbnail_encode(side,key,image,file) + if bytes==nil || key==0 {return false} + address,error:=store.put(s,bytes) + if error!=.None {return false} + buffer:[16]u8 + return store.ref_set(s,THUMBNAIL_NAMESPACE,thumbnail_name(key,&buffer),address)==.None +} + +// The key of the preview of an asset of a package at level: its object and +// every object of the build it is in (objects), with the shaders and the +// renderer. Any machine with the manifest and the build has them, not the asset. +thumbnail_package_key :: proc(level:Thumbnail_Level,programs:u64,object:store.Hash,objects:[]store.Hash) -> u64 { + object:=object + key:=base.hash_combine(base.hash_combine(base.hash_combine(THUMBNAIL_RENDERER,u64(level)),programs),0x706b67) + key=base.hash_combine(key,base.hash64(object[:])) + for &o in objects {key=base.hash_combine(key,base.hash64(o[:]))} + return key|1 +} + +// The programs previews are drawn with, as their keys take them. +@(private) +thumbnail_programs :: proc(s:^Session) -> u64 { + return base.hash_combine(base.hash64(transmute([]u8)string(gpu.SHADERS_INTERFACE)),s.app.shaders.identity) +} + +// The key of the stored preview of an asset: of a package, by its objects; +// of the project's own files, by what the registry has of it. +@(private) +thumbnail_key_of :: proc(s:^Session,id:base.Asset_Id,subject:Thumbnail_Subject,level:Thumbnail_Level) -> u64 { + r:=&s.app.assets.registry + programs:=thumbnail_programs(s) + if file,found:=asset.registry_find(r,id);found && file.listed && !subject.captured { + if pin,pinned:=library_pin_of(s,r,file);pinned { + object,objects,_,built:=library_build_objects(s,pin.entry.manifest,&pin.p,id) + if built {return thumbnail_package_key(level,programs,object,objects)} + } + // An output of this machine's library that no root pins (content_library.odin mounts them). + if manifest,p,known:=library_package_of_output(s,r,id);known { + object,objects,_,built:=library_build_objects(s,manifest,&p,id) + if built {return thumbnail_package_key(level,programs,object,objects)} + } + } + return thumbnail_store_key(r,id,subject,level,programs) } // The image of a cell in the atlas, which goes to the renderer with the next flush. @@ -677,6 +787,65 @@ preview_geometry :: proc(s:^Session,shape:host.Shape,mesh:base.Asset_Id,points:f return preview_request(s,thumbnail_key(mem.ptr_to_bytes(&named)),&subject,points,mesh) } +// The stored preview of a material: the sphere of the studio wearing it. +@(private) +preview_material :: proc(s:^Session,material:base.Asset_Id,points:f32=0) -> ui.Quad { + subject:=Thumbnail_Subject{mesh={shape=.Sphere},tint={color={1,1,1,1}}} + subject.mesh.materials[0]=material + named:=[3]u64{material[0],material[1],0x6d6174} + return preview_request(s,thumbnail_key(mem.ptr_to_bytes(&named)),&subject,points,material) +} + +// The stored preview of a texture: its smallest levels that cover the cell, +// read on the processor (thumbnail_texture_image), never the whole texture. +@(private) +preview_texture :: proc(s:^Session,texture:base.Asset_Id,points:f32=0) -> ui.Quad { + subject:=Thumbnail_Subject{provider=asset.KIND_TEXTURE} + id:=texture + copy(subject.data[:],mem.ptr_to_bytes(&id)) + named:=[3]u64{texture[0],texture[1],0x746578} + return preview_request(s,thumbnail_key(mem.ptr_to_bytes(&named)),&subject,points,texture) +} + +// The image of a texture at side*side, premultiplied display colors, from the +// largest of its levels no larger than side (or its smallest), decoded alone. +@(private) +thumbnail_texture_image :: proc(s:^Session,texture:base.Asset_Id,side:int,out:[]u8) -> bool { + a:=s.app.assets + file,found:=asset.registry_find(&a.registry,texture) + if !found {return false} + bytes,read_error:=asset.registry_read(&a.registry,file.path,context.temp_allocator) + if read_error!=.None {return false} + opened:asset.Asset + if asset.asset_open(&opened,bytes)!=.None {return false} + defer asset.asset_close(&opened) + view,view_error:=asset.texture_open(&opened,&a.types) + if view_error!=.None || len(view.mips)==0 {return false} + level:=len(view.mips)-1 + for mip,at in view.mips { + if int(max(mip.width,mip.height))<=side {level=at;break} + } + if asset.texture_decode(&opened,&a.types,&view,level)!=.None {return false} + mip:=view.mips[level] + width,height:=int(mip.width),int(mip.height) + texels:=make([][4]f32,width*height,context.temp_allocator) + if !asset.texture_level_linear(view.info.format,view.info.flags,asset.texture_layer(view,level,0),width,height,texels) {return false} + srgb:=.Srgb in view.info.flags + for y in 0.. ui.Quad { diff --git a/editor/thumbnails_test.odin b/editor/thumbnails_test.odin index f9bdb6766..3810f2277 100644 --- a/editor/thumbnails_test.odin +++ b/editor/thumbnails_test.odin @@ -11,6 +11,7 @@ import "mirai:host" import gpu "mirai:render" import "mirai:std/base" import "mirai:std/serial" +import "mirai:store" import "mirai:ui" @(private="file") PROGRAMS :: u64(0x7072) @@ -37,6 +38,9 @@ stored_thumbnails_last_until_their_asset_or_what_it_needs_changes :: proc(t:^tes texture_write(t,&types,fmt.tprintf("%s/textures/wood.tex",assets),texture,10) material_write(t,&types,fmt.tprintf("%s/wood.mat",assets),material,texture,0.5) material_write(t,&types,fmt.tprintf("%s/other.mat",assets),other,{},0.5) + st,opened:=store.open(fmt.tprintf("%s/store",root)) + if !testing.expect_value(t,opened,store.Error.None) {return} + defer store.close(&st) r:asset.Registry;asset.registry_init(&r,assets,fmt.tprintf("%s/.mirai/cache/registry",root));defer asset.registry_destroy(&r) testing.expect_value(t,asset.registry_scan(&r),asset.Error.None) @@ -54,27 +58,27 @@ stored_thumbnails_last_until_their_asset_or_what_it_needs_changes :: proc(t:^tes file:Thumbnail_File read:[dynamic]u8 defer {thumbnail_file_destroy(&file);delete(read)} - testing.expect(t,thumbnail_save(folder,material,THUMBNAIL_SIZE,key,image,&file)) - testing.expect(t,thumbnail_load(folder,material,THUMBNAIL_SIZE,key,&file,&read) && slice.equal(read[:],image)) - testing.expect(t,!thumbnail_load(folder,material,THUMBNAIL_LARGE,key,&file,&read)) - testing.expect(t,!thumbnail_load(folder,other,THUMBNAIL_SIZE,key,&file,&read)) + testing.expect(t,thumbnail_store_save(&st,THUMBNAIL_SIZE,key,image,&file)) + testing.expect(t,thumbnail_store_load(&st,THUMBNAIL_SIZE,key,&file,&read) && slice.equal(read[:],image)) + testing.expect(t,!thumbnail_store_load(&st,THUMBNAIL_LARGE,key,&file,&read)) + testing.expect(t,!thumbnail_store_load(&st,THUMBNAIL_SIZE,key+2,&file,&read)) // Saving what the asset needs gives another key: the stored preview is drawn again. time.sleep(10*time.Millisecond) texture_write(t,&types,fmt.tprintf("%s/textures/wood.tex",assets),texture,200) testing.expect_value(t,asset.registry_refresh(&r,{"textures/wood.tex"}),asset.Error.None) saved_texture:=thumbnail_store_key(&r,material,subject,.Small,PROGRAMS) - testing.expect(t,saved_texture!=key && !thumbnail_load(folder,material,THUMBNAIL_SIZE,saved_texture,&file,&read)) + testing.expect(t,saved_texture!=key && !thumbnail_store_load(&st,THUMBNAIL_SIZE,saved_texture,&file,&read)) testing.expect_value(t,thumbnail_store_key(&r,other,{},.Small,PROGRAMS),thumbnail_store_key(&r,other,subject,.Small,PROGRAMS)) - testing.expect(t,thumbnail_save(folder,material,THUMBNAIL_SIZE,saved_texture,image,&file)) - testing.expect(t,thumbnail_load(folder,material,THUMBNAIL_SIZE,saved_texture,&file,&read)) + testing.expect(t,thumbnail_store_save(&st,THUMBNAIL_SIZE,saved_texture,image,&file)) + testing.expect(t,thumbnail_store_load(&st,THUMBNAIL_SIZE,saved_texture,&file,&read)) // Writing the asset itself does too. time.sleep(10*time.Millisecond) material_write(t,&types,fmt.tprintf("%s/wood.mat",assets),material,texture,0.25) testing.expect_value(t,asset.registry_refresh(&r,{"wood.mat"}),asset.Error.None) saved_material:=thumbnail_store_key(&r,material,subject,.Small,PROGRAMS) - testing.expect(t,saved_material!=saved_texture && saved_material!=key && !thumbnail_load(folder,material,THUMBNAIL_SIZE,saved_material,&file,&read)) + testing.expect(t,saved_material!=saved_texture && saved_material!=key && !thumbnail_store_load(&st,THUMBNAIL_SIZE,saved_material,&file,&read)) // What the registry holds of it until its step is written is another content; // dropping it gives back the key of the file. @@ -88,25 +92,24 @@ stored_thumbnails_last_until_their_asset_or_what_it_needs_changes :: proc(t:^tes {dropped,_:=asset.registry_memory_drop(&r,"other.mat");testing.expect(t,dropped)} testing.expect_value(t,thumbnail_store_key(&r,other,subject,.Small,PROGRAMS),other_key) - // An asset that no file is any more loses its previews; the others keep theirs. - other_large:=thumbnail_store_key(&r,other,subject,.Large,PROGRAMS) - testing.expect(t,thumbnail_save(folder,other,THUMBNAIL_LARGE,other_large,make([]u8,THUMBNAIL_LARGE*THUMBNAIL_LARGE*4,context.temp_allocator),&file)) - testing.expect(t,thumbnail_save(folder,other,THUMBNAIL_SIZE,other_key,image,&file)) + // The views of scenes an asset that no file is any more had leave the thumbs folder. + testing.expect(t,thumbnail_save(folder,other,THUMBNAIL_SIZE,THUMBNAIL_CAPTURED,image,&file)) + testing.expect(t,thumbnail_save(folder,material,THUMBNAIL_SIZE,THUMBNAIL_CAPTURED,image,&file)) testing.expect(t,os.remove(fmt.tprintf("%s/other.mat",assets))==nil) testing.expect_value(t,asset.registry_refresh(&r,{"other.mat"}),asset.Error.None) testing.expect(t,len(r.removed)==1 && r.removed[0]==other) - thumbnail_store_forget(folder,r.removed[:]) + thumbnail_views_forget(folder,r.removed[:]) clear(&r.removed) - testing.expect(t,!os.exists(thumbnail_file(folder,other,THUMBNAIL_SIZE)) && !os.exists(thumbnail_file(folder,other,THUMBNAIL_LARGE))) + testing.expect(t,!os.exists(thumbnail_file(folder,other,THUMBNAIL_SIZE))) testing.expect(t,os.is_file(thumbnail_file(folder,material,THUMBNAIL_SIZE))) - // A file cut short is no preview. - path:=thumbnail_file(folder,material,THUMBNAIL_SIZE) - whole,_:=serial.file_read(path,context.temp_allocator) + // Bytes cut short are no preview. + whole:=thumbnail_encode(THUMBNAIL_SIZE,saved_texture,image,&file) + whole=slice.clone(whole,context.temp_allocator) for cut in ([]int{0,THUMBNAIL_FILE_HEADER-1,THUMBNAIL_FILE_HEADER,len(whole)-1}) { - testing.expect_value(t,serial.file_write_atomic(path,whole[:cut]),serial.File_Error.None) - testing.expect(t,!thumbnail_load(folder,material,THUMBNAIL_SIZE,saved_texture,&file,&read)) + testing.expect(t,!thumbnail_decode(whole[:cut],THUMBNAIL_SIZE,saved_texture,&file,&read)) } + testing.expect(t,thumbnail_decode(whole,THUMBNAIL_SIZE,saved_texture,&file,&read)) free_all(context.temp_allocator) } @@ -152,3 +155,58 @@ a_provider_image_is_drawn_between_frames_after_the_frame_that_asked :: proc(t:^t thumbnail_picture_draw(s) testing.expect(t,p.picture.key==0 && !p.dirty) } + +// The key of the preview of an asset of a package: its object and those of its +// build, its size and the programs; nothing of the asset's bytes. +@(test) +the_key_of_a_package_thumbnail_is_its_objects :: proc(t:^testing.T) { + object:=store.address_of_bytes(transmute([]u8)string("mesh")) + other:=store.address_of_bytes(transmute([]u8)string("material")) + objects:=[]store.Hash{object,other} + key:=thumbnail_package_key(.Small,PROGRAMS,object,objects) + testing.expect(t,key!=0 && key==thumbnail_package_key(.Small,PROGRAMS,object,objects),"any machine with the build computes it") + testing.expect(t,key!=thumbnail_package_key(.Large,PROGRAMS,object,objects)) + testing.expect(t,key!=thumbnail_package_key(.Small,PROGRAMS+1,object,objects)) + testing.expect(t,key!=thumbnail_package_key(.Small,PROGRAMS,other,objects)) + testing.expect(t,key!=thumbnail_package_key(.Small,PROGRAMS,object,objects[:1])) + buffer:[16]u8 + testing.expect_value(t,thumbnail_name(0x1a2b,&buffer),"0000000000001a2b") +} + +// A texture's preview is read from its smallest levels once, on a machine +// that has it, and kept in the store under its key; a browser that shows it +// again, also another session of the machine, reads it from there instead. +@(test) +a_texture_preview_is_made_once_from_its_small_levels_and_then_read :: proc(t:^testing.T) { + root:=base.test_folder(t,"thumbnail-texture") + if root=="" {return} + assets:host.Assets + asset.types_init(&assets.types) + defer asset.types_destroy(&assets.types) + os.make_directory_all(fmt.tprintf("%s/assets",root)) + pixels:=make([]u8,128*128*4,context.temp_allocator) + for &value,at in pixels {value=200 if at%4!=1 else 40} + mips:=[][]u8{pixels} + texture:=base.Asset_Id{5,5} + testing.expect_value(t,asset.texture_save(fmt.tprintf("%s/assets/brick.tex",root),&assets.types,{id=texture,width=128,height=128,format=.RGBA8,mips=mips}),asset.Error.None) + asset.registry_init(&assets.registry,fmt.tprintf("%s/assets",root)) + defer asset.registry_destroy(&assets.registry) + testing.expect_value(t,asset.registry_scan(&assets.registry),asset.Error.None) + app:=host.App{assets=&assets} + st,opened:=store.open(fmt.tprintf("%s/store",root)) + if !testing.expect_value(t,opened,store.Error.None) {return} + defer store.close(&st) + for session in 0..<2 { + p:=new(Thumbnails);defer free(p) + defer {delete(p.pixels);delete(p.image);thumbnail_file_destroy(&p.file)} + edits:history.Edits + s:=Session{app=&app,thumbnails=p,store=st,edits=&edits,frame=10} + quad:=preview_texture(&s,texture,THUMBNAIL_SIZE) + if session==0 { + testing.expect(t,p.drawn==s.frame+1 && quad.kind!=.Solid,"made from the texture the first time") + } else { + testing.expect(t,p.drawn!=s.frame+1 && p.loads==1 && quad.kind!=.Solid,"read from the store the second") + } + st=s.store + } +} diff --git a/engine/save/save_test.odin b/engine/save/save_test.odin index 3e968cd47..490122144 100644 --- a/engine/save/save_test.odin +++ b/engine/save/save_test.odin @@ -1,5 +1,6 @@ package save +import "base:intrinsics" import "base:runtime" import "core:c" import "core:fmt" @@ -151,7 +152,7 @@ a_slot_that_fails_halfway_leaves_the_world_as_it_was :: proc(t:^testing.T) { serial.writer_init(&out,0,context.allocator) defer serial.writer_destroy(&out) if !testing.expect_value(t,capture(ref,f.app.assets,f.app.allocator,&out),Error.None) {return} - // The transforms of the broken copy do not decompress. + // The transforms of the broken copy are a stream that does not decode. broken:=make([]u8,len(serial.writer_bytes(&out))) defer delete(broken) copy(broken,serial.writer_bytes(&out)) @@ -161,13 +162,18 @@ a_slot_that_fails_halfway_leaves_the_world_as_it_was :: proc(t:^testing.T) { file:asset.Asset if !testing.expect_value(t,asset.asset_open(&file,body),asset.Error.None) {return} view,_:=asset.scene_view(&file,&types) - entry:asset.Chunk_Entry + chunk:=-1 for &component in view.components { - if component.name=="Transform" {entry=asset.blob_chunk(&file.blob,component.table.columns[1].chunk)} + if component.name=="Transform" {chunk=component.table.columns[1].chunk} } + entry:asset.Chunk_Entry + if chunk>=0 {entry=asset.blob_chunk(&file.blob,chunk)} asset.asset_close(&file) - if !testing.expect_value(t,entry.compression,asset.Compression.Stream) {return} + // Stored as they are, the bytes are made a stream that does not decode: the directory says so. + if !testing.expect(t,chunk>=0 && entry.compression==.None && entry.size>=64) {return} mem.set(&body[entry.offset],0xff,int(entry.size)) + entry.compression=.Stream + intrinsics.unaligned_store((^asset.Chunk_Entry)(&body[asset.HEADER_SIZE+chunk*asset.CHUNK_ENTRY_SIZE]),entry) first:=core.entity_of(&w.entities,1) testing.expect_value(t,restore(ref,&f.app,broken),Error.Failed) diff --git a/engine/scene/scene_test.odin b/engine/scene/scene_test.odin index f59826c3b..281b1f72c 100644 --- a/engine/scene/scene_test.odin +++ b/engine/scene/scene_test.odin @@ -1,5 +1,6 @@ package scene +import "base:intrinsics" import "base:runtime" import "core:fmt" import "core:mem" @@ -291,7 +292,7 @@ a_scene_loads_in_phases_and_replaces_the_world_at_its_end :: proc(t:^testing.T) } // The bytes of a scene with a lamp on each of count entities, whose lamps -// do not decompress when broken: what a load reads after it made the +// are a stream that does not decode when broken: what a load reads after it made the // entities. The content hash follows the bytes. @(private="file") scene_test_lamps :: proc(t:^testing.T,types:^asset.Types,id:base.Asset_Id,count:int,range:f32,broken:bool) -> []u8 { @@ -306,13 +307,18 @@ scene_test_lamps :: proc(t:^testing.T,types:^asset.Types,id:base.Asset_Id,count: file:asset.Asset if !testing.expect(t,asset.asset_open(&file,data,context.temp_allocator)==.None) {return nil} view,_:=asset.scene_view(&file,types) - entry:asset.Chunk_Entry + chunk:=-1 for &component in view.components { - if component.name=="Scene_Test_Lamp" {entry=asset.blob_chunk(&file.blob,component.table.columns[1].chunk)} + if component.name=="Scene_Test_Lamp" {chunk=component.table.columns[1].chunk} } + entry:asset.Chunk_Entry + if chunk>=0 {entry=asset.blob_chunk(&file.blob,chunk)} asset.asset_close(&file) - if !testing.expect_value(t,entry.compression,asset.Compression.Stream) {return nil} + // Stored as they are, the bytes are made a stream that does not decode: the directory says so. + if !testing.expect(t,chunk>=0 && entry.compression==.None && entry.size>=64) {return nil} mem.set(&data[entry.offset],0xff,int(entry.size)) + entry.compression=.Stream + intrinsics.unaligned_store((^asset.Chunk_Entry)(&data[asset.HEADER_SIZE+chunk*asset.CHUNK_ENTRY_SIZE]),entry) (cast(^asset.Header)raw_data(data)).content_hash=asset.blob_hash(data) return data } diff --git a/exchange/ask.odin b/exchange/ask.odin new file mode 100644 index 000000000..9e6758ec0 --- /dev/null +++ b/exchange/ask.odin @@ -0,0 +1,680 @@ +package exchange + +import "mirai:store" +import wire "mirai:net" + +@(private) +want_slot :: proc(x:^Exchange,address:store.Hash) -> int { + mask:=len(x.want_slots)-1 + h:=int(u32(address[4])|u32(address[5])<<8|u32(address[6])<<16|u32(address[7])<<24)&mask + for { + k:=x.want_slots[h] + if k<0 || x.wants[k].address==address {return h} + h=(h+1)&mask + } +} + +@(private) +want_find :: proc(x:^Exchange,address:store.Hash) -> int { + return int(x.want_slots[want_slot(x,address)]) +} + +@(private) +want_add :: proc(x:^Exchange,address:store.Hash,priority:int,measure:=false) -> (int,Error) { + if i:=want_find(x,address);i>=0 { + w:=&x.wants[i] + // A want of the whole object takes over one that only measured it. + if w.measure && !measure { + w.measure=false + want_begin(x,i) + x.dirty=true + } + if priority>w.priority { + w.priority=priority + order_place(x,i) + } + // A want kept complete may have been swept since: success is what the store says now. + if w.state==.Complete && !want_present(x,w) { + want_begin(x,i) + x.dirty=true + } + return i,.None + } + for &w,i in x.wants { + if w.used {continue} + generation:=w.generation+1 + w={used=true,measure=measure,generation=generation,address=address,priority=priority,order=x.sequence,first=-1,last=-1,total=-1} + x.sequence+=1 + x.want_slots[want_slot(x,address)]=i32(i) + x.order[x.want_count]=i + w.rank=x.want_count + x.want_count+=1 + order_place(x,i) + x.dirty=true + want_begin(x,i) + return i,.None + } + return -1,.Full +} + +// Whether the root of a complete want is still in the store. +@(private) +want_present :: proc(x:^Exchange,w:^Want) -> bool { + return store.chunk_has(x.desc.store,w.address) +} + +// From what this store value knows now, without reading its root again: an +// object whose root is here walked from it by a job, or nothing. What another process committed since is fetched again, harmlessly. +@(private) +want_begin :: proc(x:^Exchange,i:int) { + needs_clear(x,i) + w:=&x.wants[i] + if w.settling {x.settling-=1} + // Results of jobs for what the want was are not taken. + w.generation+=1 + w.state,w.total,w.here=.Asking,-1,0 + w.walking,w.walked,w.blocked,w.walk_job,w.shared,w.depth=false,false,false,false,false,0 + w.settling,w.committing=false,false + order_place(x,i) + if store.chunk_here(x.desc.store,w.address) { + // Its owner reads the size of a root here from the store. + if !w.measure {want_walk_root(x,i)} + return + } + need_add(x,i,.Root,w.address,0,0) +} + +// Starts the walk of a want at its root, which is here: a job reads it. +@(private) +want_walk_root :: proc(x:^Exchange,i:int) { + w:=&x.wants[i] + w.walking,w.walked,w.blocked,w.depth=true,false,false,1 + w.stack[0]={address=w.address} + attend(x,i) +} + +@(private) +want_release :: proc(x:^Exchange,i:int) { + needs_clear(x,i) + w:=&x.wants[i] + if w.settling { + w.settling=false + x.settling-=1 + } + w.used=false + // The slot leaves the index: entries after it that probed past it move back. + mask:=len(x.want_slots)-1 + hole:=want_slot(x,w.address) + j:=hole + for { + j=(j+1)&mask + m:=x.want_slots[j] + if m<0 {break} + a:=x.wants[m].address + home:=int(u32(a[4])|u32(a[5])<<8|u32(a[6])<<16|u32(a[7])<<24)&mask + if (j>hole && (home<=hole || home>j)) || (jj) { + x.want_slots[hole]=m + hole=j + } + } + x.want_slots[hole]=-1 + k:=w.rank + copy(x.order[k:x.want_count-1],x.order[k+1:x.want_count]) + x.want_count-=1 + for j in k..0 && want_before(x,i,order[at-1]) { + order[at]=order[at-1] + x.wants[order[at]].rank=at + at-=1 + } + for at+1 bool {return w.state==.Complete || w.state==.Failed} + +@(private) +want_before :: proc(x:^Exchange,a,b:int) -> bool { + wa,wb:=&x.wants[a],&x.wants[b] + ca,cb:=want_closed(wa),want_closed(wb) + if ca!=cb {return cb} + return wa.priority>wb.priority || wa.priority==wb.priority && wa.order=0 {w.here=w.total} + order_place(x,i) + x.dirty=true +} + +@(private) +want_fail :: proc(x:^Exchange,i:int,error:Error,store_error:store.Error) { + needs_clear(x,i) + w:=&x.wants[i] + if w.settling { + w.settling=false + x.settling-=1 + } + w.state,w.error,w.store_error=.Failed,error,store_error + order_place(x,i) + x.dirty=true +} + +// The tree under a verified root does not hold together: every peer would +// send the same, so the want fails. +@(private) +want_corrupt :: proc(x:^Exchange,i:int) { + want_fail(x,i,.Store,.Corrupt) +} + +// The walk is done and nothing it found is missing: once the store commits, +// the want is complete. A chunk another want was asked for is looked up +// again by a walk over the whole tree. +@(private) +want_check :: proc(x:^Exchange,i:int) { + w:=&x.wants[i] + if w.state!=.Asking || !w.walked || w.needs>0 || w.settling || w.walk_job {return} + if w.shared { + w.shared=false + want_walk_root(x,i) + return + } + w.settling=true + x.settling+=1 + x.dirty=true +} + +@(private) +need_slot :: proc(x:^Exchange,address:store.Hash) -> int { + mask:=len(x.need_slots)-1 + h:=int(u32(address[0])|u32(address[1])<<8|u32(address[2])<<16|u32(address[3])<<24)&mask + for { + k:=x.need_slots[h] + if k<0 || x.needs[k].address==address {return h} + h=(h+1)&mask + } +} + +@(private) +need_find :: proc(x:^Exchange,address:store.Hash) -> int { + return int(x.need_slots[need_slot(x,address)]) +} + +// Records a chunk want i lacks; nothing when the table is full, so the walk +// stops there and finds it again later. +@(private) +need_add :: proc(x:^Exchange,i:int,kind:Need_Kind,address:store.Hash,level:int,size:u64) { + if x.need_count==len(x.needs) {return} + slot:=need_slot(x,address) + if x.need_slots[slot]>=0 {return} + k:=x.need_free[len(x.needs)-1-x.need_count] + x.need_count+=1 + x.need_serial+=1 + w:=&x.wants[i] + x.needs[k]={address=address,used=true,kind=kind,level=level,size=size,want=i32(i),serial=x.need_serial,peer=-1,next=-1,prev=w.last} + if w.last>=0 {x.needs[w.last].next=k} else {w.first=k} + w.last=k + x.ask_due=true + w.needs+=1 + w.free+=1 + x.free+=1 + x.need_slots[slot]=k + x.dirty=true +} + +@(private) +need_remove :: proc(x:^Exchange,k:int) { + n:=&x.needs[k] + w:=&x.wants[n.want] + x.ask_due=true + if n.prev>=0 {x.needs[n.prev].next=n.next} else {w.first=n.next} + if n.next>=0 {x.needs[n.next].prev=n.prev} else {w.last=n.prev} + w.needs-=1 + if n.peer<0 { + w.free-=1 + x.free-=1 + } + // Backward shift: the entries after the hole that probed past it move into it. + mask:=len(x.need_slots)-1 + hole:=need_slot(x,n.address) + j:=hole + for { + j=(j+1)&mask + m:=x.need_slots[j] + if m<0 {break} + a:=x.needs[m].address + home:=int(u32(a[0])|u32(a[1])<<8|u32(a[2])<<16|u32(a[3])<<24)&mask + if (j>hole && (home<=hole || home>j)) || (jj) { + x.need_slots[hole]=m + hole=j + } + } + x.need_slots[hole]=-1 + n.used=false + x.need_count-=1 + x.need_free[len(x.needs)-1-x.need_count]=i32(k) +} + +@(private) +needs_clear :: proc(x:^Exchange,i:int) { + w:=&x.wants[i] + for w.first>=0 {need_remove(x,int(w.first))} +} + +// Peer p sent a chunk of the want that failed its address: what it was asked of the want goes +// to others, and it is not asked for it again. +@(private) +lie :: proc(x:^Exchange,i:int,p:int) { + if p<0 {return} + w:=&x.wants[i] + bit:=u64(1)< (f:^Flight,own:bool) { + p:=&x.peers[i] + if p.batches==0 {return nil,false} + size:=p.batch_sizes[p.batch_head] + for k in p.answering.. int { + p:=&x.peers[i] + rtt:=f64(0) + if stats,ok:=wire.connection_stats(p.host,p.connection);ok {rtt=f64(stats.rtt)} + return int(clamp(p.receive_rate*rtt,MIN_BATCH_BYTES,ASK_BYTES)) +} + +@(private) +asking :: proc(x:^Exchange) { + periodic:=x.now-x.ask_time>=ASK_SECONDS + for x.attention_count>0 { + x.attention_count-=1 + i:=x.attention[x.attention_count] + w:=&x.wants[i] + w.attention=false + if w.used && w.state==.Asking {want_attend(x,i)} + } + if periodic { + x.ask_time=x.now + connected:=connected_mask(x) + for k in 0..=LACK_RETRY {w.lack=0} + if w.state==.Asking {want_attend(x,i)} + if w.state!=.Asking && w.state!=.Missing {continue} + // Missing when no peer may be asked for the want, or for a chunk of it. + open:=connected&~(w.wrong|w.lack) + for j:=w.first;j>=0 && open!=0;j=x.needs[j].next { + n:=&x.needs[j] + if n.lack!=0 && x.now-n.lack_since>=LACK_RETRY {n.lack=0} + if n.peer<0 && !n.arriving && open&~n.lack==0 {open=0} + } + if open==0 { + if w.state==.Asking { + w.state=.Missing + w.misses+=1 + x.dirty=true + } + } else if w.state==.Missing { + w.state=.Asking + x.ask_due=true + } + } + for &p,i in x.peers { + if !p.used || p.bad {continue} + for k in 0..=0;j=x.needs[j].next { + need:=&x.needs[j] + if need.peer>=0 || need.arriving || need.lack&bit!=0 {continue} + cost,left:=0,0 + switch need.kind { + case .Root: left=0 if w.measure else ROOT_BUDGET + case .Node: cost=NODE_ESTIMATE + case .Data: cost=int(need.size) + } + if n>0 && bytes+cost>budget {break wants} + entries[n]={need=j,serial=need.serial,want=i32(wi),generation=w.generation,address=need.address,left=i64(left),heard=x.now} + n+=1 + bytes+=cost + if n==ASK_ENTRIES {break wants} + } + } + if n==0 {return} + if !p.told { + // Before its first ask: what this end decodes. + codecs:=[3]u8{MSG_CODECS,0,0} + put_u16(codecs[1:],transmute(u16)store.codecs_read(x.desc.store)) + if !control_append(p,codecs[:]) {return} + p.told=true + } + buffer:[MSG_ASK_MAX]u8 + out:=Writer{data=buffer[:]} + write_u8(&out,MSG_ASK) + write_u16(&out,u16(n)) + for &e in entries[:n] { + write_bytes(&out,e.address[:]) + write_u32(&out,u32(e.left)) + } + if !control_append(p,buffer[:out.n]) {return} + for e in entries[:n] { + x.needs[e.need].peer=i32(i) + x.wants[e.want].free-=1 + x.free-=1 + p.flights[(p.flight_head+p.flight_count)%len(p.flights)]=e + p.flight_count+=1 + } + p.batch_sizes[(p.batch_head+p.batches)%BATCHES]=n + p.batches+=1 +} + +// A chunk is taken only as the answer to an entry asked of that peer, or as a +// descendant of the entry it is answering within its budget, and goes to a +// job that stores it only once it hashes to the address the ask or its parent +// named: one that does not is reported back, and the peer is not asked for +// that want again. A descendant whose parent was not parsed yet waits, hashed, +// until it is. +@(private) +on_chunk :: proc(x:^Exchange,i:int,data:[]u8) { + r:=Reader{data=data[1:],ok=true} + address:=read_hash(&r) + codec:=read_u8(&r) + raw:=int(read_u32(&r)) + bytes:=data[min(MSG_CHUNK_HEAD,len(data)):] + // As they are, or encoded smaller in a codec this end said it decodes. + plain:=codec==store.CODEC_NONE && raw==len(bytes) + encoded:=codec!=store.CODEC_NONE && int(codec)store.CHUNK_MAX {peer_bad(x,i);return} + // What jobs finished is applied first: a parent parsed names its descendants. + work_collect(x) + p:=&x.peers[i] + f,own:=flight_answer(x,i,address) + if f==nil {p.refused+=1;return} + if !own { + f.left-=i64(len(bytes)) + if f.left<0 {peer_bad(x,i);return} + } + bit:=u64(1)<1 {peer_bad(x,i);return} + bit:=u64(1)<1 || count>NAMES_PER_REPLY {peer_bad(x,i);return} + bit:=u64(1)< bool + +Desc :: struct { + store:^store.Store, + channel:int, // index of the first of the CHANNELS channels in the channels of every host + peers:int, // connections at once, default 8, at most MAX_PEERS + wants:int, // objects asked at once, default 256 + queries:int, // ref questions at once, default 16 + needs:int, // chunks known missing at once over every want, default 8192 + worker:jobs.Worker_Ref, // of the thread that pumps: reading, hashing, storing and committing run as its jobs; zero: on the pump + expose:Expose, // nil exposes no namespace + expose_user:rawptr, + allocator:runtime.Allocator, +} + +Error :: enum u8 { + None, + Invalid, // bad argument, or a host without the channels of the exchange + Full, // a capacity of the Desc is exhausted + Missing, // no connected peer has it + Store, // the store refused it: Progress.store_error + Closed, // the exchange was destroyed while waiting + Out_Of_Memory, // only from init +} + +State :: enum u8 { + Asking, // being received + Complete, // in the store, committed + Missing, // no connected peer has it, or one of its chunks, now; asked again when a peer joins or after a while + Failed, // error +} + +Progress :: struct { + state:State, + here:i64, // data bytes in the store + total:i64, // -1 until the root is known + error:Error, + store_error:store.Error, +} + +Peer_Progress :: struct { + receive_rate,send_rate:f64, // chunk bytes per second, smoothed + received,sent:u64, // chunk bytes + lies:int, // wants it sent a chunk for that failed its address + refused:u64, // its messages refused + bad:bool, // it sent what no peer sends: it is neither asked nor served +} + +Query :: struct {index,generation:u32} + +Query_State :: enum u8 {Waiting,Found,Not_Found} + +Name :: struct {name:string,address:store.Hash} + +Query_Result :: struct { + state:Query_State, + address:store.Hash, // of a ref + names:[]Name, // of a namespace, sorted, valid until query_end + more:bool, // the namespace has names after the last: ask again with it as after +} + +@(private) +Walk_Frame :: struct {address:store.Hash,index:int} + +@(private) +Want :: struct { + address:store.Hash, // the root of its object + used,wanted:bool, + measure:bool, // only its root is asked, for its size: nothing under it is walked or asked + waiters:int, + generation:u32, + priority:int, + order:u64, + rank:int, // its place in Exchange.order + attention:bool, // in Exchange.attention + state:State, + error:Error, + store_error:store.Error, + total,here:i64, + wrong,lack:u64, // peers that lied about it, and that lack it + lack_since:f64, + misses:u32, // times it went Missing + needs,free:int, // its needs, and those not asked of a peer + first,last:i32, // its needs, in the order its walk found them + stack:[WALK_DEPTH]Walk_Frame, + depth:int, + walking:bool, // the walk began at the root + walked:bool, // the walk reached the end + blocked:bool, // the walk waits for the node of the entry at the top of its stack + waiting_for:store.Hash, // that node + walk_job:bool, // a walk job holds the stack + shared:bool, // the walk passed a chunk another want needs + settling:bool, // complete once the batch that holds its last chunk is committed + committing:bool, // in the commit job running +} + +@(private) +Need_Kind :: enum u8 {Root,Node,Data} + +// A chunk a want lacks, found by its walk, until it is here. +@(private) +Need :: struct { + address:store.Hash, + used:bool, + kind:Need_Kind, + level:int, // of a node + size:u64, // data bytes under it, as its parent says + want:i32, + serial:u32, + peer:i32, // asked of; -1 + lack:u64, // peers that lacked it + lack_since:f64, + arriving:bool, // its chunk is in a receive slot + next,prev:i32, +} + +@(private) +Flight :: struct { + need:i32, + serial:u32, + want:i32, + generation:u32, // of the want + address:store.Hash, // the chunk asked, which answers it + left:i64, // descendant bytes it may still bring + released:bool, // what it reserved went back to be asked of others + heard:f64, // when it was sent or the peer last made progress on its want +} + +@(private) +Serve_Kind :: enum u8 {Ask,Done} + +@(private) +Serve :: struct {kind:Serve_Kind,address:store.Hash,budget:u32} + +@(private) +Peer :: struct { + used,bad:bool, + exchange:^Exchange, + host:^wire.Host, + connection:wire.Connection, + // Asking: the entries of the asks in flight, oldest first. + flights:[]Flight, + flight_head,flight_count:int, + batch_sizes:[BATCHES]int, + batch_head,batches:int, + answering:int, // the entry of the oldest ask being answered + queries:int, // credits of the questions asked of it and not answered + // Serving: the entries of the asks received, oldest first, taken by its + // serve job; the pump appends at serve_tail, the job takes at serve_head. + serve:[]Serve, + serve_head,serve_tail:int, + serving:bool, // its serve job runs and owns what follows, up to the send slots + stop:bool, // its serve job stops at the next answer + draining:bool, // it left while its serve job ran: the slot frees once it ends + epoch:u32, // of the peer in the slot: results of jobs for another are not its + started:bool, // the chunk of the first entry went + sent_under:i64, // descendant bytes of the first entry sent + stack:[WALK_DEPTH]Walk_Frame, + depth:int, + node:[]u8, // a node being streamed + decodes:store.Codecs, // what it said it decodes; until it says, chunks go as they are + told:bool, // it was told what this end decodes + scratch:[]u8, // a chunk decoded on its way to a codec it decodes + node_address:store.Hash, + node_view:store.Node_View, + node_held:bool, + serve_counter:jobs.Counter, + // Answers the serve job made and the pump sends in order, each its size as + // a u32 and its bytes padded to 4; the job writes at send_tail, the pump + // sends from send_head, both counting bytes from the start. + send:[]u8, + send_head,send_tail:int, + out_sent:int, + control:[]u8, + control_start,control_end:int, + // Receiving, per lane: a message that arrives in pieces. + assembly:[CHANNELS][]u8, + assembled:[CHANNELS]int, + overflow:[CHANNELS]bool, + received,sent,received_mark,sent_mark:u64, + receive_rate,send_rate:f64, + lies:int, + refused:u64, +} + +@(private) +Query_Kind :: enum u8 {Ref,Names} + +@(private) +Query_Slot :: struct { + used:bool, + ended:bool, // its owner is done with it; it stays until every peer asked answers, which frees their credits + generation:u32, + kind:Query_Kind, + state:Query_State, + waiters:int, + namespace,name:[store.NAME_MAX]u8, // name: the name of a ref, or after of a listing + namespace_len,name_len:int, + peer:int, + asked,answered:u64, + asked_at:f64, // when a peer was last asked it + address:store.Hash, + more:bool, + names:[]Name, + name_count:int, + name_bytes:[]u8, +} + +Exchange :: struct { + desc:Desc, + allocator:runtime.Allocator, + lock:sync.Mutex, + changed:sync.Cond, + wants:[]Want, + order:[]int, // wants in use, most important first + want_count:int, + sequence:u64, + needs:[]Need, + need_slots:[]i32, // open addressing by address, -1 empty + need_free:[]i32, + need_count:int, + need_serial:u32, + peers:[]Peer, + queries:[]Query_Slot, + want_slots:[]i32, // open addressing by address, -1 empty + settling:int, + commit_time:f64, + uncommitted:bool, // chunks arrived since the last commit + ask_due:bool, // something changed that the asking pass reads + free:int, // needs not asked of a peer, over every want + attention:[]int, // wants something happened to, for the next asking pass + attention_count:int, + ask_time:f64, // of the last asking pass + receives:[]Receive_Slot, // a ring: applied in the order received + receive_head,receive_count:int, + receive_bytes:[]u8, // of the slots, a ring in their order + receive_tail:int, // where the next slot's bytes go + walkers:[]Walker, // for received nodes + light_found:[]Walk_Found, // of the slots' own walkers + walker_used:[RECEIVE_WALKERS]bool, + walks:[]Walk_Slot, + commit:Commit_Slot, + checks:[]Check_Slot, + found:[]Walk_Found, // what applying a walk adds, deduplicated + now,rate_time:f64, + dirty:bool, + closing:bool, // destroy is waiting for the waiters to leave + waiting:int, // fetch calls waiting +} + +channels :: proc() -> [CHANNELS]wire.Channel_Desc { + return {{kind=.Reliable,queue_bytes=CONTROL_BYTES},{kind=.Reliable,queue_bytes=DATA_QUEUE_BYTES}} +} + +init :: proc(x:^Exchange,desc:Desc) -> Error { + x^={desc=desc} + d:=&x.desc + x.allocator=d.allocator.procedure==nil?context.allocator:d.allocator + if d.store==nil || d.channel<0 {return .Invalid} + if d.peers<=0 {d.peers=8} + if d.wants<=0 {d.wants=256} + if d.queries<=0 {d.queries=16} + if d.needs<=0 {d.needs=8192} + if d.peers>MAX_PEERS || d.needs>1<<24 || d.wants>1<<24 {return .Invalid} + failed:=false + make_into :: proc(x:^Exchange,slot:^$T/[]$E,count:int,failed:^bool) { + made,error:=make(T,count,x.allocator) + if error!=nil {failed^=true;return} + slot^=made + } + slots:=1 + for slots<2*d.needs {slots*=2} + want_slots:=1 + for want_slots<2*d.wants {want_slots*=2} + make_into(x,&x.wants,d.wants,&failed) + make_into(x,&x.order,d.wants,&failed) + make_into(x,&x.want_slots,want_slots,&failed) + make_into(x,&x.attention,d.wants,&failed) + make_into(x,&x.needs,d.needs,&failed) + make_into(x,&x.need_slots,slots,&failed) + make_into(x,&x.need_free,d.needs,&failed) + make_into(x,&x.peers,d.peers,&failed) + make_into(x,&x.queries,d.queries,&failed) + make_into(x,&x.receives,RECEIVE_SLOTS,&failed) + make_into(x,&x.receive_bytes,RECEIVE_BYTES,&failed) + make_into(x,&x.walkers,RECEIVE_WALKERS,&failed) + make_into(x,&x.light_found,RECEIVE_SLOTS*LIGHT_FOUND,&failed) + make_into(x,&x.walks,WALK_SLOTS,&failed) + make_into(x,&x.checks,CHECK_SLOTS,&failed) + make_into(x,&x.found,WALK_AHEAD+PREFETCH_NODES,&failed) + make_into(x,&x.commit.items,d.wants,&failed) + if failed {destroy(x);return .Out_Of_Memory} + for &s in x.need_slots {s=-1} + for &s in x.want_slots {s=-1} + for k in 0..0 {sync.cond_wait(&x.changed,&x.lock)} + for &p in x.peers {sync.atomic_store(&p.stop,true)} + work_wait_all(x) + sync.mutex_unlock(&x.lock) + for &p in x.peers { + delete(p.flights,x.allocator);delete(p.serve,x.allocator) + delete(p.send,x.allocator) + delete(p.node,x.allocator);delete(p.scratch,x.allocator);delete(p.control,x.allocator) + for a in p.assembly {delete(a,x.allocator)} + } + for q in x.queries {delete(q.names,x.allocator);delete(q.name_bytes,x.allocator)} + for k in x.walkers {delete(k.buffer,x.allocator);delete(k.found,x.allocator)} + delete(x.walkers,x.allocator);delete(x.receive_bytes,x.allocator);delete(x.light_found,x.allocator) + for k in x.walks {delete(k.walk.buffer,x.allocator);delete(k.walk.found,x.allocator)} + delete(x.wants,x.allocator);delete(x.order,x.allocator);delete(x.want_slots,x.allocator);delete(x.attention,x.allocator);delete(x.peers,x.allocator) + delete(x.needs,x.allocator);delete(x.need_slots,x.allocator);delete(x.need_free,x.allocator) + delete(x.queries,x.allocator);delete(x.receives,x.allocator);delete(x.walks,x.allocator) + delete(x.checks,x.allocator);delete(x.found,x.allocator);delete(x.commit.items,x.allocator) + x^={} +} + +// Starts exchanging with a connection of h from its start, when its Connected +// command comes: a message on it that began before is not read from its middle. +// The host must carry the channels of the exchange at Desc.channel. +peer_add :: proc(x:^Exchange,h:^wire.Host,connection:wire.Connection) -> Error { + sync.mutex_guard(&x.lock) + if h==nil || connection==0 || x.desc.channel+CHANNELS>h.channel_count {return .Invalid} + for lane in 0..=0 {return .None} + for &p,i in x.peers { + if p.used {continue} + peer_reset(&p) + p.used,p.host,p.connection=true,h,connection + clear_peer_bits(x,i) + x.dirty,x.ask_due=true,true + return .None + } + return .Full +} + +// Stops exchanging with a connection: what it gave stays, what it was asked goes to others. +peer_remove :: proc(x:^Exchange,h:^wire.Host,connection:wire.Connection) { + sync.mutex_guard(&x.lock) + if i:=peer_find(x,h,connection);i>=0 {peer_drop(x,i)} + signal(x) +} + +peer_progress :: proc(x:^Exchange,h:^wire.Host,connection:wire.Connection) -> (Peer_Progress,bool) { + sync.mutex_guard(&x.lock) + i:=peer_find(x,h,connection) + if i<0 {return {},false} + p:=&x.peers[i] + return {receive_rate=p.receive_rate,send_rate=p.send_rate,received=p.received,sent=p.sent,lies=p.lies,refused=p.refused,bad=p.bad},true +} + +@(private) +peer_find :: proc(x:^Exchange,h:^wire.Host,connection:wire.Connection) -> int { + for p,i in x.peers { + if p.used && !p.draining && p.host==h && p.connection==connection {return i} + } + return -1 +} + +// A slot free again, its buffers kept. +@(private) +peer_reset :: proc(p:^Peer) { + keep:=Peer{exchange=p.exchange,flights=p.flights,serve=p.serve,send=p.send,node=p.node,scratch=p.scratch,control=p.control,assembly=p.assembly,epoch=p.epoch+1,decodes={store.CODEC_NONE}} + p^=keep +} + +@(private) +peer_drop :: proc(x:^Exchange,i:int) { + p:=&x.peers[i] + flights_release(x,i) + for &q in x.queries { + if q.used && q.kind==.Names && q.peer==i {q.peer=-1} + } + clear_peer_bits(x,i) + if sync.atomic_load(&p.serving) { + // Its serve job still holds the slot: it frees once the job ends. + sync.atomic_store(&p.stop,true) + p.draining=true + p.epoch+=1 + } else { + peer_reset(p) + } + x.dirty=true +} + +// What a peer was marked for leaves with it, and never passes to the next peer in its slot. +@(private) +clear_peer_bits :: proc(x:^Exchange,i:int) { + mask:=~(u64(1)< bool { + if command.kind!=.Message {return false} + lane:=int(command.channel)-x.desc.channel + if lane<0 || lane>=CHANNELS {return false} + sync.mutex_guard(&x.lock) + i:=peer_find(x,h,command.connection) + if i<0 {return true} + p:=&x.peers[i] + if p.bad {return true} + if p.assembled[lane]==0 && !p.overflow[lane] && !command.more { + message(x,i,lane,command.data) + } else { + buffer:=p.assembly[lane] + if !p.overflow[lane] && p.assembled[lane]+len(command.data)<=len(buffer) { + copy(buffer[p.assembled[lane]:],command.data) + p.assembled[lane]+=len(command.data) + } else { + p.overflow[lane]=true + } + if !command.more { + whole:=buffer[:p.assembled[lane]] + if p.overflow[lane] {peer_bad(x,i)} else {message(x,i,lane,whole)} + p.assembled[lane],p.overflow[lane]=0,false + } + } + signal(x) + return true +} + +// Sends what is due: answers, pieces served and the next requests. now is in +// seconds and must not go back. +update :: proc(x:^Exchange,now:f64) { + sync.mutex_guard(&x.lock) + x.now=max(x.now,now) + work_collect(x) + for &p,i in x.peers { + if !p.used {continue} + if p.draining { + if !sync.atomic_load(&p.serving) {peer_reset(&p)} + continue + } + control_flush(x,i) + if !p.bad {serve_pump(x,i)} + } + asking(x) + settle(x) + queries_send(x) + rates(x) + signal(x) +} + +@(private) +rates :: proc(x:^Exchange) { + dt:=x.now-x.rate_time + if dt<0.1 {return} + x.rate_time=x.now + a:=min(dt/RATE_SECONDS,1) + for &p in x.peers { + if !p.used {continue} + p.receive_rate+=a*(f64(p.received-p.received_mark)/dt-p.receive_rate) + p.send_rate+=a*(f64(p.sent-p.sent_mark)/dt-p.send_rate) + p.received_mark,p.sent_mark=p.received,p.sent + } +} + +@(private) +signal :: proc(x:^Exchange) { + if !x.dirty {return} + x.dirty=false + sync.cond_broadcast(&x.changed) +} + +// Wants an object by its root: asks connected peers +// for what the store lacks of it, most important first. Asking again raises +// its priority. A want stays until forget, also once complete. +want :: proc(x:^Exchange,address:store.Hash,priority:=0) -> Error { + sync.mutex_guard(&x.lock) + i,error:=want_add(x,address,priority,measure=false) + if error==.None {x.wants[i].wanted=true} + return error +} + +// Asks only for the root of an object, without anything under it: progress +// gives its total once the root is here. For the size of an object nobody +// wants yet; forget it once known. A want of the whole object takes it over. +measure :: proc(x:^Exchange,address:store.Hash) -> Error { + sync.mutex_guard(&x.lock) + _,error:=want_add(x,address,0,measure=true) + return error +} + +forget :: proc(x:^Exchange,address:store.Hash) { + sync.mutex_guard(&x.lock) + if i:=want_find(x,address);i>=0 { + x.wants[i].wanted=false + if x.wants[i].waiters==0 {want_release(x,i)} + } +} + +progress :: proc(x:^Exchange,address:store.Hash) -> (Progress,bool) { + sync.mutex_guard(&x.lock) + i:=want_find(x,address) + if i<0 {return {},false} + w:=&x.wants[i] + return {state=w.state,here=w.here,total=w.total,error=w.error,store_error=w.store_error},true +} + +// Returns once the object is complete in the store, or once no +// connected peer has it (Missing). For job threads: the owner keeps calling update meanwhile. +fetch :: proc(x:^Exchange,address:store.Hash,priority:=0) -> Error { + sync.mutex_guard(&x.lock) + address:=address + return fetch_set(x,{address},priority) +} + +// fetch of many objects: wants them together, so they arrive in few round +// trips, and returns once all are complete, at the first that no connected +// peer has (Missing) or at the first error. +fetch_all :: proc(x:^Exchange,addresses:[]store.Hash,priority:=0) -> Error { + sync.mutex_guard(&x.lock) + return fetch_set(x,addresses,priority) +} + +// Returns the address peers name with namespace and name once what it names +// is complete in the store, or Missing once no connected peer names it or has it. +fetch_ref :: proc(x:^Exchange,namespace,name:string,priority:=0) -> (store.Hash,Error) { + names:=[1]string{name} + addresses:[1]store.Hash + error:=fetch_refs(x,namespace,names[:],addresses[:],priority) + return addresses[0],error +} + +// fetch_ref of many names, asked together: addresses (as long as names) gets +// the address of each name peers name, {} for one none names, and what they +// name is fetched as with fetch_all. Missing when a name or an object is missing. +fetch_refs :: proc(x:^Exchange,namespace:string,names:[]string,addresses:[]store.Hash,priority:=0) -> Error { + sync.mutex_guard(&x.lock) + if len(addresses)!=len(names) {return .Invalid} + slice.zero(addresses) + if x.closing {return .Closed} + Held :: struct {query,name:int} + held:[FETCH_WINDOW]Held + count,next:=0,0 + result:Error + x.waiting+=1 + for { + for next Error { + if x.closing {return .Closed} + Held :: struct {want:int,misses:u32} + held:[FETCH_WINDOW]Held + count,next:=0,0 + result:Error + x.waiting+=1 + for { + for next0 {break} + if error!=.None {result=error;break} + x.wants[i].waiters+=1 + held[count]={i,x.wants[i].misses} + count+=1 + next+=1 + } + for k:=0;k []store.Hash { + sync.mutex_guard(&x.lock) + n:=min(x.want_count,len(buffer)) + for k in 0.. (Query,Error) { + sync.mutex_guard(&x.lock) + peer:=-1 + if h!=nil { + if peer=peer_find(x,h,connection);peer<0 {return {},.Invalid} + } + return query_add(x,.Ref,namespace,name,peer) +} + +// Asks a peer for the names of a namespace after after, at most NAMES_PER_REPLY. +names_ask :: proc(x:^Exchange,h:^wire.Host,connection:wire.Connection,namespace:string,after:="") -> (Query,Error) { + sync.mutex_guard(&x.lock) + i:=peer_find(x,h,connection) + if i<0 {return {},.Invalid} + return query_add(x,.Names,namespace,after,i) +} + +query_result :: proc(x:^Exchange,q:Query) -> (Query_Result,bool) { + sync.mutex_guard(&x.lock) + slot:=query_slot(x,q) + if slot==nil {return {},false} + return {state=slot.state,address=slot.address,names=slot.names[:slot.name_count],more=slot.more},true +} + +query_end :: proc(x:^Exchange,q:Query) { + sync.mutex_guard(&x.lock) + if slot:=query_slot(x,q);slot!=nil && slot.waiters==0 {slot.ended=true} +} + +@(private) +query_slot :: proc(x:^Exchange,q:Query) -> ^Query_Slot { + if int(q.index)>=len(x.queries) {return nil} + slot:=&x.queries[q.index] + if !slot.used || slot.ended || slot.generation!=q.generation {return nil} + return slot +} + +@(private) +query_add :: proc(x:^Exchange,kind:Query_Kind,namespace,name:string,peer:int) -> (Query,Error) { + if !store.name_valid(namespace) || !(store.name_valid(name) || kind==.Names && name=="") {return {},.Invalid} + for &slot,i in x.queries { + if slot.used {continue} + generation:=slot.generation+1 + names,bytes:=slot.names,slot.name_bytes + slot={used=true,generation=generation,kind=kind,peer=peer,names=names,name_bytes=bytes} + copy(slot.namespace[:],namespace) + copy(slot.name[:],name) + slot.namespace_len,slot.name_len=len(namespace),len(name) + x.dirty=true + return {u32(i),generation},.None + } + return {},.Full +} + +@(private) +connected_mask :: proc(x:^Exchange) -> (mask:u64) { + for p,i in x.peers { + if p.used && !p.bad {mask|=1<=QUERY_SECONDS { + cost:=q.kind==.Ref?1:NAMES_CREDITS + for &p,i in x.peers { + if late&(u64(1)<=0 {targets=connected&(u64(1)<QUERY_CREDITS {continue} + buffer:[MSG_NAMES_GET_MAX]u8 + message:=encode_question(buffer[:],q.kind==.Ref?MSG_REF_GET:MSG_NAMES_GET,namespace,name) + if !control_append(&p,message) {continue} + q.asked|=bit + q.asked_at=x.now + p.queries+=cost + } + if targets&~q.answered==0 { + q.state=.Not_Found + x.dirty=true + } + } +} + +// Queues a message on the control channel of a peer; false when it has no room. +@(private) +control_append :: proc(p:^Peer,message:[]u8) -> bool { + need:=2+len(message) + if p.control_end+need>len(p.control) && p.control_start>0 { + copy(p.control,p.control[p.control_start:p.control_end]) + p.control_end-=p.control_start + p.control_start=0 + } + if p.control_end+need>len(p.control) {return false} + put_u16(p.control[p.control_end:],u16(len(message))) + copy(p.control[p.control_end+2:],message) + p.control_end+=need + return true +} + +@(private) +control_flush :: proc(x:^Exchange,i:int) { + p:=&x.peers[i] + for p.control_start bool { + if !test_world(t,w,name,2,game) {return false} + server:=test_host(w,0,.Server) + client:=test_host(w,1,.Client) + return testing.expect(t,server!=nil && client!=nil && test_connect(w,client,w.nodes[0].endpoints[0]),"the peers connect") +} + +// An object of size bytes of seed, committed: its root. +@(private) +test_put :: proc(t:^testing.T,s:^store.Store,size:int,seed:u64) -> store.Hash { + return test_object(t,s,test_bytes(size,seed)) +} + +// An object of bytes, committed: its root. +@(private) +test_object :: proc(t:^testing.T,s:^store.Store,data:[]u8) -> store.Hash { + root,error:=store.put(s,data) + testing.expect_value(t,error,store.Error.None) + testing.expect_value(t,store.commit(s),store.Error.None) + return root +} + +// Whether the object at root reads whole from a store. +@(private) +test_whole :: proc(s:^store.Store,root:store.Hash) -> bool { + _,whole:=test_read(s,root) + return whole +} + +// The bytes of an object read back from a store. +@(private) +test_read :: proc(s:^store.Store,root:store.Hash) -> ([]u8,bool) { + o,error:=store.object_open(s,root) + if error!=.None {return nil,false} + defer store.object_close(&o) + data:=make([]u8,int(store.object_size(&o)),context.temp_allocator) + return data,store.object_read(&o,0,data)==.None +} + +@(private) +test_peer :: proc(n:^Test_Node,host:=0) -> Peer_Progress { + h:=n.hosts[host] + for c in h.connection_handles { + if c==0 {continue} + p,_:=peer_progress(&n.x,h,c) + return p + } + return {} +} + +@(test) +objects_of_every_size_arrive_verified_under_loss_and_jitter :: proc(t:^testing.T) { + w:Test_World + defer test_world_destroy(&w) + if !test_pair(t,&w,"exchange-loss") {return} + M:=store.CHUNK_MAX + sizes:=[]int{0,1,store.CHUNK_MIN-1,M,M+1,3*M+17,6<<20} + addresses:=make([]store.Hash,len(sizes),context.temp_allocator) + for size,k in sizes {addresses[k]=test_put(t,&w.nodes[0].s,size,u64(k)+1)} + for e in w.nodes[0].endpoints[:1] {wire.sim_endpoint_conditions(&w.sim,e,{latency=0.02,jitter=0.005,loss=0.05})} + wire.sim_endpoint_conditions(&w.sim,w.nodes[1].endpoints[0],{latency=0.02,jitter=0.005,loss=0.05}) + for a in addresses {testing.expect_value(t,want(&w.nodes[1].x,a),Error.None)} + testing.expect(t,test_until_complete(&w,1,addresses,60),"everything arrives") + s:=&w.nodes[1].s + for size,k in sizes { + data,ok:=test_read(s,addresses[k]) + testing.expect(t,ok && string(data)==string(test_bytes(size,u64(k)+1)),"the object reads back its bytes") + p,_:=progress(&w.nodes[1].x,addresses[k]) + testing.expect(t,p.state==.Complete && p.here==i64(size) && p.total==i64(size),"progress counts the whole object") + } + rate:=test_peer(&w.nodes[1]) + testing.expect(t,rate.received>=6<<20 && rate.receive_rate>0,"the peer's rate is measured") +} + +// What matters most is asked first: a small want asked after a large one and +// raised above it completes long before it. +@(test) +a_high_priority_want_asked_later_completes_before_a_large_low_one :: proc(t:^testing.T) { + w:Test_World + defer test_world_destroy(&w) + if !test_pair(t,&w,"exchange-priority") {return} + large:=test_object(t,&w.nodes[0].s,test_bytes(24<<20,1)) + small:=test_object(t,&w.nodes[0].s,test_bytes(300<<10,2)) + wire.sim_endpoint_conditions(&w.sim,w.nodes[0].endpoints[0],{latency=0.01,bandwidth=4e6}) + x:=&w.nodes[1].x + want(x,large) + test_step(&w,300) + want(x,small) + want(x,small,10) + small_done,large_done:f64 + for w.now<60 && large_done==0 { + test_step(&w) + if small_done==0 && test_state(x,small)==.Complete {small_done=w.now} + if test_state(x,large)==.Complete {large_done=w.now} + } + testing.expectf(t,small_done>0 && large_done>0,"both arrive: %.2f and %.2f s",small_done,large_done) + testing.expectf(t,small_done0 && before.here=before.here,"with no peer the want is missing and keeps what arrived") + testing.expect(t,test_connect(&w,client,w.nodes[0].endpoints[0]),"the peers connect again") + testing.expect(t,test_until_complete(&w,1,{first},30),"the object completes after the connection returns") + + TOTAL :: 8<<20 + second:=test_object(t,&w.nodes[0].s,test_bytes(TOTAL,4)) + want(x,second) + test_step(&w,1500) + part,_:=progress(x,second) + testing.expect(t,part.here>0 && part.here0,"the reopened store resumes from the chunks it committed: %d of %d before",resumed.here,part.here) + testing.expect(t,test_until_complete(&w,1,{second},30),"the second object completes") + got:=test_peer(n) + data,_:=test_read(&n.s,second) + testing.expect(t,string(data)==string(test_bytes(TOTAL,4)),"and reads back its bytes") + // What it had is not asked again: the data it lacked and the nodes above it. + testing.expectf(t,got.received<=u64(TOTAL-resumed.here)+u64(store.CHUNK_MAX),"after the reopen %d bytes arrive of the %d missing",got.received,TOTAL-resumed.here) +} + +// Two sources, one of which lies; it is asked first, so its lie is met. A +// chunk that fails its address marks the liar for the want, never the honest +// peer, and the want still completes. +@(test) +a_liar_is_marked_and_the_object_still_arrives :: proc(t:^testing.T) { + w:Test_World + defer test_world_destroy(&w) + if !test_world(t,&w,"exchange-liar",3) {return} + test_host(&w,0,.Server) + liar:=test_host(&w,1,.Client) + honest:=test_host(&w,2,.Client) + w.nodes[1].script,w.nodes[1].lie,w.nodes[1].source=test_liar,.Bytes,&w.nodes[1].s + size:=3*store.CHUNK_MAX+1000 + object:=test_put(t,&w.nodes[1].s,size,9) + test_put(t,&w.nodes[2].s,size,9) + if !testing.expect(t,test_connect(&w,liar,w.nodes[0].endpoints[0]) && test_connect(&w,honest,w.nodes[0].endpoints[0])) {return} + r:=&w.nodes[0] + want(&r.x,object) + testing.expect(t,test_until_complete(&w,0,{object},20),"the object arrives") + data,ok:=test_read(&r.s,object) + testing.expect(t,ok && string(data)==string(test_bytes(size,9)),"with its bytes") + testing.expect(t,test_received_by(&r.x,r.hosts[0])=={1,1},"the liar is marked and the honest peer is not") +} + +// Flips the byte at at of an object's bytes where a pack of the store at root +// holds it, as damage on the disk would leave it; false when no pack holds +// those bytes. +@(private="file") +pack_damage :: proc(root:string,data:[]u8,at:int) -> bool { + start:=max(at-32,0) + needle:=data[start:min(start+64,len(data))] + entries,_:=os.read_all_directory_by_path(fmt.tprintf("%s/packs",root),context.temp_allocator) + for e in entries { + bytes,read_error:=os.read_entire_file(e.fullpath,context.temp_allocator) + if read_error!=nil {continue} + found:=strings.index(string(bytes),string(needle)) + if found<0 {continue} + file,open_error:=os.open(e.fullpath,{.Read,.Write}) + if open_error!=nil {return false} + defer os.close(file) + flipped:=[1]u8{bytes[found+at-start]~0xff} + _,write_error:=os.write_at(file,flipped[:],i64(found+at-start)) + return write_error==nil + } + return false +} + +// A peer whose copy of a chunk was damaged on its disk serves it, since nothing +// is checked on serve; the receiver reports it, the peer checks its copy and +// tombstones it, so it lacks it from then on and fetches it again. +@(test) +a_receivers_bad_makes_the_server_repair_or_lack :: proc(t:^testing.T) { + w:Test_World + defer test_world_destroy(&w) + if !test_world(t,&w,"exchange-damaged",3) {return} + test_host(&w,0,.Server) + damaged:=test_host(&w,1,.Client) + honest:=test_host(&w,2,.Client) + size:=3*store.CHUNK_MAX+1000 + data:=test_bytes(size,11) + root:=test_object(t,&w.nodes[1].s,data) + test_object(t,&w.nodes[2].s,data) + testing.expect(t,pack_damage(w.nodes[1].s.root,data,size-5),"the damage lands in a pack") + if !testing.expect(t,test_connect(&w,damaged,w.nodes[0].endpoints[0])) {return} + r:=&w.nodes[0] + want(&r.x,root) + test_step(&w,500) + testing.expect(t,!test_here(r,root),"the damaged chunk is not taken") + if !testing.expect(t,test_connect(&w,honest,w.nodes[0].endpoints[0])) {return} + testing.expect(t,test_until_complete(&w,0,{root},20),"the honest peer completes the object") + back,_:=test_read(&r.s,root) + testing.expect(t,string(back)==string(data),"which reads back its bytes") + o,_:=store.object_open(&w.nodes[1].s,root) + copy_ok:=store.object_read(&o,0,make([]u8,size,context.temp_allocator)) + store.object_close(&o) + testing.expect(t,copy_ok!=.None,"the damaged copy is tombstoned: the peer no longer reads it") + want(&w.nodes[1].x,root) + testing.expect(t,test_until_complete(&w,1,{root},20),"and fetches it again") + again,_:=test_read(&w.nodes[1].s,root) + testing.expect(t,string(again)==string(data),"whole") +} + +// A peer serves the chunks it has of an object it is still receiving. +@(test) +a_peer_still_receiving_an_object_serves_its_chunks_to_a_third :: proc(t:^testing.T) { + w:Test_World + defer test_world_destroy(&w) + if !test_world(t,&w,"exchange-relay",3) {return} + test_host(&w,0,.Server) + b_client:=test_host(&w,1,.Client) + test_host(&w,1,.Server) + c_client:=test_host(&w,2,.Client) + if !testing.expect(t,test_connect(&w,b_client,w.nodes[0].endpoints[0]) && test_connect(&w,c_client,w.nodes[1].endpoints[1])) {return} + wire.sim_endpoint_conditions(&w.sim,w.nodes[0].endpoints[0],{latency=0.01,bandwidth=2e6}) + root:=test_object(t,&w.nodes[0].s,test_bytes(6<<20,5)) + want(&w.nodes[1].x,root) + want(&w.nodes[2].x,root) + relayed:=false + for w.now<60 && !test_here(&w.nodes[2],root) { + test_step(&w) + c,_:=progress(&w.nodes[2].x,root) + if c.here>0 && !test_here(&w.nodes[1],root) {relayed=true} + } + testing.expect(t,relayed,"the third peer receives chunks before the relay has the whole object") + data,ok:=test_read(&w.nodes[2].s,root) + testing.expect(t,ok && string(data)==string(test_bytes(6<<20,5)),"the third peer completes the object") +} + +@(private) +test_expose_lib :: proc(user:rawptr,namespace:string) -> bool {return namespace=="lib"} + +@(test) +refs_are_asked_and_listed_and_a_namespace_not_exposed_gives_nothing :: proc(t:^testing.T) { + w:Test_World + defer test_world_destroy(&w) + if !test_world(t,&w,"exchange-refs",2,desc={expose=test_expose_lib}) {return} + test_host(&w,0,.Server) + client:=test_host(&w,1,.Client) + if !testing.expect(t,test_connect(&w,client,w.nodes[0].endpoints[0])) {return} + s:=&w.nodes[0].s + blob:=test_put(t,s,1000,1) + names:=make([]string,NAMES_PER_REPLY+5,context.temp_allocator) + for &name,k in names { + buffer:=make([]u8,4,context.temp_allocator) + buffer[0],buffer[1],buffer[2],buffer[3]='p',u8('a'+k/26/26%26),u8('a'+k/26%26),u8('a'+k%26) + name=string(buffer) + store.ref_set(s,"lib",name,blob) + } + store.ref_set(s,"secret","key",blob) + x:=&w.nodes[1].x + ask :: proc(w:^Test_World,x:^Exchange,q:Query) -> Query_Result { + for _ in 0..<2000 { + test_step(w) + if r,_:=query_result(x,q);r.state!=.Waiting {return r} + } + return {} + } + q,_:=ref_ask(x,"lib","pabc") + r:=ask(&w,x,q) + testing.expect(t,r.state==.Found && r.address==blob,"an exposed ref gives its address") + query_end(x,q) + q,_=ref_ask(x,"secret","key") + r=ask(&w,x,q) + testing.expect_value(t,r.state,Query_State.Not_Found) + query_end(x,q) + q,_=ref_ask(x,"lib","nothing") + testing.expect_value(t,ask(&w,x,q).state,Query_State.Not_Found) + query_end(x,q) + q,_=names_ask(x,client,client.connection_handles[0],"lib") + r=ask(&w,x,q) + testing.expect(t,r.state==.Found && len(r.names)==NAMES_PER_REPLY && r.more,"the first page of names") + last:="" + for n,k in r.names { + testing.expect(t,n.name==names[k] && n.address==blob,"names come sorted with their addresses") + last=n.name + } + after:=make([]u8,len(last),context.temp_allocator) + copy(after,last) + query_end(x,q) + q,_=names_ask(x,client,client.connection_handles[0],"lib",string(after)) + r=ask(&w,x,q) + testing.expect(t,r.state==.Found && len(r.names)==5 && !r.more && r.names[0].name==names[NAMES_PER_REPLY],"the page after the last name") + query_end(x,q) + q,_=names_ask(x,client,client.connection_handles[0],"secret") + r=ask(&w,x,q) + testing.expect(t,r.state==.Found && len(r.names)==0,"a namespace not exposed lists nothing") + query_end(x,q) +} + +@(private) +Test_Job :: struct { + x:^Exchange, + address:store.Hash, + namespace,name:string, + addresses:[]store.Hash, // fetch_all, or what fetch_refs found for names + names:[]string, + error:Error, + found:store.Hash, + done:bool, +} + +@(private) +test_job :: proc(j:^Test_Job) { + if j.names!=nil { + j.error=fetch_refs(j.x,j.namespace,j.names,j.addresses) + } else if j.addresses!=nil { + j.error=fetch_all(j.x,j.addresses) + } else if j.namespace!="" { + j.found,j.error=fetch_ref(j.x,j.namespace,j.name) + } else { + j.error=fetch(j.x,j.address) + } + sync.atomic_store(&j.done,true) +} + +// Runs a fetch on a thread while the owner pumps; false if it does not return within seconds. +@(private) +test_fetch :: proc(w:^Test_World,j:^Test_Job,seconds:f64,during:proc(w:^Test_World)=nil) -> bool { + th:=thread.create_and_start_with_poly_data(j,test_job) + defer {thread.join(th);thread.destroy(th)} + end:=w.now+seconds + for w.now1<<20 {wire.host_disconnect(w.nodes[0].hosts[0])} + }) && j.error==.Missing,"a disconnect returns Missing") + testing.expect(t,!test_whole(&w.nodes[1].s,big),"the disconnect came before the object") +} + +// A 60 Hz unreliable message from the source keeps its delay small while objects stream. +@(test) +a_game_channel_keeps_its_delay_small_during_a_transfer :: proc(t:^testing.T) { + w:Test_World + defer test_world_destroy(&w) + if !test_pair(t,&w,"exchange-game",game=true) {return} + latency:=0.02 + wire.sim_endpoint_conditions(&w.sim,w.nodes[0].endpoints[0],{latency=latency,bandwidth=12.5e6}) + wire.sim_endpoint_conditions(&w.sim,w.nodes[1].endpoints[0],{latency=latency}) + roots:=make([]store.Hash,6,context.temp_allocator) + for &b,k in roots {b=test_object(t,&w.nodes[0].s,test_bytes(8<<20,u64(k)+1))} + for b in roots {want(&w.nodes[1].x,b)} + g:=&w.nodes[1].game + g.latency,g.since=latency,w.now+1 + w.nodes[0].game.since=g.since + complete:=test_until_complete(&w,1,roots,30) + testing.expect(t,complete,"the objects arrive") + sent:=w.nodes[0].game.sent + testing.expectf(t,g.got>=sent*9/10 && sent>100,"%d of %d game messages arrive",g.got,sent) + mean:=g.mean/f64(max(g.got,1)) + testing.expectf(t,mean<0.03 && g.worst<0.08,"game messages take %.0f ms more than the latency on average, %.0f at worst",mean*1000,g.worst*1000) +} + +// A scripted peer at node k that answers as lie says, connected to the server at node 0. +@(private) +test_scripted :: proc(t:^testing.T,w:^Test_World,k:int,lie:Test_Lie) -> ^wire.Host { + h:=test_host(w,k,.Client) + w.nodes[k].script,w.nodes[k].lie,w.nodes[k].source=test_liar,lie,&w.nodes[k].s + return h +} + +@(private) +test_received_by :: proc(x:^Exchange,h:^wire.Host) -> (lies:[2]int) { + for c in h.connection_handles { + if c==0 {continue} + p,_:=peer_progress(x,h,c) + lies[0 if p.lies>0 else 1]+=1 + } + return +} + +// A peer that was asked a want and then sends only unrelated data loses the +// reservation, and the honest peer serves the object. +@(test) +a_peer_that_holds_a_want_without_progress_loses_it :: proc(t:^testing.T) { + w:Test_World + defer test_world_destroy(&w) + if !test_world(t,&w,"exchange-hold",3) {return} + test_host(&w,0,.Server) + holder:=test_scripted(t,&w,1,.Hold) + honest:=test_host(&w,2,.Client) + root:=test_object(t,&w.nodes[2].s,test_bytes(2*store.CHUNK_MAX+7,6)) + if !testing.expect(t,test_connect(&w,holder,w.nodes[0].endpoints[0]) && test_connect(&w,honest,w.nodes[0].endpoints[0])) {return} + want(&w.nodes[0].x,root) + testing.expect(t,test_until_complete(&w,0,{root},15),"the honest peer serves the object once the reservation lapses") +} + +// What a peer was marked for leaves with it: the next peer in its slot is not +// blamed for the chunks the first one damaged. +@(test) +a_peer_in_a_freed_slot_is_not_blamed_for_its_predecessor :: proc(t:^testing.T) { + w:Test_World + defer test_world_destroy(&w) + if !test_world(t,&w,"exchange-slot",3) {return} + server:=test_host(&w,0,.Server) + liar:=test_scripted(t,&w,1,.Bytes) + honest:=test_host(&w,2,.Client) + data:=test_bytes(4*store.CHUNK_MAX+1000,8) + root:=test_object(t,&w.nodes[1].s,data) + test_object(t,&w.nodes[2].s,data) + if !testing.expect(t,test_connect(&w,liar,w.nodes[0].endpoints[0])) {return} + r:=&w.nodes[0] + want(&r.x,root) + for _ in 0..<2000 { + test_step(&w) + if test_received_by(&r.x,server)[0]==1 {break} + } + testing.expect_value(t,test_received_by(&r.x,server),[2]int{1,0}) + wire.host_disconnect(liar) + for _ in 0..<2000 { + test_step(&w) + if server.connection_handles[0]==0 && server.connection_handles[1]==0 {break} + } + if !testing.expect(t,test_connect(&w,honest,w.nodes[0].endpoints[0])) {return} + testing.expect(t,test_until_complete(&w,0,{root},20),"the honest peer in the freed slot completes the object") + testing.expect_value(t,test_received_by(&r.x,server),[2]int{0,1}) +} + +// A lookup answered by a fast peer leaves the slow peer's credits to return +// with its late answers, so a later lookup that needs the slow peer finds it. +@(test) +late_ref_answers_return_their_credits :: proc(t:^testing.T) { + w:Test_World + defer test_world_destroy(&w) + if !test_world(t,&w,"exchange-credits",3,desc={expose=test_expose_lib}) {return} + test_host(&w,0,.Server) + fast:=test_host(&w,1,.Client) + slow:=test_host(&w,2,.Client) + if !testing.expect(t,test_connect(&w,fast,w.nodes[0].endpoints[0]) && test_connect(&w,slow,w.nodes[0].endpoints[0])) {return} + wire.sim_endpoint_conditions(&w.sim,w.nodes[2].endpoints[0],{latency=1}) + blob:=test_put(t,&w.nodes[2].s,100,1) + store.ref_set(&w.nodes[2].s,"lib","only",blob) + names:=[]string{"ka","kb","kc","kd"} + for name in names { + store.ref_set(&w.nodes[1].s,"lib",name,blob) + store.ref_set(&w.nodes[2].s,"lib",name,blob) + } + x:=&w.nodes[0].x + for name in names { + q,_:=ref_ask(x,"lib",name) + for _ in 0..<500 { + test_step(&w) + if r,_:=query_result(x,q);r.state!=.Waiting {break} + } + r,_:=query_result(x,q) + testing.expectf(t,r.state==.Found,"%s is found by the fast peer",name) + query_end(x,q) + } + q,_:=ref_ask(x,"lib","only") + found:=false + for _ in 0..<8000 { + test_step(&w) + if r,_:=query_result(x,q);r.state!=.Waiting {found=r.state==.Found;break} + } + testing.expect(t,found,"the slow peer is asked and names it") +} + +// The bytes of every chunk of an object in a store: what serving it once sends. +@(private) +test_object_bytes :: proc(s:^store.Store,root:store.Hash) -> (total:u64) { + buffer:=make([]u8,store.CHUNK_MAX,context.temp_allocator) + bytes,error:=store.chunk_read(s,root,buffer) + if error!=.None {return} + total=u64(len(bytes)) + n,_:=store.node_parse(bytes) + held:=make([]store.Entry,len(n.entries),context.temp_allocator) + copy(held,n.entries) + for e in held {total+=n.level==0?e.size:test_object_bytes(s,e.address)} + return +} + +// After a peer's flight lapses and its want goes to another, its late Done +// does not free what the other was asked: no chunk is served twice. +@(test) +a_late_done_does_not_release_what_another_peer_was_asked :: proc(t:^testing.T) { + w:Test_World + defer test_world_destroy(&w) + if !test_world(t,&w,"exchange-late",3) {return} + test_host(&w,0,.Server) + late:=test_scripted(t,&w,1,.Late_Done) + honest:=test_host(&w,2,.Client) + root:=test_object(t,&w.nodes[2].s,test_bytes(2*store.CHUNK_MAX,3)) + if !testing.expect(t,test_connect(&w,late,w.nodes[0].endpoints[0]) && test_connect(&w,honest,w.nodes[0].endpoints[0])) {return} + wire.sim_endpoint_conditions(&w.sim,w.nodes[2].endpoints[0],{latency=0.01,bandwidth=0.5e6}) + want(&w.nodes[0].x,root) + testing.expect(t,test_until_complete(&w,0,{root},30),"the honest peer serves the object") + served,_:=peer_progress(&w.nodes[2].x,honest,honest.connection_handles[0]) + whole:=test_object_bytes(&w.nodes[2].s,root) + testing.expectf(t,served.sent==whole,"the honest peer serves %d bytes of %d",served.sent,whole) + testing.expectf(t,w.nodes[1].requests==1,"the late peer is asked %d times",w.nodes[1].requests) +} + +// A connected peer that never answers a question is taken to know nothing of +// it in bounded time, and its credits come back: questions after the first +// credits' worth end too. +@(test) +a_question_a_silent_peer_never_answers_ends :: proc(t:^testing.T) { + w:Test_World + defer test_world_destroy(&w) + if !test_world(t,&w,"exchange-silent-question",2) {return} + test_host(&w,0,.Server) + silent:=test_scripted(t,&w,1,.Hold) + if !testing.expect(t,test_connect(&w,silent,w.nodes[0].endpoints[0])) {return} + x:=&w.nodes[0].x + for round in 0..0 {break} + time.sleep(time.Millisecond) + } + testing.expect(t,!sync.atomic_load(&j.done),"the fetch waits on the silent peer") + destroy(&n.x) + n.open=false + thread.join(th) + thread.destroy(th) + testing.expect_value(t,j.error,Error.Closed) +} + +// A connection that carried a transfer, past the start of its congestion +// window, at a round trip of rtt. +@(private) +test_warm_pair :: proc(t:^testing.T,w:^Test_World,name:string,rtt:f64) -> bool { + if !test_pair(t,w,name) {return false} + for e in ([]int{w.nodes[0].endpoints[0],w.nodes[1].endpoints[0]}) {wire.sim_endpoint_conditions(&w.sim,e,{latency=rtt/2})} + warm:=test_object(t,&w.nodes[0].s,test_bytes(2<<20,1000)) + want(&w.nodes[1].x,warm) + return testing.expect(t,test_until_complete(w,1,{warm},30),"the warming object arrives") +} + +// Small objects wanted together on a connection in use arrive in a few round +// trips however many they are, and one alone in one. +@(test) +many_small_objects_wanted_together_take_round_trips_that_do_not_grow_with_their_count :: proc(t:^testing.T) { + RTT :: 0.1 + for count in ([]int{1,8,120,250}) { + w:Test_World + defer test_world_destroy(&w) + if !test_warm_pair(t,&w,"exchange-small",RTT) {return} + roots:=make([]store.Hash,count,context.temp_allocator) + for &b,k in roots {b=test_object(t,&w.nodes[0].s,test_bytes((1+k%5)*1000,u64(k)+1))} + start:=w.now + for b in roots {want(&w.nodes[1].x,b)} + testing.expectf(t,test_until_complete(&w,1,roots,30),"%d objects arrive",count) + trips:=(w.now-start)/RTT + testing.expectf(t,trips<(count==1?1.2:4),"%d small objects take %.2f round trips",count,trips) + } +} + +// An object changed by one byte inserted sends only its new chunks and the +// nodes above them. +@(test) +a_one_byte_insert_sends_only_the_new_chunks :: proc(t:^testing.T) { + SIZE :: 200<<20 + w:Test_World + defer test_world_destroy(&w) + if !test_pair(t,&w,"exchange-insert") {return} + data:=test_bytes(SIZE+1,77) + before:=data[:SIZE] + test_object(t,&w.nodes[1].s,before) + copy(data[SIZE/2+1:],before[SIZE/2:]) + data[SIZE/2]=0x5a + after:=test_object(t,&w.nodes[0].s,data) + want(&w.nodes[1].x,after) + testing.expect(t,test_until_complete(&w,1,{after},60),"the changed object arrives") + got:=test_peer(&w.nodes[1]) + fmt.printfln("a one-byte insert in %d bytes: %d bytes received",SIZE,got.received) + testing.expectf(t,got.received<=512<<10,"%d bytes received",got.received) + back,_:=test_read(&w.nodes[1].s,after) + testing.expect(t,string(back)==string(data),"it reads back its bytes") +} + +// fetch_all and fetch_refs return once all is here, at a miss among hits, and +// at a disconnect. +@(test) +the_batch_blocking_forms_return_for_a_hit_a_partial_miss_and_a_disconnect :: proc(t:^testing.T) { + w:Test_World + defer test_world_destroy(&w) + if !test_world(t,&w,"exchange-fetch-all",2,desc={expose=test_expose_lib}) {return} + test_host(&w,0,.Server) + client:=test_host(&w,1,.Client) + if !testing.expect(t,test_connect(&w,client,w.nodes[0].endpoints[0])) {return} + x,here:=&w.nodes[1].x,&w.nodes[1].s + objects:=make([]store.Hash,40,context.temp_allocator) + for &o,k in objects {o=test_put(t,&w.nodes[0].s,(k%7)*30000+1,u64(k)+1)} + j:=Test_Job{x=x,addresses=objects} + testing.expectf(t,test_fetch(&w,&j,10) && j.error==.None,"a set of hits returns: %v",j.error) + for b in objects { + testing.expect(t,test_whole(here,b),"every object is here") + _,still:=progress(x,b) + testing.expect(t,!still,"a fetch_all leaves no want behind") + } + // More names than query slots, so the form waits for slots to free. + names:=[]string{"a","b","c","d","e","f","g","h","i","j","k","l","m","n","o","p","q","r","s","t"} + for name,k in names {store.ref_set(&w.nodes[0].s,"lib",name,test_put(t,&w.nodes[0].s,5000*(k+1),u64(100+k)))} + j={x=x,namespace="lib",names=names,addresses=make([]store.Hash,len(names),context.temp_allocator)} + testing.expect(t,test_fetch(&w,&j,10) && j.error==.None,"a set of ref hits returns") + for a in j.addresses {testing.expect(t,a!={} && test_whole(here,a),"each ref's object is here")} + absent:=store.address_of_bytes(test_bytes(10,99)) + fresh:=[]store.Hash{test_put(t,&w.nodes[0].s,3000,200),absent,test_put(t,&w.nodes[0].s,4000,201)} + j={x=x,addresses=fresh} + testing.expect(t,test_fetch(&w,&j,10) && j.error==.Missing,"a miss among hits returns Missing") + j={x=x,namespace="lib",names={"a","absent"},addresses=make([]store.Hash,2,context.temp_allocator)} + testing.expect(t,test_fetch(&w,&j,10) && j.error==.Missing && j.addresses[1]=={},"a ref miss returns Missing") + wire.sim_endpoint_conditions(&w.sim,w.nodes[0].endpoints[0],{latency=0.01,bandwidth=2e6}) + big:=[]store.Hash{test_put(t,&w.nodes[0].s,4<<20,300),test_put(t,&w.nodes[0].s,4<<20,301)} + j={x=x,addresses=big} + testing.expect(t,test_fetch(&w,&j,20,proc(w:^Test_World) { + client:=w.nodes[1].hosts[0] + if client.connection_handles[0]==0 {return} + if p,ok:=peer_progress(&w.nodes[1].x,client,client.connection_handles[0]);ok && p.received>1<<20 {wire.host_disconnect(w.nodes[0].hosts[0])} + }) && j.error==.Missing,"a disconnect returns Missing") + testing.expect(t,!test_whole(here,big[0]) || !test_whole(here,big[1]),"the disconnect came before the objects") +} + +// A batch of refs that came back missing leaves its query slots to free as +// their answers arrive: the next batch, asked at once, waits for them and +// finds what it names instead of failing as full. +@(test) +a_batch_of_refs_waits_for_the_slots_of_the_last :: proc(t:^testing.T) { + w:Test_World + defer test_world_destroy(&w) + if !test_world(t,&w,"exchange-refs-full",2,desc={expose=test_expose_lib}) {return} + test_host(&w,0,.Server) + client:=test_host(&w,1,.Client) + if !testing.expect(t,test_connect(&w,client,w.nodes[0].endpoints[0])) {return} + present:=test_put(t,&w.nodes[0].s,3000,7) + store.ref_set(&w.nodes[0].s,"lib","present",present) + x:=&w.nodes[1].x + // Every slot holds a question asked and ended whose answer is on its way. + wire.sim_endpoint_conditions(&w.sim,w.nodes[0].endpoints[0],{latency=0.1}) + asked:=make([]Query,len(x.queries),context.temp_allocator) + for &q,k in asked {q,_=ref_ask(x,"lib",fmt.tprintf("absent-%d",k))} + test_step(&w,2) + for q in asked {query_end(x,q)} + j:=Test_Job{x=x,namespace="lib",names={"present"},addresses=make([]store.Hash,1,context.temp_allocator)} + th:=thread.create_and_start_with_poly_data(&j,test_job) + defer {thread.join(th);thread.destroy(th)} + // Nothing steps until the job returned or waits: no slot frees before it asks. + for !sync.atomic_load(&j.done) { + sync.mutex_lock(&x.lock) + waiting:=x.waiting>0 + sync.mutex_unlock(&x.lock) + if waiting {break} + thread.yield() + } + for step:=0;!sync.atomic_load(&j.done) && step<10_000;step+=1 {test_step(&w)} + testing.expectf(t,sync.atomic_load(&j.done) && j.error==.None && j.addresses[0]==present,"the batch finds its ref: %v",j.error) +} + +// An object of ten gigabytes moves with the memory of each side bounded by +// its window, not by the object: its leaves are a pool of distinct chunks, each +// used many times in an order that makes every leaf node differ, so the walk +// covers every entry while the disks hold only the pool. +@(test) +a_ten_gigabyte_object_moves_within_bounded_memory :: proc(t:^testing.T) { + SIZE :: u64(10_000_000_000) + POOL :: 8192 + CHUNK :: 64<<10 + w:Test_World + defer test_world_destroy(&w) + if !test_pair(t,&w,"exchange-large") {return} + s:=&w.nodes[0].s + pool:=make([]store.Hash,POOL) + defer delete(pool) + for &a,k in pool { + bytes:=test_bytes(CHUNK,u64(k)+1) + a=store.address_of_bytes(bytes) + testing.expect_value(t,store.chunk_put(s,a,bytes),store.Error.None) + free_all(context.temp_allocator) + } + tree:store.Tree + store.tree_init(&tree,{procedure=proc(data:rawptr,kind:store.Chunk_Kind,address:store.Hash,bytes:[]u8) -> store.Error { + return store.chunk_put((^store.Store)(data),address,bytes) + },data=s}) + defer store.tree_destroy(&tree) + store.tree_group(&tree,0,0) + for p:=u64(0);p*CHUNK>29 + a:=pool[h%POOL] + if size store.Codec_Hook { + return {codec=15,encode=proc(src,dst:[]u8) -> int { + e:data.Encoder + data.encoder_init(&e) + defer data.encoder_destroy(&e) + size,error:=data.encode(&e,.Fast,src,dst) + return size if error==.None else 0 + },decode=proc(src,dst:[]u8) -> bool { + d:data.Decoder + if data.decoder_init(&d)!=.None {return false} + defer data.decoder_destroy(&d) + return data.decode(&d,data.ZSTD,src,dst)==.None + }} +} + +// Bytes that compress: runs of a few values. +@(private="file") +compressible :: proc(size:int,seed:u64) -> []u8 { + bytes:=make([]u8,size,context.temp_allocator) + state:=seed + for at:=0;at>61),1+int(state>>58) + for k in 0.. bool { + w.folder=base.test_folder(t,name) + if w.folder=="" {return false} + if !testing.expect(t,jobs.system_init(&w.js,{threads=2})) {return false} + wire.sim_network_init(&w.sim,{seed=7,endpoints=8,packets=1<<16}) + ours:=channels() + if game { + w.channels=make([]wire.Channel_Desc,3,context.temp_allocator) + w.channels[0]={kind=.Unreliable} + w.channels[1],w.channels[2]=ours[0],ours[1] + w.channel=1 + } else { + w.channels=make([]wire.Channel_Desc,2,context.temp_allocator) + w.channels[0],w.channels[1]=ours[0],ours[1] + } + w.count=nodes + for k in 0.. bool { + n:=&w.nodes[k] + error:store.Error + n.s,error=store.open(fmt.tprintf("%s/node-%d",w.folder,k)) + if error!=.None {return false} + d:=desc + d.store,d.channel=&n.s,w.channel + // Opt-in: -define:EXCHANGE_TEST_JOBS=true runs every exchange's jobs on the + // job system; by default they run on the pump, so virtual time steps fast. + if #config(EXCHANGE_TEST_JOBS,false) {d.worker={system=&w.js,worker=jobs.OWNER}} + if d.peers==0 {d.peers=4} + n.open=init(&n.x,d)==.None + return n.open +} + +@(private) +test_node_close :: proc(n:^Test_Node) { + if n.open {destroy(&n.x)} + store.close(&n.s) + n.open=false +} + +@(private) +test_world_destroy :: proc(w:^Test_World) { + for &n in w.nodes[:w.count] { + test_node_close(&n) + for h in n.hosts[:n.host_count] {wire.host_destroy(h);free(h)} + } + wire.sim_network_destroy(&w.sim) + jobs.system_destroy(&w.js) +} + +@(private) +test_host :: proc(w:^Test_World,k:int,mode:wire.Mode) -> ^wire.Host { + n:=&w.nodes[k] + h:=new(wire.Host) + endpoint:=w.endpoints + w.endpoints+=1 + if wire.host_init(h,{mode=mode,transport=wire.sim_endpoint_transport(&w.sim,endpoint),channels=w.channels,protocol_id=91,seed=u64(100+endpoint)})!=.None { + free(h) + return nil + } + n.hosts[n.host_count]=h + n.endpoints[n.host_count]=endpoint + n.host_count+=1 + return h +} + +// Connects the client host of node k to the server host at endpoint; false if it does not within 5 s. +@(private) +test_connect :: proc(w:^Test_World,client:^wire.Host,endpoint:int) -> bool { + wire.host_connect(client,wire.address_memory(endpoint)) + for _ in 0..<5000 { + test_step(w) + if stats,ok:=wire.connection_stats(client,client.connection_handles[0]);ok && client.connection_handles[0]!=0 { + _=stats + return true + } + } + return false +} + +@(private) +test_step :: proc(w:^Test_World,steps:=1) { + for _ in 0..=n.game.since {n.game.sent+=1} + } +} + +@(private) +test_bytes :: proc(size:int,seed:u64) -> []u8 { + data:=make([]u8,size,context.temp_allocator) + h:=base.Hostile{state=seed} + for &b in data {b=u8(base.hostile_next(&h))} + return data +} + +// Whether node k holds the object at address whole: as its want says once +// committed, or read whole from its store. +@(private) +test_here :: proc(n:^Test_Node,address:store.Hash) -> bool { + if n.open { + if p,known:=progress(&n.x,address);known {return p.state==.Complete} + } + _,whole:=test_read(&n.s,address) + return whole +} + +// Steps until every address is complete at node k or seconds pass. +@(private) +test_until_complete :: proc(w:^Test_World,k:int,addresses:[]store.Hash,seconds:f64) -> bool { + end:=w.now+seconds + for w.now State { + p,_:=progress(x,address) + return p.state +} + +@(private) +test_script_tick :: proc(w:^Test_World,n:^Test_Node) { + if n.lie==.Late_Done && n.held>0 && w.now>=n.held_at { + done:=[1]u8{MSG_DONE} + for _ in 0.. bool { + node,error:=store.node_parse(bytes) + if error!=.None || depth>store.NODE_LEVEL_MAX {return true} + held:=make([]u8,len(bytes),context.temp_allocator) + copy(held,bytes) + node,_=store.node_parse(held) + buffer:=make([]u8,store.CHUNK_MAX,context.temp_allocator) + for e in node.entries { + child,read_error:=store.chunk_read(sc.n.source,e.address,buffer) + if read_error!=.None {return false} + if sc.n.lie!=.Over_Budget && sc.sent+i64(len(child))>sc.budget {return false} + sc.sent+=i64(len(child)) + send_chunk(sc,e.address,child) + if node.level>0 && !descend(sc,child,depth+1) {return false} + } + return true + } + r:=Reader{data=command.data[MSG_ASK_HEAD:],ok=true} + count:=int(get_u16(command.data[1:])) + sc:=Script{w=w,n=n,h=h,c=command.connection,out=make([]u8,MSG_CHUNK_MAX,context.temp_allocator)} + buffer:=make([]u8,store.CHUNK_MAX,context.temp_allocator) + for _ in 0..0 || n.lie==.Over_Budget {descend(&sc,bytes,0)} + } + done:=[1]u8{MSG_DONE} + wire.message_send(h,command.connection,w.channel+LANE_DATA,done[:]) +} diff --git a/exchange/hostile_test.odin b/exchange/hostile_test.odin new file mode 100644 index 000000000..b4b638157 --- /dev/null +++ b/exchange/hostile_test.odin @@ -0,0 +1,218 @@ +package exchange + +import "core:testing" +import "mirai:std/base" +import "mirai:store" +import wire "mirai:net" + +@(private) +test_message :: proc(parts:..[]u8) -> []u8 { + n:=0 + for p in parts {n+=len(p)} + out:=make([]u8,n,context.temp_allocator) + at:=0 + for p in parts {copy(out[at:],p);at+=len(p)} + return out +} + +@(private) +test_le :: proc(v:u64,bytes:int) -> []u8 { + out:=make([]u8,bytes,context.temp_allocator) + for i in 0..>(8*uint(i)))} + return out +} + +@(private) +test_name :: proc(s:string) -> []u8 {return test_message({u8(len(s))},transmute([]u8)s)} + +@(private) +test_ask :: proc(entries:..store.Hash) -> []u8 { + out:=test_message({MSG_ASK},test_le(u64(len(entries)),2)) + for e in entries { + e:=e + out=test_message(out,e[:],test_le(0,4)) + } + return out +} + +@(private) +test_chunk :: proc(address:store.Hash,codec:u8,raw:int,bytes:[]u8) -> []u8 { + address:=address + return test_message({MSG_CHUNK},address[:],{codec},test_le(u64(raw),4),bytes) +} + +// Every message a peer could send is validated: the malformed ones mark the +// peer bad, the well formed ones nobody asked for are ignored, and a peer +// cannot queue more than BATCHES asks. None crashes, and random bytes on +// both channels do not either. +@(test) +hostile_messages_are_refused_without_a_crash :: proc(t:^testing.T) { + w:Test_World + defer test_world_destroy(&w) + if !test_world(t,&w,"exchange-hostile",2,desc={expose=test_expose_lib}) {return} + server:=test_host(&w,0,.Server) + client:=test_host(&w,1,.Client) + if !testing.expect(t,test_connect(&w,client,w.nodes[0].endpoints[0])) {return} + x:=&w.nodes[0].x + c:=server.connection_handles[0] + if c==0 {c=server.connection_handles[1]} + data:=test_bytes(3*store.CHUNK_MAX+5,1) + root:=test_object(t,&w.nodes[0].s,data) + other:=store.address_of_bytes(test_bytes(5,2)) + want(x,other) + zero:store.Hash + big:=make([]u8,store.CHUNK_MAX+1,context.temp_allocator) + Case :: struct {lane:int,data:[]u8,bad:bool} + cases:=[]Case{ + {LANE_CONTROL,nil,true}, + {LANE_CONTROL,{99},true}, + {LANE_DATA,{99},true}, + {LANE_CONTROL,{MSG_ASK},true}, + {LANE_CONTROL,test_ask(),true}, + {LANE_CONTROL,test_message({MSG_ASK},test_le(ASK_ENTRIES+1,2)),true}, + {LANE_CONTROL,test_message(test_ask(root),{0}),true}, + {LANE_CONTROL,test_message({MSG_BAD},root[:5]),true}, + {LANE_CONTROL,test_message({MSG_BAD},root[:],{0}),true}, + {LANE_CONTROL,test_message({MSG_REF_GET},test_name("lib")),true}, + {LANE_CONTROL,test_message({MSG_REF_GET},test_name("Lib"),test_name("a")),true}, + {LANE_CONTROL,test_message({MSG_REF_GET},test_name("lib"),{200},big[:10]),true}, + {LANE_CONTROL,test_message({MSG_NAMES_GET},test_name("lib"),test_name("../x")),true}, + {LANE_CONTROL,test_message({MSG_REF},test_name("lib"),test_name("a"),{2},zero[:]),true}, + {LANE_CONTROL,test_message({MSG_NAMES},test_name("lib"),{0},{0},test_le(3,2)),true}, + {LANE_CONTROL,test_message({MSG_NAMES},test_name("lib"),{0},{0},test_le(1,2),test_name("a")),true}, + {LANE_DATA,test_message({MSG_CHUNK},root[:]),true}, + {LANE_DATA,test_chunk(root,1,10,big[:10]),true}, + {LANE_DATA,test_chunk(root,0,11,big[:10]),true}, + {LANE_DATA,test_chunk(root,0,0,nil),true}, + {LANE_DATA,test_chunk(root,0,len(big),big),true}, + // Encoded: in a codec this end did not say it decodes, of an id past the last, no + // smaller than raw, or of a raw size past a chunk's. + {LANE_DATA,test_chunk(root,15,20,big[:10]),true}, + {LANE_DATA,test_chunk(root,store.CODECS_MAX,20,big[:10]),true}, + {LANE_DATA,test_chunk(root,255,20,big[:10]),true}, + {LANE_DATA,test_chunk(root,1,9,big[:10]),true}, + {LANE_DATA,test_chunk(root,1,store.CHUNK_MAX+1,big[:10]),true}, + {LANE_DATA,test_chunk(root,1,int(max(u32)),big[:10]),true}, + {LANE_CONTROL,{MSG_CODECS,1},true}, + {LANE_CONTROL,{MSG_CODECS,1,0,0},true}, + {LANE_DATA,test_message({MSG_LACK},root[:4]),true}, + {LANE_DATA,test_message({MSG_LACK},root[:],{0}),true}, + {LANE_DATA,test_message({4},root[:],root[:]),true}, // an unknown kind + {LANE_DATA,{MSG_DONE},true}, + {LANE_DATA,{MSG_DONE,0},true}, + // Nobody asked: ignored, counted against the peer. + {LANE_DATA,test_chunk(root,0,10,big[:10]),false}, + {LANE_DATA,test_chunk(root,1,20,big[:10]),false}, + {LANE_CONTROL,{MSG_CODECS,0xff,0xff},false}, + {LANE_DATA,test_chunk(other,0,5,test_bytes(5,2)),false}, + {LANE_DATA,test_message({MSG_LACK},other[:]),false}, + {LANE_CONTROL,test_message({MSG_REF},test_name("lib"),test_name("a"),{1},zero[:]),false}, + {LANE_CONTROL,test_message({MSG_NAMES},test_name("lib"),{0},{0},test_le(0,2)),false}, + {LANE_CONTROL,test_message({MSG_REF_GET},test_name("secret"),test_name("a")),false}, + {LANE_CONTROL,test_message({MSG_NAMES_GET},test_name("lib"),{0}),false}, + {LANE_CONTROL,test_message({MSG_BAD},other[:]),false}, + {LANE_CONTROL,test_ask(root,zero),false}, + } + for c_,k in cases { + receive(x,server,{kind=.Message,channel=u8(w.channel+c_.lane),connection=c,data=c_.data}) + p,_:=peer_progress(x,server,c) + testing.expectf(t,p.bad==c_.bad,"case %d: bad %v",k,p.bad) + if p.bad { + peer_remove(x,server,c) + peer_add(x,server,c) + } + } + q,_:=progress(x,other) + testing.expect(t,q.state==.Asking && q.total==-1,"an unasked chunk is not taken") + // A message that arrives in pieces past what this end holds. + piece:=make([]u8,wire.PIECE_BYTES,context.temp_allocator) + piece[0]=MSG_NAMES + for _ in 0..<8 {receive(x,server,{kind=.Message,channel=u8(w.channel+LANE_CONTROL),connection=c,data=piece,more=true})} + receive(x,server,{kind=.Message,channel=u8(w.channel+LANE_CONTROL),connection=c,data=piece[:1]}) + p,_:=peer_progress(x,server,c) + testing.expect(t,p.bad,"an oversized message marks the peer") + peer_remove(x,server,c) + peer_add(x,server,c) + // At most BATCHES asks queue; the next is refused, not served. + full:=make([]store.Hash,ASK_ENTRIES,context.temp_allocator) + for &e in full {e=root} + ask:=test_ask(..full) + for _ in 0..0 && base.hostile_int(&h,2)==0 {noise[0]=u8(1+base.hostile_int(&h,6))} + receive(x,server,{kind=.Message,channel=u8(w.channel+base.hostile_int(&h,CHANNELS)),connection=c,data=noise[:n],more=base.hostile_int(&h,8)==0}) + if p,_=peer_progress(x,server,c);p.bad { + peer_remove(x,server,c) + peer_add(x,server,c) + } + } + test_step(&w,500) + testing.expect(t,!test_here(&w.nodes[0],other),"nothing unverified completes a want") +} + +// Trees a peer holds whose chunks hash to their addresses but do not hold +// together: a node of the wrong level, or whose bytes disagree with its +// parent, or a wanted root that is no node, fails the want. +@(test) +hostile_trees_fail_the_want_without_a_crash :: proc(t:^testing.T) { + w:Test_World + defer test_world_destroy(&w) + if !test_pair(t,&w,"exchange-trees") {return} + s:=&w.nodes[0].s + put :: proc(s:^store.Store,bytes:[]u8) -> store.Hash { + address:=store.address_of_bytes(bytes) + store.chunk_put(s,address,bytes) + return address + } + node :: proc(s:^store.Store,level:int,entries:[]store.Entry,groups:[]store.Group={}) -> (store.Hash,u64) { + buffer:=make([]u8,store.CHUNK_MAX,context.temp_allocator) + bytes,_:=store.node_encode(buffer,level,entries,groups) + total:u64 + for e in entries {total+=e.size} + return put(s,bytes),total + } + junk:=test_bytes(300,3) + nonsense:=put(s,junk) + data:=test_bytes(1000,4) + leaf_chunk:=put(s,data) + leaf,leaf_total:=node(s,0,{{address=leaf_chunk,size=1000}}) + // A level-2 root over a node of level 0. + skipped,_:=node(s,2,{{address=leaf,size=leaf_total}},{{count=1}}) + // A level-1 root whose entry says more bytes than its child holds. + wrong_size,_:=node(s,1,{{address=leaf,size=leaf_total+1}},{{count=1}}) + // A leaf whose chunk is shorter than its entry says. + short,_:=node(s,0,{{address=leaf_chunk,size=999}},{{count=1}}) + store.commit(s) + x:=&w.nodes[1].x + for a in ([]store.Hash{nonsense,skipped,wrong_size,short}) {want(x,a)} + test_step(&w,1000) + for a in ([]store.Hash{nonsense,skipped,wrong_size,short}) { + p,_:=progress(x,a) + testing.expectf(t,p.state==.Failed && p.store_error==.Corrupt,"a tree that does not hold together fails: %v",p) + } + testing.expect(t,!test_peer(&w.nodes[1]).bad,"the peer that holds them is not bad: its chunks make their addresses") +} + +// A peer that sends descendants past the budget of an ask is bad. +@(test) +a_peer_past_the_budget_of_an_ask_is_bad :: proc(t:^testing.T) { + w:Test_World + defer test_world_destroy(&w) + if !test_world(t,&w,"exchange-budget",2) {return} + server:=test_host(&w,0,.Server) + flood:=test_scripted(t,&w,1,.Over_Budget) + if !testing.expect(t,test_connect(&w,flood,w.nodes[0].endpoints[0])) {return} + root:=test_object(t,&w.nodes[1].s,test_bytes(2<<20,5)) + want(&w.nodes[0].x,root) + test_step(&w,500) + p,_:=peer_progress(&w.nodes[0].x,server,server.connection_handles[0]) + testing.expect(t,p.bad,"the flooding peer is bad") + testing.expect(t,!test_here(&w.nodes[0],root),"and what it sent past the budget is not taken whole") +} diff --git a/exchange/rate_bench_test.odin b/exchange/rate_bench_test.odin new file mode 100644 index 000000000..e4b031851 --- /dev/null +++ b/exchange/rate_bench_test.odin @@ -0,0 +1,208 @@ +package exchange + +import "core:fmt" +import "core:os" +import "core:sync" +import "core:sys/posix" +import "core:testing" +import "core:thread" +import "core:time" +import "mirai:std/base" +import "mirai:std/jobs" +import "mirai:store" +import wire "mirai:net" + +// Opt-in: mirai test exchange objects_over_udp_loopback -define:EXCHANGE_BENCHMARK=true +// Two stores, each with a host on 127.0.0.1 whose worker thread runs the socket +// and a thread that pumps its exchange. The receiver wants one large object, +// then many small ones; it prints the rate and the CPU of each side per GiB, +// counting the pump and the worker of each, and asserts no speed. +@(test) +objects_over_udp_loopback :: proc(t:^testing.T) { + when #config(EXCHANGE_BENCHMARK,false) { + testing.set_fail_timeout(t,20*time.Minute) + folder:=base.test_folder(t,"exchange-bench") + if folder=="" {return} + stores:[2]store.Store + for &s,i in stores { + opened,error:=store.open(fmt.tprintf("%s/side-%d",folder,i)) + if !testing.expect_value(t,error,store.Error.None) {return} + s=opened + } + defer for &s in stores {store.close(&s)} + large:=[1]store.Hash{bench_large(&stores[0],2<<30 if #config(EXCHANGE_BENCHMARK_LARGE,true) else 1<<20)} + small:=make([]store.Hash,5000) + defer delete(small) + small_bytes:=0 + state:=u64(9) + buffer:=make([]u8,64<<10) + defer delete(buffer) + for &a,k in small { + size:=1024+int(base.splitmix_next(&state)%(63<<10)) + for i:=0;i+8<=size;i+=8 {(^u64)(&buffer[i])^=base.splitmix_next(&state)} + a,_=store.put(&stores[0],buffer[:size]) + small_bytes+=size + _=k + } + store.commit(&stores[0]) + if #config(EXCHANGE_BENCHMARK_LARGE,true) {bench_case(t,stores[:],large[:],2<<30,"large")} + bench_case(t,stores[:],small,small_bytes,"small") + } +} + +when #config(EXCHANGE_BENCHMARK,false) { + +// An object of size bytes of noise, committed. +@(private="file") +bench_large :: proc(s:^store.Store,size:int) -> store.Hash { + p,_:=store.put_begin(s) + block:=make([]u8,4<<20) + defer delete(block) + state:=u64(5) + for done:=0;done>20,seconds,f64(bytes)/f64(1<<20)/seconds,f64(len(wants))/seconds, + (sides[0].cpu+sides[0].jobs_cpu+worker[0])/gib,(sides[1].cpu+sides[1].jobs_cpu+worker[1])/gib, + (sides[0].cpu+sides[0].jobs_cpu)/gib,(sides[1].cpu+sides[1].jobs_cpu)/gib,worker[0]/gib,worker[1]/gib, + sides[0].cpu,sides[0].jobs_cpu,sides[1].cpu,sides[1].jobs_cpu) + testing.expectf(t,sides[1].done,"%s: everything arrives",name) +} + +// The pump of one side: polls its host and updates its exchange; the receiver +// wants its objects and stops once all are complete. +@(private="file") +bench_pump :: proc(s:^Bench_Side) { + // The pump owns the job system its exchange hands reading, hashing and storing to. + js:jobs.Job_System + jobs.system_init(&js,{threads=#config(BENCH_THREADS,2)}) + s.x.desc.worker={system=&js,worker=jobs.OWNER} + defer { + sync.mutex_lock(&s.x.lock) + work_wait_all(s.x) + s.x.desc.worker={} + sync.mutex_unlock(&s.x.lock) + for &w in js.workers[1:] {s.jobs_cpu+=bench_thread_cpu(w.thread)} + jobs.system_destroy(&js) + s.cpu=bench_own_cpu() + } + for w in s.wants {want(s.x,w)} + start:=time.tick_now() + next:=0 + for !sync.atomic_load(s.stop) { + for command in wire.host_poll(s.host) { + if command.kind==.Message {receive(s.x,s.host,command)} + } + update(s.x,time.duration_seconds(time.tick_since(start))) + for next0 && next==len(s.wants) {sync.atomic_store(&s.done,true)} + free_all(context.temp_allocator) + time.sleep(100*time.Microsecond) + } +} + +@(private="file") +bench_own_cpu :: proc() -> f64 { + spec:posix.timespec + posix.clock_gettime(.THREAD_CPUTIME_ID,&spec) + return f64(spec.tv_sec)+f64(spec.tv_nsec)*1e-9 +} + +when ODIN_OS==.Darwin { + foreign import system "system:System.framework" + @(private="file") Thread_Basic_Info :: struct {user_s,user_us,system_s,system_us,cpu_usage,policy,run_state,flags,suspend_count,sleep_time:i32} + @(private="file",default_calling_convention="c") + foreign system { + pthread_mach_thread_np :: proc(thread:posix.pthread_t) -> u32 --- + thread_info :: proc(thread:u32,flavor:i32,info:^Thread_Basic_Info,count:^u32) -> i32 --- + } +} + +// CPU seconds of another thread; 0 where the bench does not know how to ask. +@(private="file") +bench_thread_cpu :: proc(th:^thread.Thread) -> f64 { + if th==nil {return 0} + when ODIN_OS==.Darwin { + info:Thread_Basic_Info + count:=u32(size_of(info)/4) + if thread_info(pthread_mach_thread_np(th.unix_thread),3,&info,&count)!=0 {return 0} + return f64(info.user_s+info.system_s)+f64(info.user_us+info.system_us)*1e-6 + } else { + return 0 + } +} + +} diff --git a/exchange/serve.odin b/exchange/serve.odin new file mode 100644 index 000000000..68d3ca8f7 --- /dev/null +++ b/exchange/serve.odin @@ -0,0 +1,266 @@ +package exchange + +import "core:slice" +import "core:sync" +import "mirai:store" +import wire "mirai:net" + +// An ask is taken whole or refused whole: a peer keeps at most BATCHES in +// flight, so one with more is not one this end needs to answer. Nothing is +// checked on serve: the receiver checks every chunk and reports a bad one. +@(private) +on_ask :: proc(x:^Exchange,i:int,data:[]u8) { + p:=&x.peers[i] + r:=Reader{data=data[1:],ok=true} + count:=int(read_u16(&r)) + if count==0 || count>ASK_ENTRIES || len(data)!=MSG_ASK_HEAD+count*ASK_ENTRY_BYTES {peer_bad(x,i);return} + tail:=p.serve_tail + if tail-sync.atomic_load(&p.serve_head)+count+1>len(p.serve) {p.refused+=1;return} + for k in 0.. bool { + return x.desc.expose!=nil && x.desc.expose(x.desc.expose_user,namespace) +} + +// An answer that finds no room is dropped: a peer keeps at most +// QUERY_CREDITS of questions waiting, whose answers always fit. +@(private) +on_ref_get :: proc(x:^Exchange,i:int,data:[]u8) { + r:=Reader{data=data[1:],ok=true} + namespace:=read_name(&r) + name:=read_name(&r) + if !read_end(&r) {peer_bad(x,i);return} + address:store.Hash + found:=false + if exposed(x,namespace) { + hash,error:=store.ref_get(x.desc.store,namespace,name) + address,found=hash,error==.None + } + buffer:[MSG_REF_MAX]u8 + w:=Writer{data=buffer[:]} + write_u8(&w,MSG_REF) + write_name(&w,namespace) + write_name(&w,name) + write_u8(&w,found?1:0) + write_bytes(&w,address[:]) + if !control_append(&x.peers[i],buffer[:w.n]) {x.peers[i].refused+=1} +} + +// The names after after, sorted. Reading a namespace is the one answer the +// store allocates for. +@(private) +on_names_get :: proc(x:^Exchange,i:int,data:[]u8) { + r:=Reader{data=data[1:],ok=true} + namespace:=read_name(&r) + after:=read_name(&r,empty=true) + if !read_end(&r) {peer_bad(x,i);return} + refs:[]store.Ref + if exposed(x,namespace) { + listed,error:=store.ref_list(x.desc.store,namespace,x.allocator) + if error==.None {refs=listed} + } + defer store.ref_list_delete(refs,x.allocator) + slice.sort_by(refs,proc(a,b:store.Ref) -> bool {return a.nameNAMES_REPLY_BYTES { + buffer[more_at]=1 + break + } + write_name(&w,ref.name) + write_bytes(&w,ref.hash[:]) + count+=1 + } + put_u16(buffer[count_at:],u16(count)) + if !control_append(&x.peers[i],buffer[:w.n]) {x.peers[i].refused+=1} +} + +// Sends the answers the serve job of peer i made, in order, while its data +// channel has room, and hands the job more when it is idle. +@(private) +serve_pump :: proc(x:^Exchange,i:int) { + p:=&x.peers[i] + for { + head:=p.send_head + if head>8),u8(n>>16),u8(n>>24) + sync.atomic_store(&p.send_tail,tail+4+(n+3)&~3) + } +} + +// The chunk at address read into out after its Chunk head, in a codec the +// peer decodes: the size of the message, or why not. +@(private) +serve_read :: proc(x:^Exchange,p:^Peer,out:[]u8,address:store.Hash) -> (size:int,error:store.Error) { + room:=out[MSG_CHUNK_HEAD:] + bytes,codec,raw:=store.chunk_read_in(x.desc.store,address,p.decodes,room,p.scratch) or_return + if raw_data(bytes)!=raw_data(room) {copy(room,bytes)} + encode_chunk_head(out,address,codec,raw) + return MSG_CHUNK_HEAD+len(bytes),.None +} + +// The bytes of a Chunk out holds, when they are a node as they are. +@(private) +chunk_node :: proc(out:[]u8,size:int) -> (n:store.Node_View,ok:bool) { + if out[1+32]!=store.CODEC_NONE {return} + parsed,error:=store.node_parse(out[MSG_CHUNK_HEAD:size]) + return parsed,error==.None +} + +// The node at address of a stream, held until another is read. +@(private) +serve_node :: proc(x:^Exchange,p:^Peer,address:store.Hash) -> (n:store.Node_View,error:store.Error) { + if p.node_held && p.node_address==address {return p.node_view,.None} + p.node_held=false + bytes:=store.chunk_read(x.desc.store,address,p.node) or_return + if raw_data(bytes)!=raw_data(p.node) {copy(p.node,bytes)} + n=store.node_parse(p.node[:len(bytes)]) or_return + p.node_address,p.node_view,p.node_held=address,n,true + return n,.None +} + +// Writes the next answer of a peer into out: its size, 0 when there is none. +// Each entry is its chunk or a Lack; then, while their bytes fit its budget, the descendants of that chunk +// depth first, until one is missing. +@(private) +serve_build :: proc(x:^Exchange,p:^Peer,out:[]u8) -> int { + for p.serve_head0 { + if _,node:=chunk_node(out,size);node { + p.stack[0]={address=e.address} + p.depth=1 + } + } + if p.depth==0 {serve_pop(p)} + return size + } + for p.depth>0 { + f:=&p.stack[p.depth-1] + n,error:=serve_node(x,p,f.address) + if error!=.None || f.index>=len(n.entries) { + p.depth-=1 + continue + } + child:=n.entries[f.index] + f.index+=1 + size,read_error:=serve_read(x,p,out,child.address) + if read_error!=.None || p.sent_under+i64(size-MSG_CHUNK_HEAD)>i64(e.budget) { + p.depth=0 + break + } + p.sent_under+=i64(size-MSG_CHUNK_HEAD) + if n.level>0 && p.depth>8)} +@(private) +put_u32 :: proc(b:[]u8,v:u32) {for i in 0..<4 {b[i]=u8(v>>(8*uint(i)))}} +@(private) +get_u16 :: proc(b:[]u8) -> u16 {return u16(b[0])|u16(b[1])<<8} + +@(private) +Writer :: struct {data:[]u8,n:int} + +@(private) +write_u8 :: proc(w:^Writer,v:u8) {w.data[w.n]=v;w.n+=1} +@(private) +write_u16 :: proc(w:^Writer,v:u16) {put_u16(w.data[w.n:],v);w.n+=2} +@(private) +write_u32 :: proc(w:^Writer,v:u32) {put_u32(w.data[w.n:],v);w.n+=4} +@(private) +write_bytes :: proc(w:^Writer,v:[]u8) {copy(w.data[w.n:],v);w.n+=len(v)} +@(private) +write_name :: proc(w:^Writer,v:string) {write_u8(w,u8(len(v)));write_bytes(w,transmute([]u8)v)} + +// Reads fields of a message; any read past its end leaves ok false and gives zeros. +@(private) +Reader :: struct {data:[]u8,at:int,ok:bool} + +@(private) +read_bytes :: proc(r:^Reader,n:int) -> []u8 { + if !r.ok || n<0 || n>len(r.data)-r.at {r.ok=false;return nil} + b:=r.data[r.at:][:n] + r.at+=n + return b +} +@(private) +read_u8 :: proc(r:^Reader) -> u8 { + b:=read_bytes(r,1) + return r.ok?b[0]:0 +} +@(private) +read_u16 :: proc(r:^Reader) -> u16 { + b:=read_bytes(r,2) + return r.ok?get_u16(b):0 +} +@(private) +read_u32 :: proc(r:^Reader) -> (v:u32) { + b:=read_bytes(r,4) + if r.ok {for i in 0..<4 {v|=u32(b[i])<<(8*uint(i))}} + return +} +@(private) +read_hash :: proc(r:^Reader) -> (h:store.Hash) { + b:=read_bytes(r,32) + if r.ok {copy(h[:],b)} + return +} +// A ref name, or with empty an empty one too; anything else leaves ok false. +@(private) +read_name :: proc(r:^Reader,empty:=false) -> string { + n:=int(read_u8(r)) + b:=read_bytes(r,n) + if !r.ok {return ""} + s:=string(b) + if !(empty && n==0) && !store.name_valid(s) {r.ok=false;return ""} + return s +} +// Whether the whole message was read and nothing else is left. +@(private) +read_end :: proc(r:^Reader) -> bool {return r.ok && r.at==len(r.data)} + +@(private) +encode_question :: proc(buffer:[]u8,kind:u8,namespace,name:string) -> []u8 { + w:=Writer{data=buffer} + write_u8(&w,kind) + write_name(&w,namespace) + write_name(&w,name) + return buffer[:w.n] +} + +// A Lack or a Bad. +@(private) +encode_address :: proc(buffer:[]u8,kind:u8,address:store.Hash) -> int { + address:=address + buffer[0]=kind + copy(buffer[1:],address[:]) + return MSG_ADDRESS_BYTES +} + +// The head of a Chunk whose bytes, in codec, follow it in buffer. +@(private) +encode_chunk_head :: proc(buffer:[]u8,address:store.Hash,codec:u8,raw:int) { + address:=address + w:=Writer{data=buffer} + write_u8(&w,MSG_CHUNK) + write_bytes(&w,address[:]) + write_u8(&w,codec) + write_u32(&w,u32(raw)) +} + +// One whole message from peer i on a lane. +@(private) +message :: proc(x:^Exchange,i:int,lane:int,data:[]u8) { + if len(data)==0 {peer_bad(x,i);return} + if lane==LANE_DATA { + switch data[0] { + case MSG_CHUNK: on_chunk(x,i,data) + case MSG_LACK: on_lack(x,i,data) + case MSG_DONE: on_done(x,i,data) + case: peer_bad(x,i) + } + return + } + switch data[0] { + case MSG_ASK: on_ask(x,i,data) + case MSG_BAD: on_bad(x,i,data) + case MSG_CODECS: on_codecs(x,i,data) + case MSG_REF_GET: on_ref_get(x,i,data) + case MSG_NAMES_GET: on_names_get(x,i,data) + case MSG_REF: on_ref(x,i,data) + case MSG_NAMES: on_names(x,i,data) + case: peer_bad(x,i) + } +} diff --git a/exchange/work.odin b/exchange/work.odin new file mode 100644 index 000000000..057b58be7 --- /dev/null +++ b/exchange/work.odin @@ -0,0 +1,641 @@ +package exchange + +import "core:sync" +import "mirai:std/jobs" +import "mirai:store" + +// What runs on the jobs of the owner's worker: checking and storing received +// chunks, walking trees in the store, serving, committing and checking a copy +// a receiver reported. Each job owns a slot sized at init and touches nothing +// else of the exchange; the pump hands work to a slot and applies its result +// under the lock, so a job never takes the lock and the pump may wait for one. +// Without a job system each job runs on the pump when it is handed out. + +@(private) +Walk_Found :: struct {address:store.Hash,kind:Need_Kind,level:int,size:u64} + +// A depth-first walk over what a store holds of a tree, which records the +// chunks it lacks: data chunks as it passes them, and a missing node, with the +// missing nodes after it, where it stops. +@(private) +Walker :: struct { + s:^store.Store, + stack:[WALK_DEPTH]Walk_Frame, + depth:int, + root:bool, // the bottom of the stack is the root of the object + found:[]Walk_Found, + count,limit:int, + here:i64, // data bytes it found stored + total:i64, // of the root, once read; -1 + walked:bool, // the stack emptied + blocked:bool, // stopped at the missing node of the top frame's entry + waiting:store.Hash, // that node + error:store.Error, // Missing: a node it walked is gone; Corrupt: one does not hold together + buffer:[]u8, // a node read from the store; nil: it reads none and stops where it would + unread:bool, // it stopped where it would read the store + given:store.Hash, // a node whose bytes are at hand + given_bytes:[]u8, + viewing:bool, + view_address:store.Hash, + view:store.Node_View, +} + +@(private) +walker_begin :: proc(k:^Walker,address:store.Hash,root:bool,limit:int,given:[]u8={}) { + k.stack[0]={address=address} + // Room is left for the missing node it may stop at and the ones it asks ahead. + k.depth,k.root,k.count,k.limit,k.here,k.total=1,root,0,min(limit,len(k.found)-PREFETCH_NODES-1),0,-1 + k.walked,k.blocked,k.error,k.viewing,k.unread=false,false,.None,false,false + k.given,k.given_bytes=address,given +} + +@(private) +walker_node :: proc(k:^Walker,address:store.Hash) -> (n:store.Node_View,error:store.Error) { + if k.viewing && k.view_address==address {return k.view,.None} + k.viewing=false + bytes:[]u8 + if address==k.given && k.given_bytes!=nil { + bytes=k.given_bytes + } else { + if k.buffer==nil { + k.unread=true + return {},.Missing + } + bytes=store.chunk_read(k.s,address,k.buffer) or_return + if raw_data(bytes)!=raw_data(k.buffer) {copy(k.buffer,bytes)} + bytes=k.buffer[:len(bytes)] + } + n=store.node_parse(bytes) or_return + k.view_address,k.view,k.viewing=address,n,true + return n,.None +} + +@(private) +walker_found :: proc(k:^Walker,kind:Need_Kind,address:store.Hash,level:int,size:u64) { + if k.count0 { + if k.count>=k.limit {return} + f:=&k.stack[k.depth-1] + n,error:=walker_node(k,f.address) + if k.unread {return} + if error!=.None { + k.error=.Missing if error==.Missing else .Corrupt + return + } + if k.root && k.depth==1 && k.total<0 {k.total=i64(n.total)} + if f.index>=len(n.entries) { + k.depth-=1 + continue + } + e:=n.entries[f.index] + if n.level==0 { + f.index+=1 + if store.chunk_here(k.s,e.address) {k.here+=i64(e.size)} else {walker_found(k,.Data,e.address,0,e.size)} + continue + } + if store.chunk_here(k.s,e.address) { + level:=n.level + child,child_error:=walker_node(k,e.address) + if k.unread {return} + if child_error!=.None || child.level!=level-1 || child.total!=e.size || len(child.groups)>0 || len(child.meta)>0 || k.depth==WALK_DEPTH { + k.error=.Corrupt + return + } + f.index+=1 + k.stack[k.depth]={address=e.address} + k.depth+=1 + continue + } + walker_found(k,.Node,e.address,n.level-1,e.size) + // The next missing nodes are asked now, so they are here when the walk reaches them. + ahead:=0 + for next in n.entries[f.index+1:] { + if ahead==PREFETCH_NODES {break} + ahead+=1 + if !store.chunk_here(k.s,next.address) {walker_found(k,.Node,next.address,n.level-1,next.size)} + } + k.blocked,k.waiting=true,e.address + return + } + k.walked=true +} + +@(private) +Receive_Result :: enum u8 { + Stored, // checked and stored; a node also walked from + Mismatch, // it does not hash to its address + Corrupt, // it hashes right but does not hold together with its parent + Failed, // the store refused it + Checked, // a descendant whose parent was not parsed when it came: hashed, not stored +} + +// A received chunk on its way to the store. One the ask did not name and +// whose parent was not parsed yet (a descendant streamed under a budget) is +// only hashed and parsed; once its parent named it, it goes again to be stored. +@(private) +Receive_Slot :: struct { + x:^Exchange, + bytes:[]u8, // in the ring of received bytes + offset:int, + size:int, + address:store.Hash, + need:i32, // -1 while its parent was not parsed + serial:u32, + want:i32, + generation:u32, + peer:int, + epoch:u32, + codec:u8, // of the bytes received after the chunk's, stored of them + stored:int, + kind:Need_Kind, + level:int, + entry:u64, // data bytes its parent says are under it + limit:int, // chunks its walk may find + result:Receive_Result, + error:store.Error, + parsed:bool, // its bytes parse as a node, of level and total + parsed_level:int, + parsed_total:u64, + parsed_plain:bool, // without groups or meta + walk:^Walker, // a received node's: it walks from it + light:Walker, // its own, which reads nothing + counter:jobs.Counter, + done:bool, +} + +@(private) +receive_job :: proc(data:rawptr,worker:int) { + r:=(^Receive_Slot)(data) + defer sync.atomic_store(&r.done,true) + bytes:=r.bytes[:r.size] + stored:=r.bytes[r.size:][:r.stored] + s:=r.x.desc.store + // A frame is kept as it came only once the store decoded it and checked it against the + // address: one decode on the way of a chunk asked, two for a descendant that came before its + // parent was parsed. + put:=false + if r.codec!=store.CODEC_NONE && r.need>=0 { + switch e:=store.chunk_put_frame(s,r.address,r.codec,stored,bytes);e { + case .None: put=true + case .Corrupt,.Missing,.Invalid_Argument: + r.result=.Mismatch + return + case .Io,.No_Space,.Out_Of_Memory,.Unsupported_Layout: + r.error,r.result=e,.Failed + return + } + } + if r.result!=.Checked { + if r.codec!=store.CODEC_NONE && !put && store.chunk_decode(s,r.codec,stored,bytes)!=.None { + r.result=.Mismatch + return + } + if store.address_of_bytes(bytes)!=r.address { + r.result=.Mismatch + return + } + if n,error:=store.node_parse(bytes);error==.None { + r.parsed,r.parsed_level,r.parsed_total,r.parsed_plain=true,n.level,n.total,len(n.groups)==0 && len(n.meta)==0 + } + if r.need<0 { + r.result=.Checked + return + } + } + switch r.kind { + case .Root: + if !r.parsed { + r.result=.Corrupt + return + } + case .Node: + if !r.parsed || r.parsed_level!=r.level || r.parsed_total!=r.entry || !r.parsed_plain { + r.result=.Corrupt + return + } + case .Data: + if u64(r.size)!=r.entry { + r.result=.Corrupt + return + } + } + if !put { + if r.error=store.chunk_put(s,r.address,bytes);r.error!=.None { + r.result=.Failed + return + } + } + r.result=.Stored + if r.kind!=.Data && r.walk!=nil { + walker_begin(r.walk,r.address,r.kind==.Root,r.limit,bytes) + walker_run(r.walk) + } +} + +// A walk of a want over the store, from its stack. +@(private) +Walk_Slot :: struct { + x:^Exchange, + used:bool, + want:i32, + generation:u32, + walk:Walker, + counter:jobs.Counter, + done:bool, +} + +@(private) +walk_job :: proc(data:rawptr,worker:int) { + k:=(^Walk_Slot)(data) + defer sync.atomic_store(&k.done,true) + walker_run(&k.walk) +} + +@(private) +Commit_Item :: struct { + want:i32, + generation:u32, +} + +// The commit of the batch that holds the last chunks of the wants that settle. +@(private) +Commit_Slot :: struct { + x:^Exchange, + used:bool, + items:[]Commit_Item, + count:int, + committed:store.Error, + counter:jobs.Counter, + done:bool, +} + +@(private) +commit_job :: proc(data:rawptr,worker:int) { + c:=(^Commit_Slot)(data) + defer sync.atomic_store(&c.done,true) + c.committed=store.commit(c.x.desc.store) +} + +// A chunk a receiver reported damaged, checked against its address. +@(private) +Check_Slot :: struct { + x:^Exchange, + used:bool, + address:store.Hash, + counter:jobs.Counter, + done:bool, +} + +@(private) +check_job :: proc(data:rawptr,worker:int) { + c:=(^Check_Slot)(data) + defer sync.atomic_store(&c.done,true) + store.chunk_check(c.x.desc.store,c.address) +} + +// Runs a job of a slot on the owner's worker, or now without a job system. +// A chunk's work is short and the pump may wait for it, helping: it is of the +// frame class; a commit waits for the disk and is background. +@(private) +work_run :: proc(x:^Exchange,procedure:jobs.Job_Proc,data:rawptr,counter:^jobs.Counter,class:=jobs.Class.Frame) { + js:=x.desc.worker.system + if js==nil { + procedure(data,0) + return + } + jobs.run(js,x.desc.worker.worker,class,procedure,data,counter) +} + +@(private) +work_wait :: proc(x:^Exchange,counter:^jobs.Counter) { + if js:=x.desc.worker.system;js!=nil {jobs.wait(js,x.desc.worker.worker,counter)} +} + +// Waits for every job of the exchange. +@(private) +work_wait_all :: proc(x:^Exchange) { + if x.desc.worker.system==nil {return} + for k in 0..0 { + r:=&x.receives[x.receive_head] + if !sync.atomic_load(&r.done) {break} + if receive_apply(x,r) {continue} + receive_free(x,r) + x.receive_head=(x.receive_head+1)%len(x.receives) + x.receive_count-=1 + } + for &k in x.walks { + if k.used && sync.atomic_load(&k.done) { + k.used=false + walk_apply(x,&k) + } + } + if x.commit.used && sync.atomic_load(&x.commit.done) { + x.commit.used=false + commit_apply(x) + } + for &c in x.checks { + if c.used && sync.atomic_load(&c.done) {c.used=false} + } +} + +// A free receive slot with size bytes in the ring, the oldest applied first +// while there is no room. +@(private) +receive_slot :: proc(x:^Exchange,size:int) -> ^Receive_Slot { + at:=-1 + for { + if x.receive_count==0 {x.receive_tail=0} + if x.receive_count=0 {break} + r:=&x.receives[x.receive_head] + work_wait(x,&r.counter) + work_collect(x) + } + r:=&x.receives[(x.receive_head+x.receive_count)%len(x.receives)] + x.receive_count+=1 + r.offset,r.size=at,size + r.bytes=x.receive_bytes[at:][:size] + x.receive_tail=at+size + r.walk=nil + sync.atomic_store(&r.done,false) + return r +} + +// Where size bytes fit after the last slot's in the ring; -1 when they do not. +@(private) +receive_room :: proc(x:^Exchange,size:int) -> int { + if x.receive_count==0 {return 0} + head:=x.receives[x.receive_head].offset + tail:=x.receive_tail + if tail>=head { + if size<=len(x.receive_bytes)-tail {return tail} + if size bool { + if r.want<0 {return false} + w:=&x.wants[r.want] + current:=w.used && w.generation==r.generation + p:=&x.peers[r.peer] + same_peer:=p.used && p.epoch==r.epoch + if r.need<0 && r.result==.Mismatch { + if current && same_peer { + buffer:[MSG_ADDRESS_BYTES]u8 + if !control_append(p,buffer[:encode_address(buffer[:],MSG_BAD,r.address)]) {p.refused+=1} + lie(x,int(r.want),r.peer) + } + return false + } + if r.result==.Checked { + // Its parent was parsed since: stored only if it names it. + k:=need_find(x,r.address) + if !current || k<0 || int(x.needs[k].want)!=int(r.want) || x.needs[k].arriving { + if same_peer {p.refused+=1} + return false + } + n:=&x.needs[k] + n.arriving=true + r.need,r.serial,r.kind,r.level,r.entry=i32(k),n.serial,n.kind,n.level,n.size + r.limit=walk_room(x,int(r.want)) + receive_walker(x,r) + sync.atomic_store(&r.done,false) + work_run(x,receive_job,r,&r.counter) + return sync.atomic_load(&r.done)==false || receive_apply(x,r) + } + k:=int(r.need) + held:=current && x.needs[k].used && x.needs[k].serial==r.serial + if held {x.needs[k].arriving=false} + if !held {return false} + wi:=int(r.want) + switch r.result { + case .Checked,.Stored: + case .Mismatch: + if same_peer { + buffer:[MSG_ADDRESS_BYTES]u8 + if !control_append(p,buffer[:encode_address(buffer[:],MSG_BAD,r.address)]) {p.refused+=1} + lie(x,wi,r.peer) + } else { + n:=&x.needs[k] + if n.peer>=0 { + n.peer=-1 + w.free+=1 + } + } + return false + case .Corrupt: + want_corrupt(x,wi) + return false + case .Failed: + want_fail(x,wi,.Store,r.error) + return false + } + x.uncommitted=true + kind,size:=r.kind,r.entry + need_remove(x,k) + x.dirty=true + switch kind { + case .Data: w.here+=i64(size) + case .Root: + w.total=i64(r.parsed_total) + if !w.walk_job && !w.measure { + if r.walk!=nil { + w.walking,w.walked,w.blocked,w.depth=true,false,false,0 + walk_graft(x,wi,r.walk) + } else { + want_walk_root(x,wi) + } + } + case .Node: + // The walk waits for this node: what its job walked under it goes on its stack. + if w.walking && w.blocked && !w.walk_job && w.waiting_for==r.address && r.walk!=nil {walk_graft(x,wi,r.walk)} + } + attend(x,wi) + return false +} + +// Puts on a want's walk what a job walked from a node of it: the root, or the +// node the walk waits for, whose entry the top frame then passes. +@(private) +walk_graft :: proc(x:^Exchange,wi:int,k:^Walker) { + w:=&x.wants[wi] + if k.error!=.None { + if k.error==.Missing {want_begin(x,wi)} else {want_corrupt(x,wi)} + return + } + if k.root && k.total>=0 {w.total=k.total} + w.here+=k.here + if w.depth>0 {w.stack[w.depth-1].index+=1} + if !k.walked { + if w.depth+k.depth>WALK_DEPTH { + want_corrupt(x,wi) + return + } + copy(w.stack[w.depth:],k.stack[:k.depth]) + w.depth+=k.depth + } + w.blocked,w.waiting_for=k.blocked,k.waiting + walk_take(x,wi,k.found[:k.count]) + if w.depth==0 {w.walked=true} +} + +// Records what a walk found that is not asked already. +@(private) +walk_take :: proc(x:^Exchange,wi:int,found:[]Walk_Found) { + w:=&x.wants[wi] + for f in found { + if k:=need_find(x,f.address);k>=0 { + if int(x.needs[k].want)!=wi {w.shared=true} + continue + } + need_add(x,wi,f.kind,f.address,f.level,f.size) + } +} + +// How many chunks a walk of want i may find now. +@(private) +walk_room :: proc(x:^Exchange,i:int) -> int { + w:=&x.wants[i] + return max(min(WALK_AHEAD-w.free,len(x.needs)-x.need_count),0) +} + +// Hands the walk of want i to a job, when it may go on and a slot is free. +@(private) +walk_start :: proc(x:^Exchange,i:int) { + w:=&x.wants[i] + if w.state!=.Asking || !w.walking || w.walked || w.walk_job || w.settling {return} + if w.blocked && need_find(x,w.waiting_for)>=0 {return} + room:=walk_room(x,i) + if room==0 {return} + for &k in x.walks { + if k.used {continue} + k.used,k.want,k.generation=true,i32(i),w.generation + walker_begin(&k.walk,{},true,room) + copy(k.walk.stack[:],w.stack[:w.depth]) + k.walk.depth=w.depth + k.walk.given,k.walk.given_bytes={},nil + if w.total>=0 {k.walk.total=w.total} + w.walk_job=true + sync.atomic_store(&k.done,false) + work_run(x,walk_job,&k,&k.counter) + return + } +} + +@(private) +walk_apply :: proc(x:^Exchange,k:^Walk_Slot) { + wi:=int(k.want) + w:=&x.wants[wi] + if !w.used || w.generation!=k.generation {return} + w.walk_job=false + if w.state!=.Asking || !w.walking {return} + if k.walk.error!=.None { + if k.walk.error==.Missing {want_begin(x,wi)} else {want_corrupt(x,wi)} + return + } + if k.walk.total>=0 {w.total=k.walk.total} + w.here+=k.walk.here + copy(w.stack[:],k.walk.stack[:k.walk.depth]) + w.depth=k.walk.depth + w.blocked,w.waiting_for=k.walk.blocked,k.walk.waiting + walk_take(x,wi,k.walk.found[:k.walk.count]) + w.walked=k.walk.walked + attend(x,wi) + x.dirty=true +} + +// Commits for the wants that settle, at most once a COMMIT_SECONDS, and what +// arrived meanwhile once a COMMIT_IDLE. +@(private) +settle :: proc(x:^Exchange) { + c:=&x.commit + if c.used {return} + if x.settling==0 && !(x.uncommitted && x.now-x.commit_time>=COMMIT_IDLE) {return} + if x.settling>0 && x.now-x.commit_time (app:^App,error:App_Error) { } } if desc.assets_root!="" { - if error=app_assets_create(app,desc.assets_root,desc.assets_cache,desc.asset_mounts,desc.assets_imported);error!=.None { + if error=app_assets_create(app,desc);error!=.None { app_destroy(app) return nil,error } diff --git a/host/assets.odin b/host/assets.odin index ae7e690f4..78ac7bda8 100644 --- a/host/assets.odin +++ b/host/assets.odin @@ -12,6 +12,7 @@ import "mirai:rhi" import "mirai:std/base" import "mirai:std/geom" import mm "mirai:std/math" +import "mirai:store" // The assets the worlds use, by id. An asset is loaded the first time // something asks for it, as a transaction between frames, or read by a job @@ -41,13 +42,15 @@ ASSETS_LABEL_BYTES :: 32 // of what a mesh made at run time is (Mesh_Generate_Op Assets_Desc :: struct { root:string, // folder of the asset registry - imported:string, // the imported folder that goes with root (asset.Registry); "" for none + listed:[]asset.Package_Listed, // the outputs of the packages of the project root is the assets of (project_assets) + package_problems:[]string, // what those packages and the ones of the mounts could not list: kept for whoever reports asset problems cache:string, // file where the registry keeps what it scanned; "" for none renderer:^render.Renderer, // nil: nothing is uploaded allocator:mem.Allocator, // context.allocator when unset capacity:int, // the most assets of each kind loaded at once, and derived material rows: the tables grow to it as they load. 0: ASSETS_DEFAULT_CAPACITY error_capacity:int, // 0: ASSETS_DEFAULT_ERRORS mounts:[]asset.Registry_Mount, // folders seen beside root under a prefix: the assets of active plugins + store_root:string, // of the store the listed outputs are read from: "" the machine's (store.default_root); opened only when something is listed } Assets_Kind :: enum u8 {Mesh,Texture,Material,Scene,Shape,Document} @@ -61,6 +64,7 @@ Assets_Problem :: enum u8 { Rejected, // the renderer did not take it: Renderer.error says why Capacity, // no room for another asset of its kind Coverage, // a masked material whose Surface program sets alpha: a mask map (maps.mask) cuts it, never its program + Not_Here, // a part of its file is not in the store yet: it loads when asked for again once it arrives, and fails nothing } Assets_Error :: struct { @@ -183,6 +187,7 @@ Assets_Stats :: struct { reloads:int, unloads:int, // assets that assets_collect unloaded streamed:int, // ranges of levels given to textures that follow what the GPU samples + not_here:int, // loads of meshes and textures a part of whose file the store did not hold yet errors_dropped:int, // errors that did not fit in the list } @@ -225,6 +230,9 @@ Assets :: struct { missing_texture:Assets_Texture, missing_texture_ready:bool, errors:[dynamic]Assets_Error, + package_problems:[]string, // Assets_Desc.package_problems, owned + store:store.Store, // where the listed outputs of packages are read: opened when something is listed + store_root:string, // Assets_Desc.store_root, owned revision:u64, // advances when something that was given out changes: who holds handles or rows asks again structure:u64, // advances when the instances of a scene are no longer what they need: who extracted it does it again structure_changes:[ASSETS_STRUCTURE_LOG]Assets_Structure_Change, @@ -276,10 +284,10 @@ assets_init :: proc(a:^Assets,desc:Assets_Desc) -> bool { allocator:=desc.allocator.procedure!=nil?desc.allocator:context.allocator capacity:=desc.capacity>0?desc.capacity:ASSETS_DEFAULT_CAPACITY a^={allocator=allocator,renderer=desc.renderer} - listed:=asset.registry_init(&a.registry,desc.root,desc.cache,allocator,desc.imported) + listed:=asset.registry_init(&a.registry,desc.root,desc.cache,allocator,desc.listed) // A copy of an asset made outside the editor gets an id of its own. a.registry.resolve_duplicates=true - for mount in desc.mounts {listed&&=asset.registry_mount(&a.registry,mount.folder,mount.prefix,mount.imported)} + for mount in desc.mounts {listed&&=asset.registry_mount(&a.registry,mount.folder,mount.prefix,mount.listed)} asset.types_init(&a.types,allocator) a.index=make(map[base.Asset_Id]Assets_Ref,allocator) a.derived=make(map[u64]u32,allocator) @@ -304,6 +312,7 @@ assets_init :: proc(a:^Assets,desc:Assets_Desc) -> bool { a.free_scenes,errors[5]=make([dynamic]u32,0,start,allocator) a.errors,errors[6]=make([dynamic]Assets_Error,0,desc.error_capacity>0?desc.error_capacity:ASSETS_DEFAULT_ERRORS,allocator) for error in errors {ok&&=error==nil} + if ok {a.package_problems,ok=assets_texts_clone(desc.package_problems,allocator)} if desc.renderer!=nil {a.texture_assets=make([dynamic]base.Asset_Id,allocator)} ok=ok && cap(a.meshes)==capacity && cap(a.textures)==capacity && cap(a.materials)==capacity && cap(a.scenes)==capacity ok=ok && cap(a.material_descs)==capacity*2 && cap(a.row_marks)==capacity*2 && cap(a.row_hashes)==capacity*2 && cap(a.free_rows)==start && cap(a.free_meshes)==start && cap(a.rows_dirty)==ASSETS_ROWS_DIRTY @@ -314,6 +323,13 @@ assets_init :: proc(a:^Assets,desc:Assets_Desc) -> bool { a.indexed=ok && asset.registry_load(&a.registry,index)==.None delete(index,allocator) } + if ok { + no_root:mem.Allocator_Error + a.store_root,no_root=strings.clone(desc.store_root,allocator) + ok=no_root==nil + } + // What packages list is read from the store, by its parts, while the assets live. + if ok && assets_listing(desc) {assets_store(a)} if !ok || (!a.indexed && asset.registry_scan(&a.registry)!=.None) { assets_destroy(a) return false @@ -321,6 +337,31 @@ assets_init :: proc(a:^Assets,desc:Assets_Desc) -> bool { return true } +// The store the listed outputs are read from, opened the first time something +// listed is read: at the start, or once a project that had no package lists one. +// nil when it does not open. +@(private="file") +assets_store :: proc(a:^Assets) -> ^store.Store { + if a.registry.store!=nil {return a.registry.store} + folder:=a.store_root!="" ? a.store_root : store.default_root(context.temp_allocator) + if folder=="" {return nil} + opened,open_error:=store.open(folder,a.allocator) + if open_error!=.None {return nil} + a.store=opened + a.registry.store=&a.store + return a.registry.store +} + +// Whether a registry of desc lists outputs of packages. +@(private="file") +assets_listing :: proc(desc:Assets_Desc) -> bool { + if len(desc.listed)>0 {return true} + for mount in desc.mounts { + if len(mount.listed)>0 {return true} + } + return false +} + assets_destroy :: proc(a:^Assets) { if a.allocator.procedure==nil {return} for &scene in a.scenes {asset.asset_close(&scene.file)} @@ -334,9 +375,28 @@ assets_destroy :: proc(a:^Assets) { for &scope in a.scopes {delete(scope.ids)} asset.types_destroy(&a.types) asset.registry_destroy(&a.registry) + store.close(&a.store) + delete(a.store_root,a.allocator) + for problem in a.package_problems {delete(problem,a.allocator)} + delete(a.package_problems,a.allocator) a^={} } +@(private="file") +assets_texts_clone :: proc(texts:[]string,allocator:mem.Allocator) -> (kept:[]string,ok:bool) { + if len(texts)==0 {return nil,true} + no_room:runtime.Allocator_Error + if kept,no_room=make([]string,len(texts),allocator);no_room!=nil {return nil,false} + for text,i in texts { + if kept[i],no_room=strings.clone(text,allocator);no_room!=nil { + for done in kept[:i] {delete(done,allocator)} + delete(kept,allocator) + return nil,false + } + } + return kept,true +} + // Looks at the folder again: for files that appeared or went away. A folder // with a packed index is what its index says, and is not looked at. assets_scan :: proc(a:^Assets) -> bool {return a.indexed || asset.registry_scan(&a.registry)==.None} @@ -376,6 +436,8 @@ assets_unmount :: proc(a:^Assets,prefix:string) -> asset.Error { assets_errors :: proc(a:^Assets) -> []Assets_Error {return a.errors[:]} assets_errors_clear :: proc(a:^Assets) {clear(&a.errors)} +// What the packages of the project could not list, one line per package that names it. +assets_package_problems :: proc(a:^Assets) -> []string {return a.package_problems} // The id of the asset at a path of the folder, as its file is spelled // ("arena/Chrome.mat"): what code that names an asset by path resolves. @@ -454,6 +516,12 @@ assets_open :: proc(a:^Assets,id:base.Asset_Id,kind:Assets_Kind,file:^asset.Asse return } +// What reading the tables of an open file meets: a part not in the store yet, or a file that contradicts itself. +@(private="file") +assets_view_problem :: proc "contextless" (detail:asset.Error) -> Assets_Problem { + return .Not_Here if detail==.Not_Here else .Corrupt +} + // What loading an asset meets when its file does not open. @(private="file") assets_problem_of :: proc "contextless" (detail:asset.Error) -> (problem:Assets_Problem,ok:bool) { @@ -461,23 +529,44 @@ assets_problem_of :: proc "contextless" (detail:asset.Error) -> (problem:Assets_ case .None: return {},true case .Not_Found: return .Missing,false case .IO: return .Unreadable,false + case .Not_Here: return .Not_Here,false } return .Corrupt,false } -// A transaction owned by the caller, which closes file with asset_close. +// A transaction owned by the caller, which closes file with asset_close. A +// listed output of a package opens from the store, where each part is read +// when it is first asked for: one the store does not hold yet is Not_Here. assets_file :: proc(a:^Assets,id:base.Asset_Id,kind:asset.Fourcc,file:^asset.Asset) -> asset.Error { if a==nil || file==nil {return .Invalid} record,found:=asset.registry_find(&a.registry,id) if !found {return .Not_Found} if record.kind!=kind {return .Invalid} limit:=assets_byte_limit(a,record.path) - data:=asset.registry_read(&a.registry,record.path,a.allocator,limit) or_return - assets_file_take(file,data,id,kind,a.allocator,limit) or_return + if object,listed:=asset.registry_listed_object(&a.registry,record.path);listed { + s:=assets_store(a) + if s==nil {return .Not_Found} + assets_object_open(file,s,object,id,kind,a.allocator,limit) or_return + } else { + data:=asset.registry_read(&a.registry,record.path,a.allocator,limit) or_return + assets_file_take(file,data,id,kind,a.allocator,limit) or_return + } a.stats.loads+=1 return .None } +// Opens the asset of the store at object as assets_file_take opens a file: its id and its kind +// those of the record. Any thread. +@(private="file") +assets_object_open :: proc(file:^asset.Asset,s:^store.Store,object:store.Hash,id:base.Asset_Id,kind:asset.Fourcc,allocator:mem.Allocator,limit:u64,decoder:^asset.Decoder=nil) -> asset.Error { + asset.asset_open_object(file,s,object,allocator,limit,decoder) or_return + if file.blob.header.id!=id || file.blob.header.kind!=kind { + asset.asset_close(file) + return .Corrupt + } + return .None +} + // What a file of a guest mount (assets_mount) may take, read or decompressed: // what a player downloaded is refused before it takes the memory of the game. ASSETS_GUEST_LIMIT :: 64<<20 @@ -501,14 +590,16 @@ assets_file_take :: proc(file:^asset.Asset,data:[]u8,id:base.Asset_Id,kind:asset } // Where a job reads the file of an asset: a copy of what the registry holds -// in memory, or its path on disk. Its kind and content hash are those of the -// record when it was found. +// in memory, its path on disk, or the object of the store a package lists. +// Its kind and content hash are those of the record when it was found. Assets_Source :: struct { id:base.Asset_Id, kind:asset.Fourcc, hash:u64, path:string, data:[]u8, + object:store.Hash, + store:^store.Store, // of object, which outlives the source allocator:mem.Allocator, // of path and data: one any thread can use limit:u64, // the most its file may take; 0: asset.DEFAULT_MEMORY_LIMIT } @@ -519,7 +610,10 @@ assets_source :: proc(a:^Assets,id:base.Asset_Id,allocator:mem.Allocator) -> (so record:=asset.registry_find(&a.registry,id) or_return if record.state!=.Asset {return} source={id=id,kind=record.kind,hash=record.content_hash,allocator=allocator,limit=assets_byte_limit(a,record.path)} - if held,in_memory:=asset.registry_memory_find(&a.registry,record.path);in_memory { + if object,listed:=asset.registry_listed_object(&a.registry,record.path);listed { + source.object,source.store=object,assets_store(a) + if source.store==nil {return {},false} + } else if held,in_memory:=asset.registry_memory_find(&a.registry,record.path);in_memory { if held.removed {return {},false} data,error:=asset.asset_bytes_clone(held.data,allocator) if error!=.None {return {},false} @@ -536,6 +630,7 @@ assets_source_open :: proc(source:^Assets_Source,file:^asset.Asset,decoder:^asse data:=source.data source.data=nil limit:=source.limit!=0?source.limit:asset.DEFAULT_MEMORY_LIMIT + if source.store!=nil {return assets_object_open(file,source.store,source.object,source.id,source.kind,source.allocator,limit,decoder)} if data==nil {data=asset.asset_file_read(source.path,source.allocator,limit) or_return} return assets_file_take(file,data,source.id,source.kind,source.allocator,limit,decoder) } @@ -547,6 +642,28 @@ assets_source_free :: proc(source:^Assets_Source) { source^={} } +// Forgets the entry of a mesh or a texture a part of whose file is not in the +// store yet: the stand-in shows meanwhile, nothing fails, and a later +// request loads it again. +@(private="file") +assets_not_here :: proc(a:^Assets,id:base.Asset_Id) { + ref,known:=a.index[id] + if !known {return} + #partial switch ref.kind { + case .Mesh: + a.meshes[ref.index]={handle=ASSETS_MESH_NONE} + append(&a.free_meshes,ref.index) + a.stats.meshes-=1 + case .Texture: + a.textures[ref.index]={} + append(&a.free_textures,ref.index) + a.stats.textures-=1 + case: return + } + delete_key(&a.index,id) + a.stats.not_here+=1 +} + // An entry for another asset of a kind: one an unloaded asset left, or a new one while there is room. @(private="file") assets_slot :: proc(list:^[dynamic]$T,free:^[dynamic]u32,region:^base.Region) -> (at:u32,ok:bool) { @@ -636,7 +753,7 @@ assets_mesh_load :: proc(a:^Assets,id:base.Asset_Id,entry:^Assets_Mesh,replace:= @(private="file") assets_mesh_view :: proc(file:^asset.Asset,types:^asset.Types) -> (view:asset.Mesh_View,problem:Assets_Problem,detail:asset.Error,ok:bool) { view_error:asset.Error - if view,view_error=asset.mesh_view(file,types);view_error!=.None {return {},.Corrupt,view_error,false} + if view,view_error=asset.mesh_view(file,types);view_error!=.None {return {},assets_view_problem(view_error),view_error,false} if !asset.mesh_check_indices(view) {return {},.Corrupt,.Corrupt,false} return view,{},.None,true } @@ -704,6 +821,11 @@ assets_mesh :: proc(a:^Assets,id:base.Asset_Id) -> ^Assets_Mesh { if entry==nil {return assets_missing_mesh(a)} scope:=load_asset_begin(a.load_profile) problem,detail,ok:=assets_mesh_load(a,id,entry) + if !ok && problem==.Not_Here { + load_asset_end(a.load_profile,scope,id,.Mesh,0,true) + assets_not_here(a,id) + return assets_missing_mesh(a) + } if !ok { assets_fail(a,id,.Mesh,problem,detail) entry^=assets_missing_mesh(a)^ @@ -1132,12 +1254,12 @@ assets_texture_load :: proc(a:^Assets,id:base.Asset_Id,entry:^Assets_Texture,rep if problem,detail,ok=assets_open(a,id,.Texture,&file);!ok {return} defer asset.asset_close(&file) view,view_error:=asset.texture_open(&file,&a.types) - if view_error!=.None {return .Corrupt,view_error,false} + if view_error!=.None {return assets_view_problem(view_error),view_error,false} levels:Assets_Levels defer assets_levels_free(&levels) first:=0 if a.renderer!=nil {first,_=assets_texture_first(view,render.texture_limits(a.renderer),0)} - if view_error=asset.texture_decode(&file,&a.types,&view,first);view_error!=.None {return .Corrupt,view_error,false} + if view_error=asset.texture_decode(&file,&a.types,&view,first);view_error!=.None {return assets_view_problem(view_error),view_error,false} if a.renderer!=nil { if problem,detail,ok=assets_texture_levels(view,render.texture_limits(a.renderer),0,&levels,a.allocator);!ok {return} } @@ -1320,6 +1442,11 @@ assets_texture :: proc(a:^Assets,id:base.Asset_Id) -> ^Assets_Texture { if entry==nil {return nil} scope:=load_asset_begin(a.load_profile) problem,detail,ok:=assets_texture_load(a,id,entry) + if !ok && problem==.Not_Here { + load_asset_end(a.load_profile,scope,id,.Texture,0,true) + assets_not_here(a,id) + return assets_missing_texture(a) + } if !ok { assets_fail(a,id,.Texture,problem,detail) entry^=assets_missing_texture(a)^ @@ -1690,7 +1817,7 @@ assets_prepare :: proc(prepared:^Assets_Prepared,limits:Assets_Limits,types:^ass } if view_error==.None {view_error=asset.texture_decode(&prepared.file,types,&view,first)} if view_error!=.None { - prepared.problem,prepared.detail=.Corrupt,view_error + prepared.problem,prepared.detail=assets_view_problem(view_error),view_error return } prepared.texture,prepared.ok,prepared.cost=view.info^,true,len(view.texels) @@ -1740,7 +1867,9 @@ assets_take :: proc(a:^Assets,prepared:^Assets_Prepared,over_frames:=false) -> A if entry==nil {return .Taken} if ok {problem,detail,ok=assets_mesh_install(a,prepared.mesh,entry,ahead=&prepared.ahead,over_frames=split,upload=&prepared.mesh_upload)} if ok && prepared.mesh_upload!={} {taken=.Uploading} - if !ok { + if !ok && problem==.Not_Here { + assets_not_here(a,id) + } else if !ok { assets_fail(a,id,.Mesh,problem,detail) entry^=assets_missing_mesh(a)^ } @@ -1753,7 +1882,9 @@ assets_take :: proc(a:^Assets,prepared:^Assets_Prepared,over_frames:=false) -> A } else if ok { problem,detail,ok=assets_texture_install(a,id,prepared.texture,&prepared.levels,entry) } - if !ok { + if !ok && problem==.Not_Here { + assets_not_here(a,id) + } else if !ok { assets_fail(a,id,.Texture,problem,detail) entry^=assets_missing_texture(a)^ } diff --git a/host/assets_loading_test.odin b/host/assets_loading_test.odin index 6c15083cd..f03d16acf 100644 --- a/host/assets_loading_test.odin +++ b/host/assets_loading_test.odin @@ -2,14 +2,18 @@ package host import "base:runtime" import "core:fmt" +import "core:os" +import "core:strings" import "core:testing" import "core:thread" import "mirai:asset" import "mirai:core" +import "mirai:project" import "mirai:render" import "mirai:std/base" import "mirai:std/jobs" import mm "mirai:std/math" +import "mirai:store" // Three loads share a frame: each gets a part of what the ones before it // left; one whose next asset does not fit its part, having taken nothing, @@ -356,3 +360,112 @@ what_a_load_read_is_freed_in_a_job :: proc(t:^testing.T) { app_release(&alone,{count,&freed}) testing.expect(t,freed==3 && alone.asset_releases.calls==nil,"called in the call") } + +// The dev player sets up the assets of a project with a list of packages and +// no imported folder or source: an output built in the store loads from its +// object, a plugin's packages list under its prefix, and a build that is not on +// this machine is a problem that names its package, never a failure. A part of +// an object the store does not hold yet shows the stand-in, fails nothing, and +// loads once it arrives. +@(test) +the_assets_of_a_project_load_its_packages_from_the_store :: proc(t:^testing.T) { + folder:=base.test_folder(t,"host-packages") + if folder=="" {return} + root,staging:=fmt.tprintf("%s/game",folder),fmt.tprintf("%s/staging",folder) + plugin:=fmt.tprintf("%s/plugins/pack",root) + for made in ([]string{fmt.tprintf("%s/assets",root),fmt.tprintf("%s/assets",plugin),staging}) {testing.expect(t,os.make_directory_all(made)==nil)} + m:=project.manifest_default("Game") + base.text_set(m.plugins[0][:],"pack") + testing.expect(t,os.write_entire_file(fmt.tprintf("%s/%s",root,project.PROJECT_FILE),project.manifest_text(&m,context.temp_allocator))==nil) + pm:project.Plugin_Manifest + base.text_set(pm.name[:],"pack");base.text_set(pm.version[:],"1.0.0") + pm.parts={.Assets} + testing.expect(t,os.write_entire_file(fmt.tprintf("%s/%s",plugin,project.PLUGIN_FILE),project.plugin_text(&pm,context.temp_allocator))==nil) + + s,open_error:=store.open(fmt.tprintf("%s/store",folder),context.temp_allocator) + if !testing.expect_value(t,open_error,store.Error.None) {return} + types:asset.Types;asset.types_init(&types);defer asset.types_destroy(&types) + mesh:=base.Asset_Id{0x7061636b,1} + vertices:=[3]asset.Vertex{{position={0,0,0},normal=mm.normal_pack({0,1,0})},{position={1,0,0},normal=mm.normal_pack({0,1,0})},{position={0,0,1},normal=mm.normal_pack({0,1,0})}} + testing.expect_value(t,mesh_save_test(fmt.tprintf("%s/crate.mesh",staging),&types,mesh,vertices[:],{0,2,1}),asset.Error.None) + // The object is whole in another store; this one gets its nodes and front, not its bulk. + full,full_error:=store.open(fmt.tprintf("%s/full",folder),context.temp_allocator) + if !testing.expect_value(t,full_error,store.Error.None) {return} + defer store.close(&full) + blob,put_error:=asset.object_put_file(&full,fmt.tprintf("%s/crate.mesh",staging)) + testing.expect_value(t,put_error,asset.Error.None) + store.commit(&full) + lacking:=make([dynamic]store.Hash,context.temp_allocator) + { + o,_:=store.object_open(&full,blob) + defer store.object_close(&o) + testing.expect(t,o.root.level==0,"a small object is a root of leaves") + buffer:=make([]u8,store.CHUNK_MAX,context.temp_allocator) + copy_chunk :: proc(from,into:^store.Store,address:store.Hash,buffer:[]u8) { + bytes,_:=store.chunk_read(from,address,buffer) + store.chunk_put(into,address,bytes) + } + copy_chunk(&full,&s,blob,buffer) + bulk:=make(map[int]bool,context.temp_allocator) + for g in store.object_groups(&o) { + if g.tag!=u32(asset.TAG_BULK) {continue} + for leaf in int(g.first)..0) + os.remove_all(staging) + // The crate is built here; the atlas of the plugin's package is not. + crate:=asset.Package{id={20,20},name="crate"} + crate.assets=make([dynamic]asset.Package_Asset,context.temp_allocator) + append(&crate.assets,asset.Package_Asset{kind=asset.KIND_MESH,name="crate",id=mesh}) + pack:=asset.Package{id={30,30},name="pack"} + pack.assets=make([dynamic]asset.Package_Asset,context.temp_allocator) + append(&pack.assets,asset.Package_Asset{kind=asset.KIND_TEXTURE,name="atlas",id={31,31}}) + for entry in ([]struct{p:^asset.Package,root:string,built:bool}{{&crate,root,true},{&pack,plugin,false}}) { + manifest,error:=asset.package_put(&s,entry.p) + testing.expect_value(t,error,asset.Error.None) + if entry.built { + info,_:=asset.kind_info(asset.KIND_MESH) + b:=asset.Build{package_id=entry.p.id,final=true} + b.outputs=make([dynamic]asset.Build_Output,context.temp_allocator) + append(&b.outputs,asset.Build_Output{id=mesh,kind=asset.KIND_MESH,kind_version=info.version,object=blob}) + built,_:=asset.build_put(&s,&b) + digits:[64]u8 + testing.expect_value(t,store.ref_set(&s,asset.REF_BUILDS,store.hex(asset.build_select(manifest,entry.p),&digits),built),store.Error.None) + } + list:asset.Packages + list.packages=make([dynamic]asset.Project_Package,context.temp_allocator) + append(&list.packages,asset.Project_Package{id=entry.p.id,manifest=manifest,folder="props"}) + testing.expect_value(t,asset.packages_save(entry.root,&list),asset.Error.None) + } + + p,error:=project.open(root,"",context.temp_allocator) + if !testing.expect_value(t,error.code,project.Error_Code.None) {return} + found,ok:=project_assets(&p,s.root,context.temp_allocator) + if !testing.expect(t,ok && len(found.listed)==1 && len(found.mounts)>=1) {return} + testing.expect(t,found.mounts[0].prefix=="plugins/pack" && len(found.mounts[0].listed)==1) + testing.expectf(t,len(found.problems)==1 && strings.contains(found.problems[0],"package pack"),"%v",found.problems) + a:Assets + if !testing.expect(t,assets_init(&a,{root=fmt.tprintf("%s/assets",root),listed=found.listed,mounts=found.mounts,package_problems=found.problems,capacity=8,store_root=s.root})) {return} + defer assets_destroy(&a) + testing.expect_value(t,len(assets_package_problems(&a)),1) + waiting:=assets_mesh(&a,mesh) + testing.expect(t,waiting!=nil && waiting.failed && len(assets_errors(&a))==0 && a.stats.not_here==1,"a part not here shows the stand-in and fails nothing") + buffer:=make([]u8,store.CHUNK_MAX,context.temp_allocator) + for address in lacking { + bytes,_:=store.chunk_read(&full,address,buffer) + store.chunk_put(&s,address,bytes) + } + store.commit(&s) + loaded:=assets_mesh(&a,mesh) + testing.expect(t,loaded!=nil && !loaded.failed && loaded.vertex_count==3,"the mesh loads from its object once its parts are here") + object,listed_object:=asset.registry_listed_object(&a.registry,"props/crate.mesh") + testing.expect(t,listed_object && object==found.listed[0].object && asset.registry_file_path(&a.registry,"props/crate.mesh",context.temp_allocator)=="","a listed output is an object, no file") + pending,listed:=asset.registry_find_path(&a.registry,"plugins/pack/props/atlas.tex") + testing.expect(t,listed && pending.state==.Not_Built) +} diff --git a/host/assets_streaming.odin b/host/assets_streaming.odin index f472e1c4c..f3fa5bed0 100644 --- a/host/assets_streaming.odin +++ b/host/assets_streaming.odin @@ -249,7 +249,8 @@ texture_read_take :: proc(app:^App,read:^Texture_Read) -> bool { ref,known:=a.index[id] if !known || ref.kind!=.Texture || a.textures[ref.index].handle!=read.request.handle {return true} if !read.prepared.ok { - render.texture_stream_stop(r,read.request) + // Levels not in the store yet are asked for again; any other failure keeps what it has. + if read.prepared.problem!=.Not_Here {render.texture_stream_stop(r,read.request)} return true } switch render.texture_levels_queue(r,read.request,read.prepared.levels.data) { diff --git a/host/assets_watch.odin b/host/assets_watch.odin index 5be0293c3..2802587d1 100644 --- a/host/assets_watch.odin +++ b/host/assets_watch.odin @@ -1,23 +1,38 @@ package host +import "core:log" import "core:strings" import "mirai:asset" import "mirai:platform" +import "mirai:project" import "mirai:std/jobs" @(private) -app_assets_create :: proc(app:^App,root,cache:string,mounts:[]asset.Registry_Mount=nil,imported:="") -> App_Error { +app_assets_create :: proc(app:^App,desc:App_Desc) -> App_Error { + root:=desc.assets_root + // The project lists its packages and the assets of its plugins; what its packages could not list is + // told, and never keeps it from starting. + listed:[]asset.Package_Listed + problems:[]string + mounts:=make([dynamic]asset.Registry_Mount,context.temp_allocator) + if desc.root!="" { + if p,open_error:=project.open(desc.root,desc.engine_root,context.temp_allocator);open_error.code==.None { + found,ok:=project_assets(&p,desc.store_root,context.temp_allocator) + if !ok {return .Out_Of_Memory} + listed,problems=found.listed,found.problems + if _,no_room:=append(&mounts,..found.mounts);no_room!=nil {return .Out_Of_Memory} + } + } + if _,no_room:=append(&mounts,..desc.asset_mounts);no_room!=nil {return .Out_Of_Memory} + for problem in problems {log.warnf("assets: %s",problem)} assets,error:=new(Assets,app.allocator) if error!=nil {return .Out_Of_Memory} - if !assets_init(assets,{root=root,imported=imported,cache=cache,renderer=app.renderer,allocator=app.allocator,mounts=mounts}) { + if !assets_init(assets,{root=root,listed=listed,package_problems=problems,cache=desc.assets_cache,renderer=app.renderer,allocator=app.allocator,mounts=mounts[:],store_root=desc.store_root}) { free(assets,app.allocator) return .Assets_Failed } app.assets=assets if app.info!=nil && app.info.schema!=nil {assets.types.defaults=&app.info.schema.schema} - // What a program that edits a project saves its assets with; a shipped game saves with the open - // compressor. - if app.engine_root!="" {assets.types.codecs=asset.project_codecs(app.root,app.engine_root)} assets.load_profile=&app.load_profile // A frame never reads a texture for a material: it arrives in jobs (app_textures_stream). assets.arrive=app.renderer!=nil && jobs.worker_count(&app.jobs)>1 diff --git a/host/content.odin b/host/content.odin index 130b9805b..7e1a3d37d 100644 --- a/host/content.odin +++ b/host/content.odin @@ -255,13 +255,7 @@ content_open :: proc(a:^Assets,id:base.Asset_Id,kind:asset.Fourcc,file:^asset.As record,found:=asset.registry_find(&a.registry,id) if !found || record.kind!=kind {return false} stamp.size,stamp.time=record.size,record.time - data,error:=asset.registry_read(&a.registry,record.path,a.allocator) - if error!=.None || asset.asset_take(file,data,a.allocator)!=.None {return false} - if !asset.blob_verify_hash(&file.blob) || file.blob.header.id!=id || file.blob.header.kind!=kind { - asset.asset_close(file) - return false - } - return true + return assets_file(a,id,kind,file)==.None } @(private="file") @@ -808,18 +802,44 @@ content_collect :: proc(app:^App) -> (unloaded:int) { return len(gone) } -// The folders of assets of the active plugins of a project, as the registry -// mounts them: under plugins/, the path they have in a shipped game. -project_asset_mounts :: proc(p:^project.Project,allocator:=context.allocator) -> []asset.Registry_Mount { +// What the registry of a project lists besides the files of its assets. +Project_Assets :: struct { + listed:[]asset.Package_Listed, // the outputs of the packages of the project + mounts:[]asset.Registry_Mount, // the assets of each active plugin with the outputs of its packages, and the far meshes + problems:[]string, // what the packages of the project and its plugins could not list (asset.root_packages) +} + +// The assets of the project p besides its own files: the outputs of the +// packages of its root and of each active plugin with assets, from the store +// at store_root ("": the machine's), each plugin mounted under plugins/, +// the path its assets have in a shipped game. A project whose assets carry +// the index of a cook (a shipped game) opens no store and lists no package. +// False only when memory runs out. +project_assets :: proc(p:^project.Project,store_root:string,allocator:=context.allocator) -> (assets:Project_Assets,ok:bool) { mounts:=make([dynamic]asset.Registry_Mount,allocator) + problems:=make([dynamic]string,allocator) + own:=project.folder_path(p.root,.Assets,context.temp_allocator) + shipped:=os.exists(project.path(own,asset.REGISTRY_INDEX,allocator=context.temp_allocator)) + if !shipped { + packages,error:=asset.root_packages(p.root,store_root,allocator) + if _,no_room:=append(&problems,..packages.problems);error!=.None || no_room!=nil {return {},false} + assets.listed=packages.listed + } for folder in project.asset_folders(p,allocator) { - if folder.prefix!="" {append(&mounts,asset.Registry_Mount{folder=folder.path,prefix=folder.prefix,imported=folder.imported})} + if folder.prefix=="" {continue} + mount:=asset.Registry_Mount{folder=folder.path,prefix=folder.prefix} + if !shipped { + packages,error:=asset.root_packages(folder.root,store_root,allocator) + if _,no_room:=append(&problems,..packages.problems);error!=.None || no_room!=nil {return {},false} + mount.listed=packages.listed + } + if _,no_room:=append(&mounts,mount);no_room!=nil {return {},false} } // The far meshes the editor made, where the assets are not cooked: a game that runs them shows them. - assets:=project.folder_path(p.root,.Assets,context.temp_allocator) far:=project.cache_path(p.root,project.FAR_CACHE,allocator=allocator) - if !os.exists(project.path(assets,asset.REGISTRY_INDEX,allocator=context.temp_allocator)) && (os.is_dir(far) || os.make_directory_all(far)==nil) { - append(&mounts,asset.Registry_Mount{folder=far,prefix=project.FAR_PREFIX}) + if !shipped && (os.is_dir(far) || os.make_directory_all(far)==nil) { + if _,no_room:=append(&mounts,asset.Registry_Mount{folder=far,prefix=project.FAR_PREFIX});no_room!=nil {return {},false} } - return mounts[:] + assets.mounts,assets.problems=mounts[:],problems[:] + return assets,true } diff --git a/host/content_animation.odin b/host/content_animation.odin index 5eb9f6b14..ef9a60cc7 100644 --- a/host/content_animation.odin +++ b/host/content_animation.odin @@ -15,8 +15,8 @@ import "mirai:std/serial" // The .anim of a project keeps its keys; the runtime plays them compressed into streams // (asset.animation_compress). A compression is kept in App.animation_cache as the cooked // document a shipped game carries, named by the asset and by a hash of what it is made from: -// the .anim, the skeletons its clips are measured on, the container and .anim formats and the -// compressor. A load takes it from there when it is whole and holds the clips of the .anim +// the .anim, the skeletons its clips are measured on and the container and .anim formats. A +// load takes it from there when it is whole and holds the clips of the .anim // (content_animation_cache_matches); else, as without a cache, it compresses in a job of // the application (at once with fewer than two workers), and the main thread takes the library // at the start of the frame after the job ended (app_content_adopt), writing the cache then. diff --git a/host/player.odin b/host/player.odin index a810fac00..2e9f0d111 100644 --- a/host/player.odin +++ b/host/player.odin @@ -45,6 +45,7 @@ Player_Options :: struct { scene:string, capture:string, input:string, + store:string, // the store the packages of the project list from; "": the machine's frames:int, players:int, // local players; 0: one window:[4]i32, // where the window goes, in screen coordinates; zero: centered @@ -78,10 +79,11 @@ Player_Script :: struct { PLAYER_MEMORY_TEXT :: "mirai: out of memory to start the game" @(private) -PLAYER_USAGE_TEXT :: `Usage: [--project ] [--scene ] [--headless] [--no-gpu] [--dedicated] [--frames ] [--capture ] +PLAYER_USAGE_TEXT :: `Usage: [--project ] [--scene ] [--store ] [--headless] [--no-gpu] [--dedicated] [--frames ] [--capture ] [--validation] [--input